msft-preview: 3.32.0 rebase#463
Open
Redent0r wants to merge 22 commits into
Open
Conversation
* Update the default branch to msft-preview in different places for the CI to work with our fork. * Add the MSFT-required SECURITY.md and corresponding dictionary entries. Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
For runtime-go and runtime-rs. See below for details
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
tools: Add initial igvm-builder and node-builder/azure-linux scripting
This branch starts introducing additional scripting to build, deploy
and evaluate the components used in AKS' Pod Sandboxing and
Confidential Containers preview features. This includes the capability
to build the IGVM file and its reference measurement file for remote
attestation.
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
tools: Improve igvm-builder and node-builder/azure-linux scripting
- Support for Mariner 3 builds using OS_VERSION variable
- Improvements to IGVM build process and flow as described in README
- Adoption of using only cloud-hypervisor-cvm on CBL-Mariner
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
tools: Add package-tools-install functionality
- Add script to install kata-containers(-cc)-tools bits
- Minor improvements in README.md
- Minor fix in package_install
- Remove echo outputs in package_build
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
tools: Enable setting IGVM SVN
- Allow setting SVN parameter for IGVM build scripting
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: introduce BUILD_TYPE variable
This lets developers build and deploy Kata in debug mode without having to make
manual edits to the build scripts.
With BUILD_TYPE=debug (default is release):
* The agent is built in debug mode.
* The agent is built with a permissive policy (using allow-all.rego).
* The shim debug config file is used, ie. we create the symlink
configuration-clh-snp-debug.toml <- configuration-clh-snp.toml.
For example, building and deploying Kata-CC in debug mode is now as simple as:
make BUILD_TYPE=debug all-confpods deploy-confpods
Also do note that make still lets you override the other variables even after
setting BUILD_TYPE. For example, you can use the production shim config with
BUILD_TYPE=debug:
make BUILD_TYPE=debug SHIM_USE_DEBUG_CONFIG=no all-confpods deploy-confpods
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
node-builder: introduce SHIM_REDEPLOY_CONFIG
See README: when SHIM_REDEPLOY_CONFIG=no, the shim configuration is NOT
redeployed, so that potential config changes made directly on the host
during development aren't lost.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
node-builder: Use img for Pod Sandboxing
Switch from UVM initrd to image format
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Adapt README instructions
- Sanitize containerd config snippet
- Set podOverhead for Kata runtime class
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
tools: Adapt AGENT_POLICY_FILE path
- Adapt path in uvm_build.sh script to comply
with the usptream changes we pulled in
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Use Azure Linux 3 as default path
- update recipe and node-builder scripting
- change default value on rootfs-builder
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Deploy-only for AzL3 VMs
- split deployment sections in node-builder README.md
- install jq, curl dependencies within IGVM script
- add path parameter to UVM install script
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Minor updates to README.md
- no longer install make package, is part of meta package
- remove superfluous popd
- add note on permissive policy for ConfPods UVM builds
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Updates to README.md
- with the latest 3.2.0.azl4 package on PMC, can remove OS_VERSION parameter
and use the make deploy calls instead of copying files by hand for variant
I (now aligned with Variant II)
- with the latest changes on msft-main, set the podOverhead to 600Mi
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: Fix SHIM_USE_DEBUG_CONFIG behavior
Using a symlink would create a cycle after calling this script again when
copying the final configuration at line 74 so we just use cp instead.
Also, I moved this block to the end of the file to properly override the final
config file.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
node-builder: Build and install debug configuration for pod sandboxing
For ease of debugging, install a configuration-clh-debug.toml for pod
sandboxing as we do in Conf pods.
Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
runtime: remove clh-snp config file usage in makefile
Not needed to build vanilla kata
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
package_tools_install.sh: include nsdax.gpl.c
Include nsdax.gpl.c
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
node-builder: fix typo in string comparison
This also fixes a shellcheck error and lets us require the
shellcheck-required job:
In ./tools/osbuilder/node-builder/azure-linux/uvm_build.sh line 34:
if [ -z "${UVM_KERNEL_HEADER_DIR}}" ]; then
^-- SC2157 (error): Argument to -z is always false due to literal strings.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
docs: node-builder: fix static check error
This fixes the below static check error to follow up on the infra fix from
kata-containers#11646:
2025-07-31T19:32:45.0031829Z time="2025-07-31T19:32:44.990004665Z" level=fatal msg="found 2 parse errors:\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Set up environment\" (heading: {Name:Set up environment MDName:Set up environment LinkName:set-up-environment Level:2})\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Install build dependencies\" (heading: {Name:Install build dependencies MDName:Install build dependencies LinkName:install-build-dependencies Level:2})" commit=1d17f56b1aa7a880468b8e25d14467c92dca8eeb name=kata-check-markdown pid=9075 source=check-markdown version=0.0.1
Note: that is likely flagged because having two headings with the same
name, even under different sections, makes it impossible to create a
canonical heading link in Markdown.
This should eventually be squashed into the node-builder commit.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
docs: node-builder: Remove references to moby-containerd-cc
As we adopted containerd2, we remove references to our prior
forked containerd version.
Signed-off-by: Manuel Huber <mahuber@microsoft.com>
node-builder: 2Mb aligned guest image size
Build the mariner guest image using IMAGE_SIZE_ALIGNMENT_MB=2.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
to-squash: node-builder: add reference to README.md
This is needed to avoid the following static-checks error:
2025-08-05T21:27:20.0028337Z [static-checks.sh:808] ERROR: Document tools/osbuilder/node-builder/azure-linux/README.md is not referenced
This commit is to be squashed into the node-builder commit.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
node-builder: build and install runtime-rs
Build and install both runtime-rs and runtime-go configs and binaries side by side:
- runtime-go:
/usr/local/bin/containerd-shim-kata-v2-go
/usr/local/share/defaults/kata-containers/configuration-clh.toml
/usr/local/share/defaults/kata-containers/configuration-clh-debug.toml
- runtime-rs:
/usr/local/bin/containerd-shim-kata-v2-rs
/usr/local/share/defaults/kata-containers/configuration-cloud-hypervisor.toml
/usr/local/share/defaults/kata-containers/configuration-cloud-hypervisor-debug.toml
Also add USE_RUNTIME_RS variable and default to "yes". This controls which runtime binary and configuration will be installed
to /usr/local/bin/containerd-shim-kata-v2 and /usr/local/share/defaults/kata-containers/configuration.toml respectively.
Also install kata-ctl (runtime-rs equivalent of kata-runtime) so we can exec into the UVM when using runtime-rs
Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
- if no limits are specified, assign a default static amount of memory (512Mi) and vcpu (1) to the UVM - if limits are specified, use those limit values for the UVM resources (don't add any extra) Signed-off-by: Saul Paredes <saulparedes@microsoft.com> runtime: Resolve high UVM memory footprint Bug: https://microsoft.visualstudio.com/OS/_workitems/edit/43668151 Rationale: This is a temporary solution for optimizing memory usage for the current mechanism of requesting resources through pod Limit annotations: - if no Limits are specified and hence WorkloadMemMB is 0, set a default value 'StaticWorkloadDefaultMem' to allocate a default amount of memory for use for containers in the sandbox in addition to the base memory - if Limits are specified, the base memory and the sum of Limits are allocated. The end user needs to be aware of the minimum memory requirements for their pods, otherwise the pod will be stuck in the ContainerCreating state Testing: Manual testing, creating pods with Limits and without limits, and with two containers where each container has a limit, tested with integration in a SPEC file where the config variables were set via environment variables via the make command Adapted by @mfrw from 3.1.0 to apply to 3.2.0 Signed-off-by: Muhammad Falak R Wani <mwani@microsoft.com> Signed-off-by: Manuel Huber <mahuber@microsoft.com> runtime: Remove unused VMM options for mem alloc - We only ever tested these fork changes with CLH+MSHV - Remove these options as we don't use QEMU/FC Signed-off-by: Manuel Huber <mahuber@microsoft.com> runtime: improved memory overhead management After these changes: 1. The value of the K8s runtime class memory overhead: - Covers the memory usage from all the Host-side components (mainly the Kata Shim and the VMM). - Doesn't include the memory usage from any Guest-side components. 2. The value of a pod memory limit specified by the user: - Is equal to the memory size of the Pod VM. - Includes the memory usage from all the Guest-side components (mainly user's workload, the Guest kernel, and the Kata Agent) - Doesn't include the memory usage from any Host-side components. Signed-off-by: Dan Mihai <dmihai@microsoft.com> runtime: fix `make test` This addresses the following errors from `make test` to allow us to require that upstream CI: https://github.com/microsoft/kata-containers/actions/runs/16656407213/job/47142422035?pr=392#step:13:53 Signed-off-by: Aurélien Bombo <abombo@microsoft.com> runtime: Allocate default workload vcpus - similar to the static_sandbox_default_workload_mem option, assign a default number of vcpus to the VM when no limits are given, 1 vcpu in this case - similar to commit c7b8ee9, do not allocate additional vcpus when limits are provided Signed-off-by: Manuel Huber <mahuber@microsoft.com>
- if no limits are specified, assign a default static amount of memory (512Mi) and vcpu (1) to the UVM - if limits are specified, use those limit values for the UVM resources (don't add any extra) Signed-off-by: Saul Paredes <saulparedes@microsoft.com> runtime-rs: Resolve high UVM memory footprint This is a port from b03db3e into runtime-rs Rationale: This is a temporary solution for optimizing memory usage for the current mechanism of requesting resources through pod Limit annotations: - if no Limits are specified and hence WorkloadMemMB is 0, set a default value 'StaticWorkloadDefaultMem' to allocate a default amount of memory for use for containers in the sandbox in addition to the base memory - if Limits are specified, the base memory and the sum of Limits are allocated. The end user needs to be aware of the minimum memory requirements for their pods, otherwise the pod will be stuck in the ContainerCreating state Testing: Manual testing, creating pods with Limits and without limits, and with two containers where each container has a limit, tested with integration in a SPEC file where the config variables were set via environment variables via the make command Signed-off-by: Saul Paredes <saulparedes@microsoft.com> runtime-rs: improved memory overhead management This is a port from 7ddec33 into runtime-rs After these changes: 1. The value of the K8s runtime class memory overhead: - Covers the memory usage from all the Host-side components (mainly the Kata Shim and the VMM). - Doesn't include the memory usage from any Guest-side components. 2. The value of a pod memory limit specified by the user: - Is equal to the memory size of the Pod VM. - Includes the memory usage from all the Guest-side components (mainly user's workload, the Guest kernel, and the Kata Agent) - Doesn't include the memory usage from any Host-side components. Signed-off-by: Saul Paredes <saulparedes@microsoft.com> runtime-rs: Allocate default workload vcpus This is a port from 9af9844 Plus ports an existing behaviour from runtime-go to also add the vcpus. See https://github.com/fidencio/kata-containers/blob/e2476f587c472d5d217df9c75cdb80193dd85994/src/runtime/pkg/oci/utils.go#L1232 - similar to the static_sandbox_default_workload_mem option, assign a default number of vcpus to the VM when no limits are given, 1 vcpu in this case - similar to commit c7b8ee9, do not allocate additional vcpus when limits are provided Signed-off-by: Saul Paredes <saulparedes@microsoft.com> runtime-rs: add test coverage for static resource management If using static management and initial size manager uses 0 for CPU or memory, we add default static values to the hv config Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
- tests that deploy pods with too small of a memory limit - tests try to set a minimum memory limit for some containerd tests - tests that use runners we don't have - tests that depend on pushing to GHCR - disable Kata Containers CI / kata-containers-ci-on-push / run-kata-deploy-tests / run-kata-deploy-tests (qemu, k3s) Also disable these for runtime-rs that fail due to resource management patches: - run-nerdctl-tests (dragonball) - run-nydus (active, dragonball) - run-nydus (lts, dragonball) Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
If memory limit is set and less than minimum, set it to minimum. This is to to account for kata-containers@0ec3403 Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Redent0r
force-pushed
the
saul/msft-preview
branch
2 times, most recently
from
July 12, 2026 01:37
5597a2e to
c647cff
Compare
Redent0r
commented
Jul 14, 2026
| @@ -1,74 +1,41 @@ | |||
| # Security Policy | |||
Author
There was a problem hiding this comment.
Upstream added a SECURITY.md recently kata-containers@4bbbcb8
Do we want to keep using ours or adopt the upstream one. I'm assuming we prefer ours for now
Redent0r
marked this pull request as ready for review
July 14, 2026 19:19
Add new documentation to assist with rebasing Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
This makes the runtime share the host Kubelet emptyDir folder with the guest instead of the agent creating an empty folder in the container rootfs. Doing so enables the Kubelet to track emptyDir usage and evict greedy pods. In other words, with virtio-fs the container rootfs uses host storage whether this is true or false, however with true, Kata uses the k8s emptyDir folder so the sizeLimit is properly enforced by k8s. Addresses the ephemeral storage part of kata-containers#12203. History: * Initially, emptyDirs are slow because they are shared from the host with 9p. kata-containers/runtime#1472 * To address above, emptyDirs are hardcoded to be created by the agent in the pause container's rootfs, potentially leveraging devicemapper and improving perf. kata-containers/runtime#1485 * The previous PR regressed an (interesting?) use case where emptyDirs were used to share data from the host to the guest, so the behavior was made configurable and `disable_guest_empty_dir = false` is introduced, defaulting to the behavior of the previous PR. kata-containers#2056 * Another resource accounting regression remains which is addressed in this PR. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This addresses an issue where the disable_guest_empty_dir=true code paths did not take into account that hugepage-backed emptyDirs should always be recreated in the guest (using guest hugepages). Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This option should be removed for 4.0, so we don't handle `false`. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Follow-up to 3dd77bf. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This patch was submitted by Miklos Szeredi: https://lore.kernel.org/fuse-devel/20260528142306.1792392-1-mszeredi@redhat.com/ It fixes a FUSE oops with the k8s-shared-volume.bats test. Fixes: kata-containers#12589 Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Follow-up to kata-containers#12373 which defaulted disable_guest_empty_dir=true for runtime-go/rs. Here we remove the config option entirely from runtime-rs to make 4.0 secure by design, as with disable_guest_empty_dir=false, a pod could starve the host storage. Closes: kata-containers#12494 Generated-by: GitHub Copilot Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Peer pods don't support fs sharing, hence we need to be thoughtful about removing disable_guest_empty_dir there (=false for peer pods today, missed it in my previous PR). So we preserve disable_guest_empty_dir=false behavior for peer pods only (ie. using guest-local mounts) but we detect the need for guest-local mounts directly in code instead of using a config flag. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This commit is expected to be dropped in the future once runtime-rs becomes the default. Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
…nously Under sandbox_cgroup_only, the runtime adds itself to the sandbox cgroup via the cgroup manager. On the systemd driver that is an asynchronous call, so the runtime can fork virtiofsd and the VMM before it has joined the cgroup. Follow the manager placement with a synchronous cgroup.procs write, so the runtime is in the sandbox cgroup before any child is forked. No-op on cgroup v1. Generated-By: GitHub Copilot (Claude) Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Add an integration test for the runtime-rs fix that co-locates the shim, the VMM and virtiofsd in the pod's kubepods cgroup under sandbox_cgroup_only on cgroup v2. The regression it guards leaves the VMM and virtiofsd in system.slice/containerd.service while only the shim reaches kubepods. Start a BestEffort Kata pod and, from the host, assert the VMM and virtiofsd share the shim's cgroup and that it is under kubepods. The shim is selected by the pod UID in its cgroup path, so a pod in another namespace is not picked. Skips dragonball (in-process VMM). Generated-By: GitHub Copilot (Claude) Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
accounts for kata-containers@c3d98fe Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Temporary patch to test rebase (retargeted to up/3.32.0). Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Redent0r
force-pushed
the
saul/msft-preview
branch
from
July 14, 2026 21:05
f44f0f5 to
ba50956
Compare
For CLH, handle larger sizes for incoming deployments by resizing the VM inherited from the template VM. i.e. incoming pod is 1024M and the template VM is 512M? Plug in a zone of size 512M. Assisted-By: Claude Opus 4.8 Signed-Off-By: Cameron Baird <cameronbaird@microsoft.com>
This patch is for the msft fork only. It wires up the static size configs such that factory respects the static_ fields. This means we have the same VM size model as is in production today: 1. With no limits specified, the user gets the static size 512Mi,1vcpu 2. With limits specified, the user gets the exact specified size which is limits.memory,limits.vcpus for the deployment Assisted-By: Clause Opus 4.8 Signed-Off-By: Cameron Baird <cameronbaird@microsoft.com>
Redent0r
force-pushed
the
saul/msft-preview
branch
from
July 15, 2026 21:22
282d599 to
9513e05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This should look very simmilar to https://github.com/microsoft/kata-containers/pull/448/commits plus:
disable_guest_empty_dir = trueby default by sprt · Pull Request runtime: Setdisable_guest_empty_dir = trueby default kata-containers/kata-containers#12373 · kata-conta…validataion:
[old, without latest templating changes]
preliminary test run: https://dev.azure.com/mariner-org/mariner/_build/results?buildId=1158094&view=logs&j=1d103282-d184-539c-6f02-9cecc7887239&t=7581b7d8-55b2-5d9a-91e4-4f9c30e5e89d (pass)