[ajaysundar.k (ajaysundark)]

Run micro-VM containers on EROFS layer images mapped over virtio-pmem with DAX.

Motivation

With the host-merged rootfs, each guest copies the image bytes it reads into its own page cache and its memory snapshot, and every open, stat and read is a FUSE round trip. With DAX the guest maps the node's page cache for the layer image directly, so actors on the same layer share those pages and snapshots don't carry them. Kata already carries the dependencies: erofs, virtio_pmem and FS_DAX in the guest kernel.

Fixes #2440

What changed

• atelet pulls micro-VM images as Form{EROFS: true} (imagecache: build EROFS layers #2609). Cleanup for removing the unused host-merged code from ateworker-microvm is a follow-up.
• Each distinct layer image is a read-only virtio-pmem device (/dev/pmem1..). Layer devices sit on their own PCI segments from segment 2 up (31 slots / layers) per segment, and the VM creates only the segments it needs. Currently the limit is set as 5 segments, allowing up to 155 layer images per actor across its containers.
• The kata agent mounts each image erofs ro,dax when it creates the sandbox. The guest mounts the overlay, with the upper and work dirs on the virtio-fs share (<cid>/rw, userxattr, volatile).
• A checkpoint records each container's layer images and digests in erofs-layers.json and archives upper and work together in rootfs-rw-<cid>.tar. Restore checks the node's layer digests against this record, and accepts a layer pmem device from the snapshot config only if the record names its file and discard_writes is set. MEMORY and ROOTFS snapshots taken before this change cannot be restored.
• If the actor deleted a directory the host writes into (/etc for resolv.conf), a ROOTFS restore replaces that whiteout with an opaque directory, as overlayfs does for a mkdir over a whiteout.
• Image volumes still go over virtio-fs; ateworker-microvm: mount image volumes from EROFS layer images in the guest #2826 moves them to layer images as a follow-up, so they can also share layers across guests.

Numbers

SWE-bench astropy (14 layers), one c3-highcpu-192-metal node (COS), suspend to object storage, 15 users, 10 min per run, same tree.

Metric	host-merged	DAX
Tasks completed	158	374
Resume p50 / p99 (ms)	1,900 / 2,800	1,200 / 2,500
Suspend p50 / p99 (ms)	4,500 / 25,000	1,800 / 7,600
Task compute p50 / p99 (s)	11.0 / 13.0	7.6 / 9.2
Task wall clock p50 / p99 (s)	52 / 83	22 / 31

Verified against swe-perf prow jobs (150 users on the c3-highcpu-metal), for two days against main of the same day.

Metric	main	this PR
Tasks completed	673 / 605	729 / 745
Resume p50 (s)	2.4 / 2.6	1.6 / 1.3
Snapshot size p50 (MB)	358	167

Inside a guest, reading 568 MB of .so files raised Cached by 579 MB with the host-merged rootfs and by 0 with DAX.

Testing

• go test -race ./cmd/ateworker-microvm/... ./cmd/atelet/... ./internal/imagecache/... ./internal/ocispec/..., the root-gated imagecache tests under docker run --privileged (7.1 kernel), and hack/verify/{gofmt,go-modules,licenses,boilerplate,golangci-lint}.sh. Each commit builds alone.

• On a c3 metal node (COS): cold boot, MEMORY suspend and resume, and golden restore with a 14-layer image (segment 2) and a 40-layer image (segments 2 and 3). In the guest, the /dev/pmemN sizes match the VM config's order for all 40 devices, and a file from the 40th layer reads through the overlay. Golden restores (solo, 15 concurrent, and 38 in a 15-user run) took 0.09 to 0.25 s.

• Tests pass

• Appropriate changes to documentation are included in the PR (hack/microvm-assets/README.md, docs/api-guide.md)