Adding bundles for an airgap install on Raspberry

[tco942]

Hello,

I'm trying to install Kairos 3.5.0 with K3S on a Raspberry PI 4. I'm flashing the OS on an SSD +SATA adapter pluggued through USB to my PI. I have been following this documentation: https://kairos.io/docs/installation/raspberry/

I'm using the following container image: quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1.

Once the raw file is flashed to my Raspberry' SSD (I used BalenaEtcher), Kairos installs just fine, and the system reboot on its primary partition. The system is accessible through SSH on the local IP I have set in my cloud-init file. I have noticed that K3S is not running : I suppose that K3S images are not on the system, which prevents the service starting.

My goal is to install Kairos on this Raspberry in an airgapped environment. I have seen the following example: https://kairos.io/docs/examples/airgap/ however I have been struggling to fit this documentation to the Raspberry. I am using Auroraboot through Docker, which run in WSL2 (Ubuntu 24.04 distro) on my Windows host. I used QEMU to run Auroraboot on linux/arm64 platform.

The Kairos installation on Raspberry is using the parameter --set "disk.raw=true" to generate the raw file that will be provided to Etcher to flash the Pi. However, the Kairos airgap documentation is using the parameter --set "iso.data=<path-to-mountpoint>", which suggest that this only work for generating ISOs. This may be dumb but I have tried to combine --set "disk.raw=true with --set iso.data=..., without success. Here are the logs, just in case:

docker run --platform linux/arm64 -v $PWD/config.yaml:/config.yaml -v $PWD/build:/tmp/auroraboot -v /var/run/docker.sock:/var/run/docker.sock -v $PWD/data:/tmp/data --rm -ti quay.io/kairos/auroraboot:v0.13.0 --set "disable_http_server=true" --set "disable_netboot=true" --set "container_image=docker://$IMAGE" --set "iso.data=/tmp/data" --cloud-config /config.yaml --set "state_dir=/tmp/auroraboot" --set "disk.raw=true"

2025-11-14T15:47:07Z INF [1] Copying quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 source to /tmp/auroraboot/temp-rootfs
2025-11-14T15:47:07Z INF Copying cloud config cloudConfig="#cloud-config\n\nhostname: \"kubevip-{{ trunc 4 .MachineID }}\"\n\ninstall:\n  auto: true\n  device: \"auto\"\n  reboot: true\n bundles:\n - targets:\n   - run:///run/initramfs/live/bundle.tar\n   local_file: true\n\nusers:\n- name: \"kairos\"\n  passwd: \"kairos\"\n  groups:\n  - admin\n  ssh_authorized_keys:\n  - ssh-rsa <my-ssh-key> <my-mail>\n\nkubevip:\n  enable: true\n  eip: \"192.168.100.110\"\n\np2p:\n  # enforce the discovery of nodes on local network\n  disable_dht: true\n  network_token: \"<my-network-token>\"\n\nk3s:\n  enabled: true\n\nvpn:\n  create: false\n  use: false\n\nauto:\n  # implicit if network_token is set. Enable role assignment\n  enable: true\n  ha:\n    enable: true\n    master_nodes: 1\n\nstages:\n  initramfs:\n    - name: network-configure-static-ip\n      files:\n      - path: /etc/systemd/network/10-eth0.network\n        permissions: 0644\n        content: |\n          [Match]\n          Name=eth0\n          [Network]\n          Address=192.168.100.120/24\n"
2025-11-14T15:52:20Z INF [1] Finished copying quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 into /tmp/auroraboot/temp-rootfs
2025-11-14T15:52:20Z INF Generating raw disk '/tmp/auroraboot' from '/tmp/auroraboot/temp-rootfs' with final size 0Mb
2025-11-14T15:52:20Z INF Creating RECOVERY image
2025-11-14T15:52:22Z INF [1] Creating file system image /tmp/auroraboot-raw-image-712572143/recovery/cOS/recovery.img with size 3065Mb
2025-11-14T15:52:21Z ERR [1] failed to open /dev/loop-control
2025-11-14T15:52:21Z ERR failed to create recovery image error="open /dev/loop-control: no such file or directory" image={"FS":"ext2","File":"/tmp/auroraboot-raw-image-712572143/recovery/cOS/recovery.img","Label":"COS_SYSTEM","LoopDevice":"","MountPoint":"/tmp/auroraboot-raw-image-712572143/recovery-img","Size":3065,"Source":{},"URI":""} source=/tmp/auroraboot/temp-rootfs
2025-11-14T15:52:21Z ERR failed to create recovery partition error="open /dev/loop-control: no such file or directory"
2025-11-14T15:52:21Z ERR Generating raw disk '/tmp/auroraboot' from '/tmp/auroraboot/temp-rootfs' failed with error 'open /dev/loop-control: no such file or directory'
1 error occurred:
        * open /dev/loop-control: no such file or directory

I'm also not sure about where to place the bundle in the cloud-init config file. Here is my cloud-init file:

#cloud-config

hostname: "kubevip-{{ trunc 4 .MachineID }}"
install:
  auto: true
  device: "auto"
  reboot: true
 bundles:
 - targets:
   - run:///run/initramfs/live/bundle.tar
   local_file: true
users:
- name: "kairos"
  passwd: "kairos"
  groups:
  - admin
  ssh_authorized_keys:
  - ssh-rsa <my-ssh-key> <my-mail>
kubevip:
  enable: true
  eip: "192.168.100.110"
p2p:
  # enforce the discovery of nodes on local network
  disable_dht: true
  network_token: "<my-network-token>"
k3s:
  enabled: true
vpn:
  create: false
  use: false
auto:
  # implicit if network_token is set. Enable role assignment
  enable: true
  ha:
    enable: true
    master_nodes: 1
stages:
  initramfs:
    - name: network-configure-static-ip
      files:
      - path: /etc/systemd/network/10-eth0.network
        permissions: 0644
        content: |
          [Match]
          Name=eth0
          [Network]
          Address=192.168.100.120/24

My main question is: is there a way to provide bundles into the a raw file, and if so, how to proceed with the cloud-init config file?

Thank you.

[jimmykarily]

I had a look and unless I missed it, there doesn't seem to be a way to add data to the raw image through auroraboot flags. But I guess you can extract the source image (e.g. using luet util unpack <image_here> <output_dir>), copy the data you want to the directory and then use the directory as source for auroraboot (dir: type of source).

Regarding the error you see (open /dev/loop-control: no such file or directory), I think it's a problem with the loop devices. Could be the wsl+docker nesting or something.

[tco942]

Hello,

Thank you for your answer. I have tried extracting the source image using luet, as I have seen in the prepare_nvidia_orin_images script. However, it seems luet does not handle multi-arch manifests and isn't using Qemu (my host is linux/amd64 while I'm using an linux/arm64 container image). Here are the logs.

mkdir test
luet util unpack quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 test

INFO   Downloading quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 to /home/<my-build-dir>/test
  ERROR    no child with platform linux/amd64 in index quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1

As I have seen in this discussion, I tried using crane instead of luet to provide a --platform parameter. Here is what I've done :

crane export --platform linux/arm64 quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 container-image.tar
tar -xf container-image.tar -C base-image

Now I indeed have the rootfs in the base-image directory. I have supplied my bundle element into the system/oem/bundles directory.

However, I cannot find any mention of source type dir in the documentation, and generally speaking how to call Auroraboot with this type of source. In the Nvidia AGX documentation, it is explained how to build from a directory :

• Mount the rootfs to the container
• Set directory environment variable to state where the rootfs is

However this example overrides the entrypoint for the AGX use case.

Here are my logs if I try something similar. Note that I'm not overriding Auroraboot's entrypoint :

docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock --platform linux/arm64 -v $PWD/cloud-confi
g.yaml:/cloud-config.yaml -v $PWD/output:/output -v $PWD/base-image:/rootfs -e directory=/rootfs quay.io/kairos/auroraboot:latest --debug --set "disable_http_server=true" --set "disable_netboot=true" --set "state_dir=/output" --set "disk.raw=true" --cloud-config /cloud-config.yaml

2025-11-20T15:12:52Z DBG Cloud config cc="#cloud-config\n\nhostname: \"kubevip-{{ trunc 4 .MachineID }}\"\n\ninstall:\n  auto: true\n  device: \"auto\"\n  reboot: true\n\nbundles:\n- targets:\n  - run:///system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz\n  local_file: true\n\nusers:\n- name: \"kairos\"\n  passwd: \"kairos\"\n  groups:\n  - admin\n  ssh_authorized_keys:\n  - ssh-rsa <my-ssh-key> <my-mail>\n\nkubevip:\n  enable: true\n  eip: \"192.168.100.110\"\n\np2p:\n  # enforce the discovery of nodes on local network\n  disable_dht: true\n  network_token: \"<my-network-token>\"\n\nk3s:\n  enabled: true\n  version: \"v1.33.2+k3s1\"\n  airgap: true\n\nvpn:\n  create: false\n  use: false\n\nauto:\n  # implicit if network_token is set. Enable role assignment\n  enable: true\n  ha:\n    enable: true\n    master_nodes: 1\n\nstages:\n  initramfs:\n    - name: network-configure-static-ip\n      files:\n      - path: /etc/systemd/network/10-eth0.network\n        permissions: 0644\n        content: |\n          [Match]\n          Name=eth0\n          [Network]\n          Address=192.168.100.120/24\n          Gateway=192.168.100.1\n  boot:\n    - name: configure-keyboard\n      commands:\n      - loadkeys fr\n  after-install:\n    - name: \"copy-k3s-addons-images\"\n      commands:\n      - mkdir -p /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/\n      - cp /system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/\n\n"
2025-11-20T15:12:52Z DBG 1.
2025-11-20T15:12:52Z DBG  <prepare-dirs> (background: false)
2025-11-20T15:12:52Z DBG
2025-11-20T15:12:52Z DBG 2.
2025-11-20T15:12:52Z DBG  <copy-cloud-config> (background: false)
2025-11-20T15:12:52Z DBG
2025-11-20T15:12:52Z DBG 3.
2025-11-20T15:12:52Z DBG  <gen-raw-efi-disk> (background: false)
2025-11-20T15:12:52Z DBG
2025-11-20T15:12:52Z DBG 4.
2025-11-20T15:12:52Z DBG
2025-11-20T15:12:52Z DBG 5.
2025-11-20T15:12:52Z DBG
2025-11-20T15:12:52Z DBG Preparing destination temporal directory destination=/output
2025-11-20T15:12:52Z DBG Preparing temp rootfs directory destination=/output/temp-rootfs
2025-11-20T15:12:52Z DBG Preparing temp netboot directory destination=/output/netboot
2025-11-20T15:12:52Z INF Copying cloud config cloudConfig="#cloud-config\n\nhostname: \"kubevip-{{ trunc 4 .MachineID }}\"\n\ninstall:\n  auto: true\n  device: \"auto\"\n  reboot: true\n\nbundles:\n- targets:\n  - run:///system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz\n  local_file: true\n\nusers:\n- name: \"kairos\"\n  passwd: \"kairos\"\n  groups:\n  - admin\n  ssh_authorized_keys:\n  - ssh-rsa <my-ssh-key> <my-mail>\n\nkubevip:\n  enable: true\n  eip: \"192.168.100.110\"\n\np2p:\n  # enforce the discovery of nodes on local network\n  disable_dht: true\n  network_token: \"<my-network-token>\"\n\nk3s:\n  enabled: true\n  version: \"v1.33.2+k3s1\"\n  airgap: true\n\nvpn:\n  create: false\n  use: false\n\nauto:\n  # implicit if network_token is set. Enable role assignment\n  enable: true\n  ha:\n    enable: true\n    master_nodes: 1\n\nstages:\n  initramfs:\n    - name: network-configure-static-ip\n      files:\n      - path: /etc/systemd/network/10-eth0.network\n        permissions: 0644\n        content: |\n          [Match]\n          Name=eth0\n          [Network]\n          Address=192.168.100.120/24\n          Gateway=192.168.100.1\n  boot:\n    - name: configure-keyboard\n      commands:\n      - loadkeys fr\n  after-install:\n    - name: \"copy-k3s-addons-images\"\n      commands:\n      - mkdir -p /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/\n      - cp /system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/\n\n"
2025-11-20T15:12:52Z INF Generating raw disk '/output' from '/output/temp-rootfs' with final size 0Mb
2025-11-20T15:12:52Z ERR failed to get image label error="open /output/temp-rootfs/etc/os-release: no such file or directory"
2025-11-20T15:12:52Z ERR failed to get image label error="open /output/temp-rootfs/etc/os-release: no such file or directory"
2025-11-20T15:12:52Z DBG Got output name name=kairos--.raw
...

I truncated the logs. But here, we see that once the cloud config is copied, it follows up with the raw disk generation. While, back when I used a container_image, the steps were :

1. copying cloud config
2. copying rootfs to /output/temp-rootfs
3. generating the raw disk

I can't figure out why Auroraboot is skipping the rootfs copy step. This is the reason why etc/os-release file is missing. Could you please provide me more info ?

Thanks.

[tco942]

Hello again,

I've successfully managed to add my bundle. I'll provide the detailed instructions here in case anybody has the same issue.

• Pull and unpack your base container image (the one you used to provide to Auroraboot using --set "container_image=...") using crane.

crane export --platform linux/arm64 quay.io/kairos/ubuntu:22.04-standard-arm64-rpi4-v3.5.0-k3s-v1.33.2-k3s1 container-image.tar
mkdir base-image
sudo tar -xf container-image.tar -C base-image

• You will get a base-image folder containing the rootfs of the container image. Add your bundle there. In my use case, I put it in /system/oem/bundles/k3s.

• Call Auroraboot and provide the unpacked rootfs as a container image :

  • Mount the unpacked rootfs into Auroraboot (-v $PWD/base-image:...)
  • Set the --set "container_image=..." to where you mounted the unpacked rootfs, and don't forget to prefix it with dir://.
  • FYI, here is the full command :

docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock --platform linux/arm64 \
    -v $PWD/cloud-config.yaml:/cloud-config.yaml \
    -v $PWD/output:/output -v $PWD/base-image:/rootfs \
    quay.io/kairos/auroraboot:latest --debug \
    --set "disable_http_server=true" \
    --set "disable_netboot=true" \
    --set "state_dir=/output" \
    --set "disk.raw=true" \
    --set "container_image=dir:///rootfs" \
    --cloud-config /cloud-config.yaml

I've a few issues that I should fix now :

• K3S isn't enabled by default. Currently I'm enabling and starting it manually.
• I'm looking for the hook I should use in my cloud-init config file to move the k3s-airgap-images-arm64.tar.gz to where it is expected. Here is a snippet of my cloud-init config file :

bundles:
- targets:
  - run:///system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz
  local_file: true

stages:
  after-install:
    - name: "copy-k3s-addons-images"
      commands:
      - mkdir -p /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/
      - cp /system/oem/bundles/k3s/k3s-airgap-images-arm64.tar.gz /usr/local/.state/var-lib-rancher.bind/k3s/agent/images/

I've also tried the provider-kairos.bootstrap.before hook, without success. I'll look for the right hook next week.

[jimmykarily]

Maybe this helps: #3737 (comment)

[tco942]

Indeed it helped : I set the the copy into the boot.after hook, the archive is now where it is expected. I can start the k3s service manually and everything is up, including traefik pods.

I've noticed K3S is disabled and stopped by default. I added another stage, in the boot.after hook, to enable and start it. Works fine, but is this expected ? I was wondering if k3s should'nt be enabled by default ?

Thanks for your help.

[jimmykarily]

It should have been enabled when the installer script was run. I see that we just log the error though and we don't fail the installation. Maybe you can find what went wrong in the installation logs?

[tco942]

From what I see in the buildEvent.go, the environment variable INSTALL_K3S_SKIP_ENABLE is set to true, thus skipping both enable and start steps of k3s as stated in their install script.

Didn't look further so it might be overriden somewhere else. I'll keep my hook to enable and start k3s in the meantime.

[jimmykarily]

Enabled here? https://github.com/kairos-io/provider-kairos/blob/53e60133ef9e03fd2cd9827cb4d11cd4004eae72/internal/provider/bootstrap.go#L221