Trying to make offline ISO on macOS

[jsquyres]

Greetings!

I'm trying to make an offline ISO from the instructions in "2. Create an offline ISO" section in https://kairos.io/docs/reference/auroraboot/. I'm running on the most recent stable macOS on an ARM-based Mac, with docker desktop.

While trying to learn, I have a fairly trivial config.yaml:

#cloud-config

hostname: jms-kairos-alma9x-{{ trunc 4 .MachineID }}

users:
- name: "kairos"
  passwd: kairos
  groups:
    - admin

install:
  auto: true
  device: "auto"
  reboot: true

I'm getting two kinds of errors, but both seem similar.

With upstream images

When I use an upstream image from quay.io, I can see that AuroraBoot is actually downloading the image and copying it into $(pwd)/build/temp-rootfs. Specifically, it takes a little time to download/copy, so I can see via du -sh build that the directory is growing, finally peaking around 1.3 GB -- that seems expected. But then AuroraBoot fails. Here's the whole output:

$ docker run -v "$PWD"/config.yaml:/config.yaml \
                    -v "$PWD"/build:/tmp/auroraboot \
                    -v /var/run/docker.sock:/var/run/docker.sock \
                    --rm -ti quay.io/kairos/auroraboot \
                    --set container_image=quay.io/kairos/ubuntu:24.04-core-amd64-generic-master \
                    --set "disable_http_server=true" \
                    --set "disable_netboot=true" \
                    --cloud-config /config.yaml \
                    --set "state_dir=/tmp/auroraboot"
...
2025-06-07T22:40:44Z INF Copying cloud config cloudConfig="#cloud-config\n\n ...snipped for brevity...
2025-06-07T22:40:44Z INF [1] Copying quay.io/kairos/ubuntu:24.04-core-amd64-generic-master source to /tmp/auroraboot/temp-rootfs
WARN[0012] ignored xattr security.capability in archive  error="operation not supported"
WARN[0054] ignored xattr security.capability in archive  error="operation not supported"
2025-06-07T22:41:55Z ERR Failed to clean up tmp root dir error="2 errors occurred:\n\t* dumping the source image quay.io/kairos/ubuntu:24.04-core-amd64-generic-master to /tmp/auroraboot/temp-rootfs: mkdir /tmp/auroraboot/temp-rootfs/var/cache/luet/repos/kairos/treefs/container: permission denied\n\t* 'gen-iso' deps dump-source failed\n\n"
2 errors occurred:
	* dumping the source image quay.io/kairos/ubuntu:24.04-core-amd64-generic-master to /tmp/auroraboot/temp-rootfs: mkdir /tmp/auroraboot/temp-rootfs/var/cache/luet/repos/kairos/treefs/container: permission denied
	* 'gen-iso' deps dump-source failed

And when AuroraBoot quits, the whole temp-rootfs is wiped, so I can't see any residual errors. Since the download+copy seems to take some time, I watched this in another shell while AuroraBoot ran:

$ while true; do if [[ -d build/temp-rootfs/var/cache/luet/repos/kairos/treefs ]]; then ls -l build/temp-rootfs/var/cache/luet/repos/kairos/treefs; fi ; done

I eventually see this output (repeated a bunch of times, of course):

total 0
ls: fts_read: Permission denied

So it seems to be a real error. If I change the inner ls to look at the parent kairos directory, I see that treefs has 600 permissions on it:

total 16
-rwxr-xr-x@ 1 jsquyres  staff  692 Apr 30 06:48 202504300949-git421ee1de-repository.yaml*
drwxr-xr-x@ 3 jsquyres  staff   96 Jun  7 18:58 metafs/
-rwxr-xr-x@ 1 jsquyres  staff   20 Apr 30 06:48 SYNCTIME*
drw-------@ 2 jsquyres  staff   64 Apr 30 06:48 treefs/

Is this an error in the upstream image (i.e., it shouldn't have 600 perms), or is there an error in copying the image to the local filesystem (i.e., it should automatically handle writing directories with 600 perms correctly)?

Using a local image

When trying to use my own locally-built image, I get a slightly different error.

Specifically, I run a local registry container on my Mac and have a Dockerfile to make my own image. I build and push that image to my local registry and then give that image on the command line to AuroraBoot. Note that macOS runs CommandCenter on port 5000, so I have to map my local registry to localhost port 5001.

In this case, I see that AuroraBoot creates build/temp-rootfs, but even though it takes about 15 seconds, nothing gets put in that directory before AuroraBoot fails and removes the directory. Here's the full output:

$ docker run -v "$PWD"/config.yaml:/config.yaml \
                    -v "$PWD"/build:/tmp/auroraboot \
                    -v /var/run/docker.sock:/var/run/docker.sock \
                    --rm -ti quay.io/kairos/auroraboot \
                    --set container_image=localhost:5001/jms-kairos-alma:9.6 \                   
                    --set "disable_http_server=true" \
                    --set "disable_netboot=true" \
                    --cloud-config /config.yaml \
                    --set "state_dir=/tmp/auroraboot"
2025-06-07T22:43:23Z INF Copying cloud config cloudConfig="#cloud-config\n\n ...snipped for brevity...
2025-06-07T22:43:23Z INF [1] Copying localhost:5001/jms-kairos-alma:9.6 source to /tmp/auroraboot/temp-rootfs
2025-06-07T22:43:39Z ERR Failed to clean up tmp root dir error="2 errors occurred:\n\t* dumping the source image localhost:5001/jms-kairos-alma:9.6 to /tmp/auroraboot/temp-rootfs: chmod /tmp/auroraboot/temp-rootfs/root: permission denied\n\t* 'gen-iso' deps dump-source failed\n\n"
2 errors occurred:
	* dumping the source image localhost:5001/jms-kairos-alma:9.6 to /tmp/auroraboot/temp-rootfs: chmod /tmp/auroraboot/temp-rootfs/root: permission denied
	* 'gen-iso' deps dump-source failed

If I change my shell loop from the previous error to ls -l build/temp-rootfs, I eventually see something similar as the error, but a little different:

total 0
drwxr-xr-x  2 jsquyres  staff  64 Jun  7 19:00 etc/
dr-xr-x---  2 jsquyres  staff  64 Jun  7 19:00 root/

Could the 770 perms be causing an issue here?

Thank you!

[jsquyres]

With some more testing overnight, I found a few more things:

1. I think both of my cases actually resolve down to the same issue. My while true ... testing is inherently race-condition-prone, and it seemed to just miss that the same actual problem was happening in the 2nd case, too (i.e., with a directory of permissions of 0600).
2. I tried running Auroraboot on Linux and did not run into this problem. The issue appears to be a difference in how macOS and Linux treat directories that have 0600 permissions. Specifically:
   • In a Linux container that is bind-mounted to a macOS filesystem, you can't mkdir a child directory inside a directory with 0600 permissions:

     $ docker run --rm -it -v `pwd`/testdir:/tmp/testdir almalinux:9 /bin/bash
     [root@690ba01ba9ca /]# cd /tmp/testdir
     [root@690ba01ba9ca testdir]# ls -l
     total 0
     -rw-r--r-- 1 root root 0 Jun  8 13:50 this-is-in-the-macos-host.txt
     [root@690ba01ba9ca testdir]# mkdir foo
     [root@690ba01ba9ca testdir]# chmod 600 foo
     [root@690ba01ba9ca testdir]# mkdir foo/bar
     mkdir: cannot create directory ‘foo/bar’: Permission denied
     [root@690ba01ba9ca testdir]# 
     

   • In a Linux container that is bind-mounted to a Linux filesystem, you can mkdir a child directory inside a directory with 0600 permissions:

     $ docker run --rm -it -v `pwd`/testdir:/tmp/testdir almalinux:9 /bin/bash
     [root@fd91e8aaf40b /]# cd /tmp/testdir
     [root@fd91e8aaf40b testdir]# ls -l
     total 0
     -rw-rw-r-- 1 1000 1000 0 Jun  8 13:50 this-is-in-the-linux-host.txt
     [root@fd91e8aaf40b testdir]# mkdir foo
     [root@fd91e8aaf40b testdir]# chmod 600 foo
     [root@fd91e8aaf40b testdir]# mkdir foo/bar
     [root@fd91e8aaf40b testdir]# 
     

Hence, when I run AuroraBoot on a container on macOS, it's failing when it's trying to create contents inside a directory that it just created with 0600 permissions.

Is this something that can be fixed in AuroraBoot? Should I file a github issue?

[jsquyres]

Sorry; I wasn't clear in my previous post: kairos-init is invoked -- and is what created the directory with 0600 perms -- in the upstream image. Specifically, this is the Dockerfile used to create upstream image (essentially from https://kairos.io/v3.4.2/docs/advanced/creating_custom_cloud_images/):

ARG BASE_IMAGE=almalinux:9

FROM quay.io/kairos/kairos-init:v0.4.9 AS kairos-init

FROM ${BASE_IMAGE} AS base-kairos
ARG VARIANT=core
ARG MODEL=generic
ARG TRUSTED_BOOT=false
ARG KUBERNETES_DISTRO=k3s
ARG KUBERNETES_VERSION=latest
ARG FRAMEWORK_VERSION=latest
ARG VERSION

COPY --from=kairos-init /kairos-init /kairos-init
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug -s install -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}" --version "${VERSION}"
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug -s init -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}" --version "${VERSION}"
RUN /kairos-init -f "${FRAMEWORK_VERSION}" -l debug --validate -m "${MODEL}" -v "${VARIANT}" -t "${TRUSTED_BOOT}" -k "${KUBERNETES_DISTRO}" --k8sversion "${KUBERNETES_VERSION}" --version "${VERSION}"
RUN rm /kairos-init

[jimmykarily]

Oh! Now I get it 🤦 . Indeed:

~ $ docker run --rm -it quay.io/kairos/ubuntu:24.04-core-amd64-generic-master /bin/bash

root@4c0e047ea6c9:/# stat -c '%a %n' /var/cache/luet/repos/kairos/treefs
600 /var/cache/luet/repos/kairos/treefs

I'm trying to find at which exact point kairos-init creates that directory with no success. @Itxaka ?

[Itxaka]

I don't think that's Kairos init, that's the luet cache and init did not used luet at all as far as I know

[jimmykarily]

That's what I see too. But the Dockerfile above is "almalinux:9" -> kairos init -> final image

I don't see anything else that may have altered the upstream almalinux:9 image 🤷

[jsquyres]

FWIW, by adding multiple of these lines in my Dockerfile:

RUN ls -l /var/cache/luet/repos/kairos || :

I can see that the 0600 the /var/cache/luet/... tree is definitely being created during kairos-agent -s install ...

---

That being said, even if I add the following line at the bottom of my Dockerfile:

RUN chmod 755 /var/cache/luet/repos/kairos/treefs

AuroraBoot still fails to mirror resulting image down in a macOS-running container:

2025-06-10T15:58:42Z ERR Failed to clean up tmp root dir error="2 errors occurred:\n\t* dumping the source image MY_IP_ADDRESS:5001/jms-kairos-alma:9.6 to /tmp/auroraboot/temp-rootfs: chmod /tmp/auroraboot/temp-rootfs/root: permission denied\n\t* 'gen-iso' deps dump-source failed\n\n"
2 errors occurred:
	* dumping the source image MY_IP_ADDRESS:5001/jms-kairos-alma:9.6 to /tmp/auroraboot/temp-rootfs: chmod /tmp/auroraboot/temp-rootfs/root: permission denied
	* 'gen-iso' deps dump-source failed

The problem appears to be the /root directory in the source image, which is 0550:

[root@311cf01f7aa1 /]# ls -ld /root
dr-xr-x--- 1 root root 4096 Jun 10 15:51 /root

That being said, I'm not sure why that would be a problem; I'm unable to replicate the issue manually:

$ docker run --rm -it -v ./build:/build/ localhost:5001/jms-kairos-alma:9.6 /bin/bash
[root@839f22d2356c /]# cd /build
[root@839f22d2356c build]# mkdir root
[root@839f22d2356c build]# chmod 0550 root
[root@839f22d2356c build]# ls -l
total 4
-rw-r--r-- 1 root root 1317 Jun 10 15:58 config.yaml
dr-xr-x--- 2 root root   64 Jun 10 16:07 root
[root@839f22d2356c build]# touch root/foo
[root@839f22d2356c build]# ls -l root
total 0
-rw-r--r-- 1 root root 0 Jun 10 16:07 foo
[root@839f22d2356c build]#