FreeBSD guest support?

[nohajc]

Has anyone tried to boot FreeBSD under libkrun on Apple Silicon? I know the supported ways of booting on arm64 (under Qemu) use either UEFI or u-boot firmware. I was wondering if direct kernel boot would be feasible like with Linux.

I already tried to build a custom kernel based on a modified FIRECRACKER config (which is for amd64 originally), I'm able to produce the correct image format more or less (raw not ELF) but when I run it, it gets stuck with CPU consumption at 100%.

I'm wondering how to even troubleshoot this. Are there any facilities for Apple hypervisor debugging?

[slp]

For that to work we would need to resurrect #284 and land it properly.

I'd recommend developing this on Linux/aarch64 and then porting it to macOS.

[nohajc]

Hm, isn't PVH boot protocol Intel-specific? I remember reading that somewhere. Might not be true. It is gated in the PR code though.

[nohajc]

Anyway, I thought I don't need PVH if the image can be copied into memory without any preprocessing and the entrypoint is known somehow (or there's a trampoline at offset 0). It looks like FreeBSD build tools can already produce a raw image using the Linux convention.

Well, I'm going to try with a Linux host and see if I can find out more. I assume it's easier to debug kvm than hvf?

[slp]

Ah, yes, you're right. I read too fast and missed that you're already using a raw image. That should work start booting.

In fact, I gave it a quick try with a prebuilt disk from here with boot_efi (with uses EDK2), and starts booting but hangs very early. Most likely, the kernel is trying to do something we don't support (PCI access?).

FWIW, the way in which I would approach this, given that the kernel (or was it the bootloader?) is able to provide some serial output, is to build a custom FreeBSD kernel and put some traces here and there, to find out where it hangs. Also, disabling everything related to PCI and ACPI (if present) would be a good idea.

[nohajc]

I don't have much experience with early boot code but I was hoping a hypervisor would basically allow me to attach a debugger and step through individual instructions. Or is that not as easy as it sounds?

[slp]

I don't have much experience with early boot code but I was hoping a hypervisor would basically allow me to attach a debugger and step through individual instructions. Or is that not as easy as it sounds?

We would need to have a gdb integration, which we don't have right now. And even when you have it, like QEMU does, it's not very reliable with hardware virtualization (works great with emulation though).

I got curious so I re-ran the boot_efi test with RUST_LOG=trace, and noticed the guest gets stuck initializing the virtio console. Most likely either FreeBSD or us aren't agreeing on some behavior. Shouldn't be hard to fix, but most likely needs playing a bit with FreeBSD's kernel to debug it.

[nohajc]

Thanks for the hint, I'm gonna take a stab at this. The only other issue I'm currently aware of is that BSD doesn't have support for virtiofs root. Should still work with a virtio-blk device though.

[nohajc]

@slp Ok, rechecked my kernel config and I didn't even have virtio_console enabled. :D

EDIT: Those VM images do have virtio console module installed though, so it must be as you said.

[slp]

Long story short, doesn't look like neither FreeBSD nor NetBSD are able to use virtio-console as kernel console. At least not in aarch64. It's possible to use the legacy console, but that revealed some latent issues in libkrun because we weren't using it that much.

In any case, building an EFI variant (cargo build --features efi,blk,net,gpu) with this branch (needs some clean up) should get FreeBSD booting with boot_efi. Tested with FreeBSD-14.3-RELEASE-arm64-aarch64.raw and everything seems to work just fine.

[nohajc]

Ok, thanks for investigating!

[nohajc]

@slp So the bottom line is, virto-console support for booting is not implemented in the *BSD, right? It's not just a bug...

[nohajc]

@slp Great news, using your branch, I was able to direct-boot the system without EFI too. Just used the GENERIC arm64 kernel config (copied from the official qemu image - they already provide the raw format as kernel.bin).

It is even possible to pass kernel command line arguments via FDT bootargs if they're prefixed like this:

FreeBSD:vfs.root.mountfrom=ufs:/dev/gpt/rootfs init_path=/init-freebsd

Also, I can already get a root shell using a slightly modified libkrun init.

[slp]

@slp So the bottom line is, virto-console support for booting is not implemented in the *BSD, right? It's not just a bug...

I might have missed something, but that's what I got from reading FreeBSD's code. I suppose it could be implemented, but I'm not sure if it's worth the effort.

[nohajc]

Of course. What are the practical benefits of virtio-console over a classic serial console anyway?

I only have problems so far with handling of ^C which doesn't terminate a process running inside the guest but instead the whole VM. Maybe that's just a configuration issue though... ^D works fine on the other hand.