Codesigning My Way Into a Hypervisor

2026.06.15

Being June it is Tech Conference Season here in 2026, if you missed it then let me give you the run down. Have you heard the good news about AI? and that’s pretty much it. Microsoft talked about stopping shoving copilot down people’s throats, then immediately changed their mind and did a 2.5 hour keynote talking about the new ways they want to shove copilot down peoples throat. Google AI IO happened (though admittedly at the end of May). But the weirdest was Apple’s WWDC event. Unlike every other year, they didn’t announce a whole lot outside of maybe fixing Siri ™️. But most the event outside of the AI talk was back pedalling Tahoe and doing a shit load of performance improvements to the system and swift.

A session that particularly caught my eye outside of the Apple container implementation was the WWDC26: Expand the capabilities of your Virtualization app session. I remember the massive improvements they did last year and have been meaning to play with the tech for a while but had no actual reason to use it. But then I had an idea, what if I deployed my tiny Bingo image from the Booting Go on Bare Metal post on the Apple Virtualization Framework? How much of a pain will this be? Will I run out of nurofen before I get this working?

For those who want this to be a nightmare of a project and watch me struggle, I will have to apologise as the framework itself was annoyingly simple. The two things that did fight me both turned out to live in the same place, the seam between macOS and the guest, and neither was where I expected. You can rage quit now and check out the result at Charlie.

Stubbing My Toe

So, I switched to Mac around 2 years ago. I was originally running Fedora as my main machine for multiple years, but everyone at work used Apple and it became progressively annoying doing work with them, often they would run things on Mac and shit would hit the fan and because I never used a Mac I had no idea how to fix things. So I eventually just decided to bite the bullet and switch getting a Mac Mini M2 a few weeks before the M3 came out. I, like an idiot, got the 256GB storage model. So between the applications I have, the size of the OS, compliance applications and the fact that Docker will just make storage disappear into thin air, I typically have around 2-15GB of free storage. Due to how Xcode works, I just don’t have enough storage to download and use Xcode.

So instead of using Xcode which handles absolutely everything for making Apple applications, I’ve had to install Swift directly from their website so I can use the toolchain and then manually deal with everything. I have never written Swift up until this point as well. So the game plan is, learn a new language and do a hypervisor at the same time. Things Rust people could only dream of.

Structuring the Project

The Apple documentation is very good. Though it didn’t really go through a structure for the project, so I did the following:

./guest: A fork of Bingo
./host: The swift code

The fork of Bingo strips all the x86 and build tooling out of it, leaving it to be an ARM64 UEFI .img. Everything else was left the same. The goal was to deploy it with Swift using the Virtualization Framework and being able to do a curl request to the VM and getting my Hello, World! response.

To get a VM running I need 2 things, a VZVirtualMachineConfiguration which is the definition of the VM and a VZVirtualMachine which takes the config and allows you to run vm.start(). The other things I did in the project at this point was a console reader which would parse the virtio-console serial output and write it to the terminal.

Configuring the VM

The config was reasonably simple, first was setting how much resources you wanted to give it:

let config = VZVirtualMachineConfiguration()
config.cpuCount = 1                     // 1 vCPU
config.memorySize = 256 * 1024 * 1024   // 256 MB

The memory size for Bingo does matter, as the whole rootfs runs in RAM and doesn’t mount to a volume. The next thing to do was set how to actually boot the VM which in our case is a EFI.

let bootLoader = VZEFIBootLoader()
let variableStoreURL = try Self
    .supportDirectory() // Custom "Application Support" directory
    .appendingPathComponent("efi-vars.fd")

bootLoader.variableStore = try efiVariableStore(at: variableStoreURL)
config.bootLoader = bootLoader

There is a little bit to unpack here. So the supportDirectory() function that will create an Application Support directory for this application which is where we can store the guest’s runtime state in a EFI variable store. This ends up being ~/Library/Application Support/charlie/efi-vars.fd. The next thing to define was the rootfs disk and attach it as readonly.

let attachment = try VZDiskImageStorageDeviceAttachment(
    url: Bundle.module.url(forResource: "bingo.arm64", withExtension: "img")!,
    readOnly: true,
)

config.storageDevices = [VZVirtioBlockDeviceConfiguration(attachment: attachment)]

The building of the Bingo image needs to be moved to the host/Sources/host/bingo.arm64.img, this way we can Bundle the resource into the swift build. The just build target deals with this. I also had to update the Package.swift to state that this resource exists. The next thing is setting up the network and entropy for DHCP.

let network = VZVirtioNetworkDeviceConfiguration()
network.attachment = VZNATNetworkDeviceAttachment()
network.macAddress = VZMACAddress(string: "ce:a5:71:e0:00:01")!

config.networkDevices = [network]
config.entropyDevices = [VZVirtioEntropyDeviceConfiguration()]

I pin a fixed local unicast MAC so the guest gets a stable DHCP lease run to run rather than a new one every boot. The VZNATNetworkDeviceAttachment gives me a vmnet shared NAT device, the guest sits behind the host on its own little subnet and gets a lease from the built in DHCP server. I originally wanted bridged networking so the VM would land directly on my LAN, but that uses VZBridgedNetworkDeviceAttachment, which needs the com.apple.vm.networking entitlement. Unlike the virtualization entitlement I’ll fight with later, that one is restricted, Apple has to grant it against a provisioning profile tied to a paid developer account, and you cannot just self sign it on. So NAT it is. Foreshadowing that this whole post is a story about which entitlements you are and aren’t allowed to hand yourself.

The last thing I setup was the virtio-console serial device which writes to the console reader so we can get the console output from the VM and write it to the terminal.

let console = VZVirtioConsoleDeviceSerialPortConfiguration()
console.attachment = VZFileHandleSerialPortAttachment(
  fileHandleForReading: nil,
  fileHandleForWriting: consoleWriteHandle // Console reader handler
)
config.serialPorts = [console]

This is the half of the console plumbing that lives on the host. The other half is the guest actually agreeing to talk on it. A VZVirtioConsoleDevice shows up inside Linux as hvc0 (hypervisor console 0), not the ttyS0/ttyAMA0 serial ports a normal kernel logs to by default. So if the guest kernel doesn’t know to use it, you wire up this whole pipe and 2 tenths of fuck all. The fix is one token on the kernel command line in Bingo:

console=tty0 console=${CONSOLE} console=hvc0 random.trust_cpu=on

While I was here I also stubbed in a VZVirtioSocketDeviceConfiguration. I’m not using it yet, but it wires up a vsock so I can do direct host to guest IPC later without going over the network. Future me problem.

Swift and this framework is quite nice and allow us to validate the config before we actually run it by doing a try config.validate(). But once we have a valid config then we can just do the following to run it:

let machine = VZVirtualMachine(configuration: config)
machine.delegate = self

try await machine.start()

This is all the boilerplate to run the system. The rest is just parsing the logging, waiting for signals and putting things into classes, the part of any project nobody cares about. Bingo, meanwhile, is still the same do-everything PID 1 monolith I swore I would replace with coffey, a proper Go init system that has been vaporware since the day I name dropped it.

Codesigning My Way In

I run swift run and it immediately shits the bed. The moment it tries to spin up the VM it dies, because macOS will not hand you a hypervisor unless the process carries the com.apple.security.virtualization entitlement. Xcode injects this for you behind the scenes when you hit the run button, which is exactly the kind of thing I signed up to do by hand when I decided to pick the smallest hard drive for my Mac Mini.

Fortunately, the entitlement itself is tiny, just a plist with one key set to true:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>com.apple.security.virtualization</key>
  <true/>
</dict>
</plist>

This is the entitlement I mentioned earlier that you are allowed to give yourself, unlike the bridged networking one. You attach it to the binary with codesign. I don’t have a paid developer certificate and don’t need one just to run my own binary, so an ad hoc signature (--sign -) is enough:

bin="$(swift build --package-path host --show-bin-path)/host"
codesign --force --sign - --entitlements host/host.entitlements "$bin"

This is where swift run actually bites. It compiles and runs in a single step, so there is no window to slot codesign in between building the binary and executing it. The build produces an unsigned binary and immediately runs that unsigned binary, which is precisely the thing the kernel refuses to let near the Virtualization framework. So the justfile splits the two apart: just build does the swift build and then signs the binary it produced, while just run builds, signs, and only then execs the signed binary directly.

It Boots

With the binary signed, just run boots the guest headless and streams its console straight to my terminal. The host watches that console for the line where Bingo prints its DHCP lease, pulls the IP out of it, and tells me where the VM landed:

$ just run
[host] starting VM (cpu=1, mem=256MiB, image=bingo.arm64.img)
[host] VM running. Press Ctrl-C to shut down.
... kernel boot spam ...
virtio_blk virtio2: [vda] 18432 512-byte logical blocks (9.44 MB/9.00 MiB)
random: crng init done
Run /init as init process
time=1970-01-01T00:00:00.138Z level=INFO msg="mounting filesystems"
time=1970-01-01T00:00:00.139Z level=INFO msg="interface up" network.iface=eth0
time=1970-01-01T00:00:00.139Z level=INFO msg="starting DHCP exchange" network.iface=eth0
time=1970-01-01T00:00:00.152Z level=INFO msg="DHCP lease acquired" network.ip=192.168.64.3/24 network.gateway=192.168.64.1 network.dns=[192.168.64.1] network.lease_time=1h0m0s
[host] guest IP 192.168.64.3
time=1970-01-01T00:00:00.153Z level=INFO msg="server starting" app.addr=:8080

The whole boot finishes in about 150ms of guest time, which is a little ridiculous. Bingo thinks it’s 1970 because there is no clock to sync against, don’t worry about it, happens all the time. Importantly it has a lease and an HTTP server listening on :8080. So from a second terminal I do the thing this whole project existed to do:

# From another terminal
$ curl http://192.168.64.3:8080/
Hello, World!

And there it is. My tiny bootable image, the same one from Booting Go on Bare Metal, running inside Apple’s hypervisor and answering a request. Those two words have now greeted me from a scratch container, a tarball I recomputed the digests of by hand, bare QEMU metal, and a hypervisor. Watching the streamed console you can even see the request land on the guest:

time=1970-01-01T00:00:16.576Z level=INFO msg="request started" app.request_id=11bb499eaae12cc2 app.method=GET app.path=/ app.remote_addr=192.168.64.1:50343

The remote_addr is 192.168.64.1, the gateway, not my machine, because the NAT device rewrites everything to look like it came from the host side of the network. As far as the project’s stated goal goes, this is a win. I should probably have stopped reading my own logs here.

The Shutdown that Wouldn’t

Here is the part I glossed over. Hitting Ctrl-C did nothing useful. The host did its part perfectly. The signal gets trapped, hops back onto the main actor, and calls vm.requestStop(), which is the framework’s polite way of pressing the guest’s power button. Then it sets a 6 second timer, and if the guest hasn’t gone away by then, it gets terminated:

private func requestStop() {
  guard !stopping, let vm else { return }
  stopping = true
  note("requesting graceful shutdown (ACPI power button)...")

  try? vm.requestStop()

  Task { @MainActor in
    try? await Task.sleep(for: .seconds(6))
    note("guest did not stop in time; exiting")
    exit(1)
  }
}

Every single time, six seconds later, “guest did not stop in time”. The guest was very much alive, still answering HTTP, completely oblivious that the smoke alarm was blaring and it was time to away. It was getting executed, not shut down. The worst part was that the guest said absolutely nothing about it. No error, no warning, no log line. The host asked it to stop, the guest carried on, and the only evidence anything had happened was the host giving up after six seconds.

The annoying part is the guest already had all the code to handle this. It came over from the Booting Go on Bare Metal post, where Bingo scans /sys/class/input for a device named “Power Button”, reads 24 byte input_event structs off it, and watches for an EV_KEY / KEY_POWER press. On press it stops the HTTP server, syncs the filesystems, and calls reboot(POWER_OFF). That code was fine. The problem is the power button device never existed for it to find, so the listener had nothing to attach to and quietly went nowhere.

This is where Apple’s arm64 platform does something different to what my kernel was built for. On QEMU and most x86 hardware, the ACPI power button is a fixed hardware button and CONFIG_ACPI_BUTTON alone is enough to get a “Power Button” input device. On Apple’s virtual machine the power button is wired as a GPIO line on a PL061 controller, and the ACPI tables describe it with a GpioInt event that, when the host fires it, runs an ACPI method that Notify()s the button device. So the real chain is more along the lines of:

host requestStop()
  -> ACPI event
    -> PL061 GPIO line
      -> ACPI GPIO event handler
        -> Notify() the ACPI button device
          -> /dev/input/eventN "Power Button"
            -> my Go code reads KEY_POWER

My custom kernel had CONFIG_ACPI_BUTTON, the one I so confidently listed under “what we need” last post while gleefully deleting everything that wasn’t virtio, but it had no idea what a PL061 was. PL061 was collateral damage in that purge. Past me declared clean shutdown a solved problem requiring no external tools, and present me is reaping exactly that. The GPIO line was merely for show, the ACPI event stared blankly at the wall, and the button device was more of a concept. Bingo’s scan of /sys/class/input had no “Power Button” to find, so nothing downstream of it ever ran. requestStop() had been shooting blanks the whole time, which is exactly why the guest never had anything to say about it.

The fix was three lines in the arm64 kernel config:

CONFIG_GPIOLIB=y
CONFIG_GPIOLIB_ACPI=y
CONFIG_GPIO_PL061=y

Then several beers later the compilation of the kernel was done with what we needed. GPIO_PL061 is the driver for the actual controller, GPIOLIB_ACPI is the glue that lets ACPI event methods drive GPIO lines, and GPIOLIB is the subsystem both sit on top of. Rebuild the kernel, rebuild the image, and the boot log finally gains a line that simply was never there before:

time=1970-01-01T00:00:00.153Z level=INFO msg="listening for power button" acpi.device=/dev/input/event0

Now Ctrl-C does what it says. The host requests the stop, the guest catches fire safely and tears down the HTTP server, syncs, and powers off cleanly. The host’s guestDidStop delegate fires and it exits 0 well inside the 6 second grace window, no execution required.

So, Annoyingly Simple?

Mostly, yes. The Virtualization framework itself is genuinely simple. Resources, boot loader, disk, network, console, and validation are maybe sixty lines of Swift, and the first boot worked on basically the first honest attempt. I went in expecting to burn through a strip of nurofen and the framework gave me almost no reason to. The strip I saved did not stay saved for long. It went, in full, into Mango, a project I am not getting into here beyond saying it has caused more headaches than everything in this post combined. That one gets its own post.

What is funny is that both things that actually fought me lived in the same place, the seam between macOS and the guest, and both were really about permission. Getting in needed an entitlement I was allowed to grant myself (com.apple.security.virtualization, ad hoc signed), while the one I actually wanted for bridged networking I’m not (com.apple.vm.networking, Apple granted). Getting out cleanly needed the guest kernel to actually be wired for the way Apple’s hardware delivers the power button, which was three config lines and an afternoon of dicking about with a guest that gave me nothing to debug with because I refused to touch them.

No Xcode, no bridge networking, no developer certificate, just a signed binary, a 9MB disk image, and a kernel that finally knows what a power button is. Docker says ship your machine when it works on yours, this is just the gloriously wrong way round of doing exactly that. A man who will hand mangle a tarball at 1am to avoid standing up a registry has, predictably, ended up shipping an entire operating system as a 9MB file instead. The crusade continues. The hardest parts were both about asking nicely.

<< Previous Post

~~Next Post >>~~

#Swift #Virtualisation #Linux #Kernel