Fun with EKS and multiarch

The allure of arm64 instances

Getting started, but not from scratch

So, if you haven’t been living under a rock, I would guess the emergence of arm processors in the cloud has been of interest to you. Promises of better price/performance, availability on the three major clouds, and a roadmap onwards, are touted as the future.

But obviously, it’s not without pitfalls or engineering effort. What do you think would happen if you swapped all your k8s nodes today with arm64 instead of amd64?

Changing architectures, real-world example. We probably don’t want to bulldoze production, or at least not face any consequence for it. Image credit — pexels

The recommended gradual process for doing so is usually done with affinity/anti-affinity or with taints and tolerations:

One way is to taint the node with its architecture, then slowly add tolerations to all the pods.

The other is to draw the pods to the right architecture with affinity and anti-affinity, slowly letting things settle after they may error for a bit.

But there is another, much quicker way. No taints. No tolerations. No affinity and anti-affinity — and still non-destructive.

Enter QEMU, playing its role as a Linux user mode emulator.

User mode emulation, a quick primer

Using binfmt_misc, qemu will register ‘new’ binary formats for the kernel to understand and register itself as the execution wrapper.

Normally, you would see something like this:

bash: ./program: cannot execute binary file: Exec format error
program: ELF-32-bit LSB executable, ARM, EABI4 version 1 (SYSV), dynamically linked (uses share libs), for GNU/LINUX 2.6.16, not stripped

Obviously, if not assisted, an x86_64 kernel/machine can’t do much with an arm/aarch binary. It’s not the same language, so to speak.

I know a genuine banana when I see one. and look here’s a lemon! and some lime! Totally not emulated. Image credit: Pexels

But qemu can emulate the environment sufficiently for such a binary to run. When you register qemu with the linux kernel’s binfmt_misc facility, instead of trying to directly execute the file, the kernel will first check if the file’s format matches a registered one. Locating a valid binfmt registration, the appropriate qemu user emulation would be called and the executable will run. It may still have issues with loading dynamic libraries etc, if they are missing.

Now, as you may recall — containers are processes. When you run a container, sooner or later the container runtime will simply call ‘exec’ on the entrypoint, spawning a process.

So what happens if that process is not a native binary or eventually calls a non-native binary? With qemu it’s seamless. The kernel can’t even ‘tell’ it’s not running a native binary. It ‘just works’ regardless of your container runtime etc.

But slower. Definitely and noticeably slower.

So, when would this approach be recommended?

• You have many containers in your cluster, some of them third-party or otherwise hard to track architecture availability for (otherwise, you may feel comfortable with just replacing it all)
• You know most of them will run natively (otherwise you’re in for a world of performance pains, making this a moot exercise.
• You’re willing to trade in some performance degradation in turn for a quicker and safer deployment.

Also, if you’re setting up a cluster for embedded testing etc, you get all sorts of embedded architectures for free. For certain embedded targets, the host itself is so slow that a server emulating it would be much much faster.

Making the magic happen

For today’s magic we need 2 things:

• The QEMU static binary packages. The dynamic ones won’t do!
• A script to load binfmt_misc and register the executors

The qemu package is available on amazon linux 2 and will even register itself — though only after a reboot. Since we don’t want a reboot, we’ll also use a qemu registration script, available courtesy of the qemu project.

I had issues with the dynamically linked qemu (this was a pain to debug as is) and things seemed better using the statically linked version. Using the static binaries requires adding –qemu-suffix=-static parameter to the script and that’s it. You will also need –persistent to make sure emulation seeps into other kernel namespaces, or else containers won’t get the benefit of qemu.

For the next part, I’m going to need a volunteer k8s node. Image credit: Pexels

Time to add this to our user data script:

yum -y install qemu-user-static
wget https://raw.githubusercontent.com/qemu/qemu/e75941331e4cdc05878119e08635ace437aae721/scripts/qemu-binfmt-conf.sh
chmod u+x qemu-binfmt-conf.sh
./qemu-binfmt-conf.sh — qemu-path=/usr/bin — qemu-suffix=-static –persistent=yes

And presto! As a cluster node starts up, it will install and configure qemu, the configuration holding a reboot too.

Let’s test this by running an arm64 workload on an x86_64 machine:

➜ ~ kubectl exec -n laminar mycontainer-69c45b944c-99gf8 — uname -a
Linux mycontainer-69c45b944c-99gf8 5.4.209–116.363.amzn2.x86_64 #1 SMP Wed Aug 10 21:19:18 UTC 2022 aarch64 GNU/Linux

As you can see, it recognizes the kernel architecture as x86_64 but reports its own architecture as aarch64.

The opposite works too, an amd64 workload under arm64:

✗ kubectl exec -n laminar mycontainer-c69f4b4c6–4b9kq — uname -a
Linux mycontainer-c69f4b4c6–4b9kq 5.4.209–116.363.amzn2.aarch64 #1 SMP Wed Aug 10 21:19:18 UTC 2022 x86_64 GNU/Linux

Wonderful. Now we can upgrade our cluster to Graviton instances and it will keep on humming even if by chance we still have a few x86_64-only containers.