Posted inVirtualization

Running ARM and RISC-V Code Directly on Linux: Performance “Hacks” with QEMU User-Mode

Virtualization tutorial - IT technology blog
Virtualization tutorial - IT technology blog

The Problem: Why Waste Time Waiting for Physical Boards?

If you work with Embedded Systems, you’ve likely spent an entire morning just building code on an old Raspberry Pi. Budget ARM or RISC-V chips often struggle to handle modern compilers. Typically, we cross-compile on an X86 machine and then use scp to push the file to the board for testing. This repetitive process is extremely time-consuming and frustrating.

In my homelab, I manage about 12 VMs on Proxmox to test IoT firmware. Simulating an entire ARM virtual machine (System Emulation) is usually heavy and consumes unnecessary resources. To optimize this, I chose to combine QEMU User-Mode and binfmt_misc. This duo allows you to run binaries from different architectures directly on X86 Linux as if they were native applications.

This technique completely eliminates the need to boot an emulated operating system. You no longer need to worry about managing RAM or setting up complex networking for virtual machines.

Core Concepts: QEMU User-Mode and binfmt_misc

How Does QEMU User-Mode Work?

Unlike System Emulation, which simulates the entire hardware, User-Mode focuses only on CPU instructions and system calls (syscalls). When an ARM program calls the open() command, QEMU intercepts it and translates it into the corresponding command on the X86 Linux kernel. By stripping away the hardware emulation layer, execution performance is significantly improved.

binfmt_misc: The Kernel’s “Interpreter”

The Linux kernel has an interesting feature called binfmt_misc. It identifies file formats based on “magic bytes” at the beginning of an executable file. When you run an ARM ELF file, the kernel immediately recognizes the architecture. If QEMU is registered as the handler, the kernel automatically calls QEMU to run that file. You can simply type ./my_arm_app instead of typing long emulation commands.

Hands-on: Configuring a Multi-Architecture Environment

I will guide you through the process on Ubuntu/Debian, the standard environment for embedded developers today.

Step 1: Tool Installation

You need to install the static version of QEMU packages. The static version is crucial because it doesn’t depend on dynamic libraries, making it stable within a chroot environment.

sudo apt update
sudo apt install -y qemu-user-static binfmt-support qemu-user

To check if the kernel is ready, look at the following directory:

ls /proc/sys/fs/binfmt_misc/

If you see files like qemu-arm or qemu-riscv64, it means the system is ready to recognize these architectures.

Step 2: Running an ARM Program on X86

Let’s try writing a small C code snippet, compiling it for ARM64 (AArch64), and running it directly. First, install the cross-compiler suite:

sudo apt install -y gcc-aarch64-linux-gnu

Create a hello.c file with simple content:

#include <stdio.h>
#include <unistd.h>

int main() {
    printf("Hello itfromzero.com team!\n");
    printf("Current architecture: ");
    fflush(stdout);
    system("uname -m");
    return 0;
}

Compile with the -static option to bundle all libraries into a single file:

aarch64-linux-gnu-gcc -static hello.c -o hello-arm64

Now, run this file like a normal Linux application:

./hello-arm64

The result will return aarch64 even though you are on an X86 machine. Amazing, right?

Step 3: Using chroot for Complex Applications

For applications that require many .so libraries, building statically is not feasible. The solution is to chroot into a complete root filesystem (rootfs) of the target architecture.

# Copy the QEMU binary to the board's rootfs
sudo cp /usr/bin/qemu-arm-static /path/to/raspberry-rootfs/usr/bin/

# Chroot into the ARM environment
sudo chroot /path/to/raspberry-rootfs /bin/bash

At this point, you can use apt install or debug code freely. You are leveraging the host machine’s CPU power to handle heavy tasks for the embedded board.

Pro-tip: Accelerating CI/CD with Docker

I often use this trick to build ARM Docker images directly on an X86 server. Thanks to binfmt_misc, Docker can execute RUN commands in a Dockerfile intended for other architectures. To enable this, you only need to run the following command once:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

After that, you can build ARM64 images as easily as X86 images. This makes the CI/CD workflow much smoother. You don’t need to maintain a “build farm” of physical boards, which is both power-hungry and difficult to manage.

Important Considerations

  1. Performance: User-Mode is still emulation, so it will be about 2-5 times slower than native code. However, this is still much faster than running directly on the weak chips of embedded boards.
  2. Syscall Limitations: Some commands that interact deeply with hardware, such as GPIO or specific registers, may not work through QEMU.
  3. Multithreading: With extremely heavy multithreaded applications, you may occasionally encounter race conditions. However, for most common applications, QEMU is very stable.

Conclusion

QEMU User-Mode and binfmt_misc are simple yet incredibly powerful tools. They change the way we develop embedded software, saving hours of compilation time. If you’re looking to experiment with RISC-V but don’t have the budget for a board yet, this is the shortest path to get started.

Share:
Tags: