Written by Moritz
on May 11, 2017

Making a Raspbian Cross Compilation SDK

After setting up your Raspberry Pi with Raspbian jessie lite, this time we will now create the necessary cross compilation Software Development Kit (SDK). The SDK will contain the toolchain, headers and libraries to compile binaries compatible with the Raspberry Pi on a different (more powerful) machine (e.g. your laptop or desktop computer).

While the Raspberry Pi 2 and Raspberry Pi 3 are armv7 and aarch64 the old Raspberry Pi 1 and Raspberry Zero are armv6; to simplify matters Raspbian is compiled for armv6hf+vfpv2 to have a single distribution supporting all. The target’s triple is then arm-none-linux-gnueabihf.

We will therefore create a SDK targeting armv6 and the programs built with it should be compatible with all Raspberry Pis. The downside is that programs built with this toolchain will not be able to take advantage of advanced features only available in the Raspberry Pi 2 and 3.

It is instructive to build the SDK by hand to better understand the how everything fits together, and be able to make educated adjustments to the SDK’s toolchain, headers and libraries. For our SDK we will focus on clang from the llvm project as our underlying compiler for our toolchain as we will later use GHC’s LLVM cross compilation backend, and want to keep things simple and consistent. You can find a script to build the SDK in our zw3rk/scripts GitHub repository.

The SDK

What is a SDK and why do we need one? When compiling on a more powerful (or better supported) build machine to a different one, we need a toolchain that contains a compiler that can target the machine that will finally host the resulting binary. The toolchain will also contain a linker, to be able to link object files that are foreign to the build machine as well as an archiver, and a tool to extract symbols from those object files. In addition to the toolchain, we also need access to the headers and libraries on the host, against we want to link. Let us start out by creating a folder raspbian-sdk for the SDK, with two subfolders prebuilt (which will hold the toolchain) and sysroot (which will hold the headers and libraries).

mkdir -p raspbian-sdk/{prebuilt,sysroot}

The Compiler

As mentioned above, we’ll use clang from the llvm project as our compiler. You can obtain a copy from their release download website. At the time of writing LLVM 4.0.0 is the latest release. clang is a multi-target compiler, which means that the same compiler can produce object code for different architectures. This can be controlled via the --target flag.

LLVM can be unpacked and installed into raspbian-sdk/prebuilt as follows:

curl -L -O http://releases.llvm.org/4.0.0/clang+llvm-4.0.0-x86_64-apple-darwin.tar.xz

xz -d clang+llvm-4.0.0-x86_64-apple-darwin.tar.xz
tar xf clang+llvm-4.0.0-x86_64-apple-darwin.tar \
    -C "/path/to/raspbian-sdk/prebuilt" --strip-components=1

Note: This will install the clang+llvm for macOS, the commands for linux would be similar.

The Linker and Other Utilities

With the compiler in hand, we can now produce object files =.o=, but when combining multiple object files, these need to be linked together. To illustrate this, assume two object files a.o and b.o. Object file b.o refers to a function f in a.o. It does so via an external symbol reference (by name). The linker will, when combining a.o and b.o, look for the symbol with the specified name in a.o and resolve the reference.

GNU Binutils provide a suite of tools containing not only one linker (ld.bfd), but a second more modern one (ld.gold) as well. In addition to the linkers, binutils also contain an archiver (ar), a symbol listing tool (nm), and a few other tools. The latest version can be obtained from their ftp server, the latest version as of this writing is binutils-2.28.

As this is a source distribution, you will need to build binutils yourself.

./configure --prefix="/path/to/raspbian-sdk/prebuilt" \
            --target=arm-linux-gnueabihf \
            --enable-gold=yes \
            --enable-ld=yes \
            --enable-targets=arm-linux-gnueabihf \
            --enable-multilib \
            --enable-interwork \
            --disable-werror \
            --quiet
make && make install

This should configure, compile and install the suite of binutils into /path/to/raspbian-sdk/prebuilt. While binutils can be configured to be multi-target, I advise against doing so, as it requires passing the target to tools when there is ambiguity. This becomes more complicated in cases where the tools are called indirectly.

The Headers and Libraries

We finally need to get hold of a copy of the headers and libraries from the Raspberry Pi. We need those so that the compiler has access to the headers and libraries available on the Raspberry Pi. /This also means, that if you install new libraries (with their headers), you will need to refresh your SDK; make sure to install the/ -dev packages if needed.

To copy the headers and libraries from the Raspberry Pi to the build machine, we will use rsync. It is not available by default in raspbian jessie lite and needs to be installed:

ssh pi@raspberrypi 'sudo aptitude install -y rsync'

With that in place, we can start to copy the data to our build system:

Note: as Rob van den Bogaard kindly pointed out/, the/ rsync that comes with macOS Sierra 10.12 does not support multiple */remote sources/*/. Using the/ rsync installed via brew install rsync however does.

mkdir sysroot
rsync -rzLR --safe-links \
      pi@raspberrypi:/usr/lib/arm-linux-gnueabihf \
      pi@raspberrypi:/usr/lib/gcc/arm-linux-gnueabihf \
      pi@raspberrypi:/usr/include \
      pi@raspberrypi:/lib/arm-linux-gnueabihf \
      sysroot/

Running rsync at a later time again, has the benefit of copying over only the new data, to keep your local copy in sync with the one on the Raspberry Pi. You should run the rsync command in the sysroot folder again, if you installed additional libraries on your Raspberry Pi, against which you want to link.

Using the SDK

Assuming we now have the following structure:

├── prebuilt
│   ├── arm-linux-gnueabihf
│   ├── bin
│   ├── include
│   ├── lib
│   ├── libexec
│   └── share
└── sysroot
    ├── lib
    └── usr

(After unpacking clang+llvm into prebuilt and setting prebuilt as the target for =binutils=/)/

We can compile a simple hello.c with the following content

#include <stdio.h>

int
main(int argc, char ** argv) {
  printf("Hello World!\n");
  return 0;
}

via our cross compilation SDK for arm-linux-gnueabihf like so:

export COMPILER_PATH=sysroot/usr/lib/gcc/arm-linux-gnueabihf/4.9
./prebuilt/bin/clang --target=arm-linux-gnueabihf \
                     --sysroot=./sysroot \
                     -isysroot ./sysroot \
                     -L${COMPILER_PATH} \
                     --gcc-toolchain=./prebuilt/bin \
                     -o hello hello.c

Finally copy hello over to the Raspberry Pi

scp hello pi@raspberrypi:

and execute it

ssh pi@raspberrypi ./hello

Conclusion

We have seen that building and using a cross compilation SDK is not as simple as clang --target arm-linux-gnueabihf -o hello hello.c, but is still something manageable. The tricky part is getting all the compiler flags right.

Luckily we can hide all of them with a simple shell wrapper. Putting

#!/bin/bash
BASE=$(dirname $0)
SYSROOT="${BASE}/../../sysroot"
TARGET=arm-linux-gnueabihf
COMPILER_PATH="${SYSROOT}/usr/lib/gcc/${TARGET}/4.9"

exec env COMPILER_PATH="${COMPILER_PATH}" \
     "${BASE}/clang" --target=${TARGET} \
                     --sysroot="${SYSROOT}" \
                     -isysroot "${SYSROOT}" \
                     -L"${COMPILER_PATH}" \
                     --gcc-toolchain="${BASE}" \
                     "$@"

into prebuilt/bin/arm-linux-gnueabihf-clang and making it executable with

chmod +x prebuilt/bin/arm-linux-gnueabihf-clang

allows us to simply say

prebuilt/bin/arm-linux-gnueabihf-clang -o hello hello.c

to compile hello.c on our build machine, into an executable that can run on the host.

← → Top