electric-snow.net

So my rate of post­ing to this blog has dropped from once a year to… well… it's been al­most four years. But I can as­sure you, the re­ports of my death have been great­ly ex­ag­ger­at­ed (to quote Mark Twain). In any case, a few things have hap­pened.

First off — and this has prob­a­bly been the big­gest block­er — I fi­nal­ly fin­ished my PhD the­sis, woo! Sec­ond, there was (and is, sad­ly) this lit­tle thing called a glob­al pan­dem­ic, so get vac­ci­nat­ed if you haven't yet! While many have tak­en to bak­ing sour­dough bread in their free time, I didn't have much ex­tra time at all, but a gen­er­al­ly high­er stress lev­el. But Rust has re­freshed a lot of the fun in pro­gram­ming for me (the lan­guage is on­ly a tiny part of that, the great tools and in­cred­i­ble com­mu­ni­ty are far more im­por­tant), so I de­cid­ed I might as well write a bit about that for a while, in­stead of C++ with which I have more of a love-hate re­la­tion­ship.

Since things that make or ma­nip­u­late sound are par­tic­u­lar­ly mo­ti­vat­ing for me, I re­cent­ly ac­quired a Bela (well, ac­tu­al­ly the gi­gan­tic Bela+BEAST stack, but more on that in a lat­er post if ev­ery­thing goes ac­cord­ing to plan).

The Bela is a Cape (anal­o­gous to a Rasp­ber­ry Pi HAT) for the Bea­gle­Bone Black. But the Bela is more than just an ex­ten­sion board for a sin­gle-board com­put­er: it comes with a Xeno­mai re­al-time Lin­ux for ul­tra-low la­ten­cy. Yes, an ac­tu­al hard re­al-time op­er­at­ing sys­tem (RTOS)! Plus a web-based IDE and all kinds of bells and whis­tles. But the IDE is for C++ and I'd much rather write Rust now, so we're go­ing to ig­nore that com­plete­ly and cross-com­pile from our com­fort­able Win­dows (or mac­OS, or Lin­ux, if that's what you pre­fer) desk­top with rust-an­a­lyz­er-based IDE. This first post in what is planned to be a se­ries will deal with just that: how do you cross-com­pile for and run Rust code on an em­bed­ded Lin­ux sys­tem?

First things first, what kind of ma­chine is it? We could look up all the da­ta sheets… or just ask it! So let's log in.

ssh root@bela.local

Yep, pass­word-less root lo­gin via ssh. Sounds safe. But I don't plan on con­nect­ing it di­rect­ly to the web and there's noth­ing on there but the OS and open source sam­ples, so we should be ok for now (fa­mous last words). Now let's find out what we're deal­ing with.

root@bela:~# gcc -dumpmachine
arm-linux-gnueabihf

This is the tar­get triple, which tells us that we have an ARM pro­ces­sor, run­ning Lin­ux, in a GNU en­vi­ron­ment, us­ing the em­bed­ded ap­pli­ca­tion bi­na­ry in­ter­face with hard float. Let's al­so have a look at what tar­gets Rust sup­ports by run­ning.

rustc --print=target-list

This will print the long list of tar­gets that Rust sup­ports. In­clud­ing a whole bunch of arm<something>-unknown-linux-gnueabihf (the unknown part is the ven­dor, which for Lin­ux is typ­i­cal­ly unknown or none). Which <something> is right for the Bela? Let's dig in­to the hard­ware a lit­tle deep­er.

root@bela:~# lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
Model:                 2
Model name:            ARMv7 Processor rev 2 (v7l)
BogoMIPS:              995.32
Flags:                 half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpd32

The op­er­a­tive part be­ing armv7, since arm alone means AR­Mv6 for Rust/LLVM (as can be seen in the list of Tier 2 tar­gets with host tools). This is as good a time as any to ac­tu­al­ly in­stall the tar­get sup­port (just that it's sup­port­ed doesn't mean the sup­port is al­ready in­stalled). If you in­stalled Rust via rustup that's pret­ty easy:

rustup target add armv7-unknown-linux-gnueabihf

Let's get start­ed with a sim­ple ex­am­ple

cargo new --bin hello
cd hello
cargo build --target armv7-unknown-linux-gnueabihf

Drum­roll, please…

error: linker `cc` not found
  |
  = note: The system cannot find the file specified. (os error 2)

Well that's not what we were hop­ing for. Ok, we don't have cc on Win­dows, but that's the wrong link­er for cross com­pi­la­tion any­way. So we need an ARM-com­pat­i­ble cross-plat­form GCC to use as a link­er. There are two op­tions that worked for me:

• The old GCC 7.5 Linaro toolchain
• The cur­rent GCC 10.x of­fi­cial ARM toolchain

UP­DATE 2022-02-10: While writ­ing a lat­er is­sue of this se­ries, I dis­cov­ered that even the old Linaro GCC 7.5 com­pil­er ex­pects a new­er libc.so (GLIBC) than is avail­able on the Bela with the v0.3.8b firmware. By down­grad­ing to the even old­er Linaro GCC 6.3, I was able to get things up and run­ning again.

For mac­OS, you can check out the Bela project's own toolchain dis­tri­bu­tion. While oth­ers rec­om­mend us­ing Zig, that op­tion did not work for me, as ARM sup­port isn't quite there yet. Af­ter in­stalling the toolchain in a lo­ca­tion of our choice and adding it to the PATH, we can pro­ceed:

$env:CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_LINKER="arm-linux-gnueabihf-gcc"
cargo build --target armv7-unknown-linux-gnueabihf

It works (you may have to add a none in there de­pend­ing on the toolchain you chose)! But it is kin­da clum­sy hav­ing to pass all that by hand, isn't it? But there's a so­lu­tion, Car­go con­fig files! So let's add a sim­ple one to our project un­der .cargo/config.toml:

[build]
target = "armv7-unknown-linux-gnueabihf"

[target.armv7-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"

Now, we can just run cargo build with­out all the man­u­al pa­ram­e­ter pass­ing shenani­gans. It works! But I said we al­so want to run the code and guess what: cargo run, cargo test, and cargo bench won't work. No sur­prise there, since a Lin­ux ARM ex­e­cutable won't ex­act­ly work on a Win­dows x86_64 ma­chine. Sure, we could just use scp to copy it over, then ssh to run it, but then we lose a bunch of nice fea­tures, such as the afore­men­tioned cargo com­mands (and rust-an­a­lyz­er IDE equiv­a­lents). But Car­go (awe­some tool that it is) has a so­lu­tion for this, the runner op­tion which spec­i­fies a wrap­per script for run­ning ex­e­cuta­bles. Since I want­ed some­thing that works on both Win­dows and Lin­ux (and maybe mac­OS?), I wrote a Python script that up­loads the work­ing di­rec­to­ry via rsync (via WSL on Win­dows) to the tar­get, runs the ex­e­cutable via ssh, and fi­nal­ly copies back gen­er­at­ed files via rsync (use­ful for bench­mark re­ports etc.). The script is fair­ly com­plex, so I won't go in­to de­tail here and just present the fin­ished thing:

# no shebang-line as Python scripts aren't really executable on Windows
# use runner = ["python", "runner.py"] in your Cargo config instead

from hashlib import sha3_224
from shlex import quote
import os
import pathlib
import platform
import sys
import subprocess as sp

# "parse" command line
EXECUTABLE = sys.argv[1]
ARGUMENTS = sys.argv[2:]

# get relative part of executable path and convert to POSIX (as host may be Windows)
EXECUTABLE_RELATIVE = pathlib.Path(os.path.relpath(EXECUTABLE)).as_posix()

# create a (statistically) unique name for the remote working directory copy
WORKDIR = sha3_224((platform.node() + ':' + os.getcwd()).encode('utf-8')).hexdigest()

# the target hardware (Bela.io) has passwordless root login
# for normal systems you'll need to handle user authentication in a smarter manner
SSH_NONINTERACTIVE = ['ssh', '-qTo', 'BatchMode yes', 'root@bela.local']
SSH_INTERACTIVE = ['ssh', '-qt', 'root@bela.local']

# use rsync via WSL when on Windows
RSYNC = (['wsl'] if platform.system() == 'Windows' else []) + ['rsync']

# ensure base directory exists
sp.run(SSH_NONINTERACTIVE + [
    'mkdir', '-p', '.cargo_runner'
], stdout=sp.DEVNULL, stderr=sp.DEVNULL, check=True)

# synchronize working directory to remote
sp.run(RSYNC + [
    '-rlptz',
    # prevent syncing the .git folder
    '--exclude', '.git',
    # delete old files (otherwise they'll get copied back later)
    '--delete',
    '.',
    f'root@bela.local:.cargo_runner/{WORKDIR}/'
], stdout=sp.DEVNULL, stderr=sp.DEVNULL, check=True)

# run executable remotely, explicitly without checking, as partial results should still be copied
code = sp.run(SSH_INTERACTIVE + [
    f'cd .cargo_runner/{WORKDIR} && {quote(EXECUTABLE_RELATIVE)} {" ".join(map(quote, ARGUMENTS))}'
]).returncode

# synchronize working directory from remote (for Criterion reports etc.)
sp.run(RSYNC + [
    '-rlptz',
    f'root@bela.local:.cargo_runner/{WORKDIR}/',
    '.',
], stdout=sp.DEVNULL, stderr=sp.DEVNULL, check=True)

# exit with the code from the actual run
sys.exit(code)

Now we on­ly have to up­date our Car­go con­fig to add the run­ner:

[build]
target = "armv7-unknown-linux-gnueabihf"

[target.armv7-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"
rustflags = ["-Ctarget-cpu=cortex-a8", "-Ctarget-feature=+neon"]
runner = ["python", "runner.py"]

Ok, I added two more things that we'll want in the fu­ture: -Ctarget-cpu=cortex-a8 to op­ti­mize for the Bea­gle­Bone Black's CPU and -Ctarget-feature=+neon since the Bea­gle­Bone has Neon SIMD sup­port, which not ev­ery Cor­tex A8 has. In any case, let's try it!

PS D:\development\hello> cargo run
   Compiling hello v0.1.0 (D:\development\hello)
    Finished dev [unoptimized + debuginfo] target(s) in 0.80s
     Running `python runner.py target\armv7-unknown-linux-gnueabihf\debug\hello`
Hello, world!

Beau­ti­ful! Next time, I'll go in­to more de­tail on the project(s) I'm plan­ning to try on the Bela and some SIMD op­ti­miza­tion fun.

As there is no com­ment func­tion any­more since I switched to a stat­ic site gen­er­a­tor, feel free to fol­low me and send me a DM on Mastodon if you have any ques­tions or com­ments.