Cross-compiling for OpenWrt with various programming languages

I have been messing about with the Nim programming language quite a lot recently. Because it compiles to C code, this means that it's very good for programming lots of devices - since most everything has a C compiler. I have to say that I am drawn to Nim's main goal of being efficient, because I dislike bloat. However, Nim has some disadvantages, and for me the biggest is that it has not reached a v1 release yet. So the language is not as stable as it could be, and I get nervous each time a new version of the compiler is released, wondering if my code will still work.

In fairness, Nim is quite stable, I don't want to be misleading. But until it reaches a version 1 release I will continue to be cautious. I would not use Nim for important work (yet) - although I am using it for hobbyist type stuff. And I've been enjoying it too.

But, in recent weeks I thought that it would be only fair for me to try out some other languages. So I've been doing some experiments on my linux box, mostly inside docker containers. Cross-compiling for embedded linux is high on my personal list of requirements, so I have been trying to target the latest stable release of OpenWrt (15.05.1) running on the Ralink RT5350F chipset. This can be done very easily with Nim. So, I tried that with a few other languages to see how I got on. Thank goodness for docker, it has made this process so much easier and when I'm finished it's very easy to tidy up my mess!

1. Go

At first, I thought that it was going to be easy with Go. It looked as simple as this command:

~# GOOS=linux GOARCH=mipsle /usr/local/go/bin/go build -v hello.go

Which looked all-good at first glance:

~# file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), 
statically linked, not stripped

But when I ran it on my device, I just got this 'Illegal instruction' message:

~# /tmp/hello 
Illegal instruction
~# 

I then found several issues on the golang GitHub repo about MIPS32 support. So it looks like the easy way is not so easy after all. Anyway, I did eventually get a 'hello world' program cross compiled for OpenWrt with Go. And this time I was able to run the executable on my target device. In the end I followed the first set of these instructions. However, the resulting binary was 1.8 Mb just for the hello world program. At this point I decided to give up, and by that time I had read posts from other people saying that cross-compiling Go code for embedded devices resulted in binaries that were big and slow. Maybe in the future things will get better. I hope so.

2. Rust

It appears that Rust would be easier for cross-compiling to OpenWrt if I was using the trunk version of OpenWrt, because support for the target mipsel-unknown-linux-musl used by OpenWrt trunk is much better than mipsel-unknown-linux-uclibc which I need for the stable release (which I'm running). So I expect that this will become easier in time. In any case, this seems to work:

~# rustup target add mipsel-unknown-linux-musl
info: downloading component 'rust-std' for 'mipsel-unknown-linux-musl'
 15.3 MiB /  15.3 MiB (100 %)   1.5 MiB/s ETA:   0 s                
info: installing component 'rust-std' for 'mipsel-unknown-linux-musl'

Whereas this version (with uClibc) doesn't:

~# rustup target add mipsel-unknown-linux-uclibc
error: toolchain 'stable-x86_64-unknown-linux-gnu' does not contain 
component 'rust-std' for target 'mipsel-unknown-linux-uclibc'

So, maybe I should just try that again when I've moved onto a later version of OpenWrt (or one of the LEDE releases perhaps).

3. Vala

I found the Vala language by accident, but it is apparently used a lot by the Elementary OS project - which gives it some kudos in my book. Additionally, the syntax is very much like C#, which means it should be somewhat familiar to me. Like Nim, it also compiles to C so I thought that support for embedded linux should be good. However, it depends on some libraries (like GLib2) which I don't have available on my device. I could probably make this work if I wanted, but I had little interest in doing that. So I didn't proceed any further.

4. Crystal

I had a very brief look at Crystal which does have some support for cross-compiling. It looked interesting, although I found some of the syntax to be a bit strange in places. However I don't think it supports the target device I am looking for. But I did get this far, purely out of interest:

~# crystal build hello.cr --cross-compile --target "x86_64-unknown-linux-gnu"
cc hello.o -o hello  -rdynamic  -lpcre -lgc -lpthread 
/opt/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib ~#

I didn't go any further than that, but it was interesting to have a little play.

In summary...

It actually looks like Nim is a pretty sensible choice for what I'm doing. In comparison to other languages it's amazingly simple to set up for cross-compilation. So maybe I just need to put up with any minor instabilities in syntax that are likely to appear. But it is a good reminder that I need to try Nim on OpenWrt trunk (or one of the LEDE releases), because they've moved from uClibc to musl. I don't know if that will introduce any issues. I need to come back to that...

At least in the short term I am happy to keep on messing about with Nim. And I'm happy that I have at least tried some alternatives. I will try to keep my eye on Go and Rust, both of which showed signs of promise. When I have a device running OpenWrt trunk, or LEDE, then perhaps I will give Rust another try.

Running H2D2 cross platform

So one of my goals in developing DALIS/H2D2 was to make it possible to run a single instance of a program on multiple platforms. Since the H2D2 virtual machine can run code for a timeslice and then persist the entire state of the running program, it should be possible to put an executing H2D2 program into hibernation, move it to a different platform (ie a machine with an entirely different type of OS or processor) and then carry on running the same program. No matter what point the program was frozen at, the code should be able to carry on where it left off.

Well I've now gotten to the point where I can test that theory. The first experiment is to run a program on Windows for a few milliseconds and then complete the execution on Linux. To make this work I needed to make sure that all my data was persisted in sizes that are the same on different platforms, so I need to use int32_t instead of int, that type of thing. Since I've written my code in a cross-platform way and since I'm using data sizes that will be the same on different platforms everything should just work. So here we go, I'm running my mandelbrot program on Windows for 200ms:

...so that outputs a file called 'demo.hby' which is the H2D2 bytecode including its persisted state (all the program instructions, the call and data stacks and the values of all variables). Now I need to move that file to my Linux box and run the code from where it stopped. On the Linux machine I have already compiled the H2D2 virtual machine from source using GCC of course. Here goes:

Awesome! It works! I guess it's not much more than a neat trick at the moment, but I think it's an achievement of sorts. If you had some kind of long running process, it might be handy to be able to wake it, run it on whatever machine was available, and then put it back into hibernation. Okay, you can't start re-writing all your business logic in H2D2 just yet... but it's early days. This is why I always imagined DALIS / H2D2 to be a cloud based language, where you don't care what type of platform or processor is being used from one moment to the next.

So the next obvious experiment is to do the same thing, but on the Raspberry Pi... maybe I'll do it in reverse, by starting the program on the Raspberry Pi and then finishing it on Windows.