Cross-compiling for OpenWrt with various programming languages

I have been messing about with the Nim programming language quite a lot recently. Because it compiles to C code, this means that it's very good for programming lots of devices - since most everything has a C compiler. I have to say that I am drawn to Nim's main goal of being efficient, because I dislike bloat. However, Nim has some disadvantages, and for me the biggest is that it has not reached a v1 release yet. So the language is not as stable as it could be, and I get nervous each time a new version of the compiler is released, wondering if my code will still work.

In fairness, Nim is quite stable, I don't want to be misleading. But until it reaches a version 1 release I will continue to be cautious. I would not use Nim for important work (yet) - although I am using it for hobbyist type stuff. And I've been enjoying it too.

But, in recent weeks I thought that it would be only fair for me to try out some other languages. So I've been doing some experiments on my linux box, mostly inside docker containers. Cross-compiling for embedded linux is high on my personal list of requirements, so I have been trying to target the latest stable release of OpenWrt (15.05.1) running on the Ralink RT5350F chipset. This can be done very easily with Nim. So, I tried that with a few other languages to see how I got on. Thank goodness for docker, it has made this process so much easier and when I'm finished it's very easy to tidy up my mess!

1. Go

At first, I thought that it was going to be easy with Go. It looked as simple as this command:

~# GOOS=linux GOARCH=mipsle /usr/local/go/bin/go build -v hello.go

Which looked all-good at first glance:

~# file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), 
statically linked, not stripped

But when I ran it on my device, I just got this 'Illegal instruction' message:

~# /tmp/hello 
Illegal instruction
~# 

I then found several issues on the golang GitHub repo about MIPS32 support. So it looks like the easy way is not so easy after all. Anyway, I did eventually get a 'hello world' program cross compiled for OpenWrt with Go. And this time I was able to run the executable on my target device. In the end I followed the first set of these instructions. However, the resulting binary was 1.8 Mb just for the hello world program. At this point I decided to give up, and by that time I had read posts from other people saying that cross-compiling Go code for embedded devices resulted in binaries that were big and slow. Maybe in the future things will get better. I hope so.

2. Rust

It appears that Rust would be easier for cross-compiling to OpenWrt if I was using the trunk version of OpenWrt, because support for the target mipsel-unknown-linux-musl used by OpenWrt trunk is much better than mipsel-unknown-linux-uclibc which I need for the stable release (which I'm running). So I expect that this will become easier in time. In any case, this seems to work:

~# rustup target add mipsel-unknown-linux-musl
info: downloading component 'rust-std' for 'mipsel-unknown-linux-musl'
 15.3 MiB /  15.3 MiB (100 %)   1.5 MiB/s ETA:   0 s                
info: installing component 'rust-std' for 'mipsel-unknown-linux-musl'

Whereas this version (with uClibc) doesn't:

~# rustup target add mipsel-unknown-linux-uclibc
error: toolchain 'stable-x86_64-unknown-linux-gnu' does not contain 
component 'rust-std' for target 'mipsel-unknown-linux-uclibc'

So, maybe I should just try that again when I've moved onto a later version of OpenWrt (or one of the LEDE releases perhaps).

3. Vala

I found the Vala language by accident, but it is apparently used a lot by the Elementary OS project - which gives it some kudos in my book. Additionally, the syntax is very much like C#, which means it should be somewhat familiar to me. Like Nim, it also compiles to C so I thought that support for embedded linux should be good. However, it depends on some libraries (like GLib2) which I don't have available on my device. I could probably make this work if I wanted, but I had little interest in doing that. So I didn't proceed any further.

4. Crystal

I had a very brief look at Crystal which does have some support for cross-compiling. It looked interesting, although I found some of the syntax to be a bit strange in places. However I don't think it supports the target device I am looking for. But I did get this far, purely out of interest:

~# crystal build hello.cr --cross-compile --target "x86_64-unknown-linux-gnu"
cc hello.o -o hello  -rdynamic  -lpcre -lgc -lpthread 
/opt/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib ~#

I didn't go any further than that, but it was interesting to have a little play.

In summary...

It actually looks like Nim is a pretty sensible choice for what I'm doing. In comparison to other languages it's amazingly simple to set up for cross-compilation. So maybe I just need to put up with any minor instabilities in syntax that are likely to appear. But it is a good reminder that I need to try Nim on OpenWrt trunk (or one of the LEDE releases), because they've moved from uClibc to musl. I don't know if that will introduce any issues. I need to come back to that...

At least in the short term I am happy to keep on messing about with Nim. And I'm happy that I have at least tried some alternatives. I will try to keep my eye on Go and Rust, both of which showed signs of promise. When I have a device running OpenWrt trunk, or LEDE, then perhaps I will give Rust another try.

Another HooToo HT-TM02

A while back, my wife noticed that the HooToo HT-TM02 was on a special deal at Amazon. So she bought me another one. This is the second one I have. The one I got before is in daily use as an internet radio and is doing a fantastic job. But the problem is that I don't want to mess about with it, because I might break my radio! So this *second* one gives me something to actually play about with...

As before, I immediately installed OpenWrt on it. The installation procedure involves adding some extra storage which gives some additional space during the install process. So you need to plug in a correctly formatted USB thumbdrive. The instructions advise you to check whatever USB storage you have added is working before proceeding. But when I did it this time I forgot to check, so there was a slightly tense moment! Anyway it must have been fine, because the install went without a hitch. Phew.

Then I upgraded to the latest released version of OpenWrt (which was Chaos Calmer 15.05.1) and reset it to the default configuration. I like to work from a clean starting point. So this gives me a good hackable little linux toy to play with. I expect that most of my effort will be continue to be programmed in Nim and cross-compiled.

So here are my notes on configuring the basics when setting up a HooToo HT-TM02 with OpenWrt. This should make it easy to repeat if I need to do a factory reset. It assumes that there is a home router with an address of 192.168.1.1, so works around that. Obviously, it assumes that OpenWrt has already been installed on the HooToo. The aim is to have the HooToo connecting to an existing wifi network, so you can get to it wirelessly, and it can download packages and updates. But also, a network cable still works if required. Here goes:

  1. Start with OpenWrt defaults. Connect with a network cable and get an IP address from DHCP
  2. Don't connect to any other networks, because it will clash with the router
  3. Now point a browser to 192.168.1.1
  4. Log-in without a password and then set a new password straight away
  5. Edit the LAN address of the HooToo to 192.168.0.1 but leave DHCP on
  6. Save and then reboot the HooToo, it now should not clash with your router
  7. After the reboot, unplug and re-plug the network cable, just to be sure
  8. Point your browser to the new address, 192.168.0.1 and log in again
  9. Now, scan for your own wifi and click "Join Network"
  10. Enter your wifi passphrase
  11. Assign the wifi connection to the firewall zone "LAN", but otherwise, use the defaults and save
  12. Check the HooToo has an internet connection (eg go to: System->Software and click "Update Lists")

Connecting via a cable should now give a DHCP address in the 192.168.0.x range, so using a cable is also possible in case wifi is not an option. For me, this is a pretty good starting point; now the fun stuff begins.

First steps into the world of Nim

I was recently doing a bit of reading up on the Rust programming language, but a stray comment somewhere about the Nim programming language sent me off on a bit of a tangent. The thing that really got me interested in Nim was that it compiles to C, and I noticed that this brings quite some options for portability. One of the reasons why I like programming in C is that you can run the code on all kinds of machines, from tiny embedded devices to supercomputers. But that does come at a cost, because it takes longer and you often have lots more typing to do because of all that boilerpate stuff.

But it looks like Nim would allow you to be quite productive, whilst the resulting executables should still be efficient and fast. To get some ideas, I went off to Rosetta Code and found the Nim code for a simple webserver which looks like this:

import asynchttpserver, asyncdispatch
 
proc cb(req: Request) {.async.} =
  await req.respond(Http200, "Hello, World!")
 
asyncCheck newAsyncHttpServer().serve(Port(8080), cb)
runForever()

Being able to create a webserver with just a few lines of code, and with the potential for the code to be portable and run on all kinds of devices seemed very tempting! By this point, I'd forgotten about Rust and wanted to explore Nim a bit further.

So whilst Nim has not reached v1.0 yet, it certainly looked very compelling. The compiler documentation made it look like cross-compilation was not that difficult, so I decided to try and cross-compile some code for a router running OpenWrt. Before going off and spendng a lot of time learning a new language, I wanted to see that I *really* can write portable code. I was very happy to see that in just a few minutes I had the example webserver code running on my TP-Link WR740N, as shown here:

To my amazement and joy, this really wasn't that hard. After installing Nim, I had to edit Nim/config/nim.cfg to point to my cross-compile toolchain, so I ended up with these lines:

mips.linux.gcc.exe =
 "/home/openwrt/openwrt/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/
bin/mips-openwrt-linux-gcc"
mips.linux.gcc.linkerexe =
 "/home/openwrt/openwrt/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/
bin/mips-openwrt-linux-gcc"

Then, after doing that I simply needed to pass the correct parameters to the Nim compiler, like this:

nim c --cpu:mips --os:linux webserver.nim

Which cross-compiled the example webserver ready for it to run on my TP-Link WR740N. That actually seems pretty awesome. I think that I need to learn more, perhaps I'll even go and buy the Nim in Action book. All this stuff worked perfectly without any trouble, which is pretty rare, so I am left feeling very impressed so far.

Radio streaming with OpenWrt

I have been doing a bit more playing around with USB sound cards and OpenWrt. I thought that a pretty good use for my HooToo HT-TM02 would be as an internet radio player. So I have been experimenting. I installed the following packages (I already had USB support working):

  • kmod-usb-audio
  • madplay
  • alsa-utils

Which enables me to play internet radio streams from the command line. I have been trying the stations listed on www.intenet-radio.com. If the station shows a PLS link, I found that you can download it as a playlist-file and extract the URL from inside using a text editor. In many cases, this seems to work very well (but not all streams worked). Then just use this command on OpenWrt and replace the URL:

wget -O - [URL] | madplay - -a-30 -o wave:- | aplay

Which will play the stream (I'm dropping the volume with -30 in that example). In reality, I've been using all the commands in quiet mode and in the background, like this:

wget -q -O - [URL] | madplay -Q - -a-30 -o wave:- | aplay -q &

...which just plays the stream in the background without any other output to the console. In addition, I also found the BBC radio station streams listed on this website which is pretty useful (and also the French station, Fip for good measure):

  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio1_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio1xtra_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio2_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio3_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio4fm_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio4lw_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio4extra_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_radio5live_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_6music_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_asianet_mf_p
  • http://bbcmedia.ic.llnwd.net/stream/bbcmedia_nangaidheal_mf_p
  • http://audio.scdn.arkena.com/11016/fip-midfi128.mp3

A better way to set the volume from the command line, is to use the amixer command, I'm using something like this:

amixer sset Headphone 50%

Although the more fancy alsamixer command seems to work fine as well.

After using that for a while, I think it's a very good use for the HooToo, so I am building a simple front end and will keep it as a miniature internet radio player.

Hootoo + OpenWrt + Velleman K8055N

One of the things that I had kicking around at home was a Velleman K8055N (actually mine was the pre-made version, the the VM110N) - but I think they are essentialy the same thing. I decided that it would be really cool to try it out on OpenWrt ... so that eventually I could try interfacing a tiny embedded Linux machine with the outside world.

Some quick googling led me to this driver which I tried on a Debian Virtual Machine first, and then decided to run it on my Hootoo HT-TM02. I used a docker container to build the kernel module according to the instructions, and then copied it over onto the device for installation.

The first attempt at installing the module gave the following error:

Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for
 kmod-k8055d:
 * 	kernel (= 3.18.23-1-e2416fa0abee73ea34d947db4520f4e9) *

Which I assumed was just OpenWrt being a bit over cautious, because I was actually running kernel v3.18.23. So I took the risk and overrode the error, with this command:

opkg install /tmp/kmod-k8055d_0.2.1_3.18.23-ramips-1_ramips_24kec.ipk --nodeps

Perfect, it worked! So the K8055N is now accessible in the file system under /proc. To celebrate, I hooked up the digital-to-analoge converter to an old voltmeter, so I could watch the needle bounce around in response to what I'm typing on the command line:

I think that's pretty awesome, although I don't know what I'll use it for. Perhaps I'll build some kind of Web API and allow inputs and outputs to be controlled from a browser. Or ... come to think of it, I could build a really cool CPU meter with the output showing via the needle of the voltmeter.

Hacking your Hootoo

For my wedding anniversary this year, my wife bought me a Hootoo HT-TM02 (amongst other things). How nerdy is that? But it's a nice little machine which can have the default firmware replaced with OpenWrt. For about USD $16.00 it's a very cheap way to have an embedded Linux machine. Personally, I didn't even try the standard firmware and put OpenWrt on straight away. It was pretty easy, but you need a USB stick to give the device some extra storage during the initial upgrade process. The instructions are here and the process was very smooth.

As soon as I had the initial OpenWrt installation, I decided to switch to trunk OpenWrt, so I used the sysupgrade command to install the 15.05 trunk image from here. So this left me with a device running trunk OpenWrt. I then held the reset button in (whilst the device was powered on) for 30 seconds to make sure that the configs were all at default settings.

Next, I decided to install relayd to bridge the LAN (ethernet) port with my existing Wi-Fi network. I followed the instructions here and that seemed to work fine. I felt that I had followed those instructions exactly; but the OpenWrt firewall was still getting in the way. So I just went ahead and disabled the firewall by using:

/etc/init.d/firewall disable

After doing that, I can switch off the Wi-Fi on my laptop, connect a network cable between the Hootoo and the laptop and I'm on my network. The Hootoo is acting as a bridge between my own Wi-Fi and its own ethernet socket, very nice!

The next thing I need to do is cross-compile some code and get it running on the Hootoo.

More Docker as a cross-compiler

Now that I work in the Silicon Shed, I decided that I needed to have some more Linux in there ... and restore some balance to the universe. So I thought that a good place to start would be with OpenWrt running on my router. I already had a TP-Link WR740n lying around which I had been playing with before. So I went and got the latest version of OpenWrt (which was Chaos Calmer, 15.05) and installed that. Cool! Now I have a nice little router with plenty of features to play with, and it's an embedded Linux box as well.

But having gotten that far, I decided to set up a new cross-compile environment. I had done this before using Virtual Machines, but this time I wanted to use Docker to make it easier to compile my C programs using whatever machine I happen to be sitting in front of. I started by creating a base Docker image with all the files in place to give me a buildroot. In my Dockerfile I used Debian Jessie as the starting point and then added all the files by cloning them from OpenWrt's git repository. That generic image can be found here at davidsblog/openwrt-build-15-05. It's an automated build with the source files coming from GitHub. I also wanted to do the same with the next stage - actually compiling OpenWrt from sources ... but when I tried, DockerHub timed out the build. The OpenWrt build process can take a few hours. So the rest could not be done as an automated build.

I used my base image to create a .config file for the WR740n using the make menuconfig command and then copied that .config file and referenced it in my next Dockerfile. This new Dockerfile takes the base image, adds the config file and then calls make to actually build my specific cross-compile environment for the WR740n. If somebody wanted to make a cross-compiler for a different device they would just need to change the config file for their own device and use docker build to create an image.

So I built the image and pushed it out to DockerHub as davidsblog/openwrt-build-wr740n. As long as you have the bandwidth, it's much easier to be able to pull a pre-configured cross-compiler than to set one up from scratch. And it's really easy to use.

This is how I'm using it: I created a script called /usr/local/bin/740ncc which contains this:

#!/bin/bash
docker run --rm -v ${PWD}:/src davidsblog/openwrt-wr740n:latest \
     /bin/sh -c "cd /src; $*"

So then, on my local machine, I navigate to the folder containing the C sources I want to cross-compile. Now I can type something like 740ncc make and the make command will be routed to a docker container which will do the cross-compilation for the WR740n. The compiled program will be on your local machine (not in the Docker container) just as if you had compiled it locally. I think that's very not-bad. I am also using Dockers --rm parameter so that the container is automatically removed afterwards. Here's an example where I'm building my rCPU monitoring webserver for the TP-Link WR740n:

I also discovered something interesting during all this: using the find command inside the same Docker image but on different machines does not always show the results in the same order. This had me puzzling for a while when I was using the find command in one of my scripts. I used the image on Ubuntu and the order of the find results was different to the same image running on my laptop on Elementary OS. In my experience, on the same machine the order of results from find is the same. I was expecting it to be the same for a container too, but obviously you can't rely on that. Interesting.

Recovering a slightly broken Vocore

OK, I managed to mess up the network settings on my Vocore. Oh well, it happens. It's not too bad... the device was still booting up and running... but was not accessible by Wi-Fi or by plugging in a network cable. So it's back to a good old serial connection. This is what I have done:

I've used a USB to Serial 3.3v TTL cable that I already had, and then just poked some jumper wires into the right places. Surprisingly it seems good enough. Anyway, in case I need to do this again, I thought that it would be worth taking a photo. I'm using PuTTY on a Windows box as the terminal. The settings I'm using are:

  • Speed: 57600
  • Data bits: 8
  • Stop bits: 1
  • Parity: None
  • Flow control: XON/XOFF

I connected the USB-to-serial cable first, then opened the connection using PuTTY. You then have a blank terminal screen. Then I connect the power to the Vocore, and you get to see all the logging information as the device boots. After a couple of minutes I press enter in the terminal and end up with the usual OpenWrt startup screen. After that you can use the console like normal. Anyway, it's handy to have that written down somewhere, because I'll probably break it again at some point. But I can report that I was able to fix the network config and things are back to normal now. Phew.

Vocore: mounting USB drives

So my next piece of work on the Vocore was to mount the filesystem from a USB stick. I found some instructions here. So I started adding all the necessary kernel modules, hoping that it would be as simple as getting the USB soundcard to work. It wasn’t quite that easy. I thought that I had done everything, but just got strange messages when I tried to mount the disk. Then I found this helpful link.

It described exactly what I was seeing. In short, you can’t rely on the error messages being returned by mount on OpenWrt. So to diagnose the problem, the most important thing was using the logread command to get some ideas why the USB drive was not mounting. In my case (I was mounting a vfat formatted thumbdrive) I needed to add support for Codepage 437 and ISO 8859–1 into the kernel. So when building OpenWrt, I needed to go into Kernel Modules -> Native Language Support, like this:

But both those character set problems were quite clear from the log entries, so working out what I needed to do wasn’t that difficult … once I knew that I should not trust the error messages being returned by mount.

And I did find some more detailed information about filesystems and character set problems here … albeit for a different type of device, but I think the principle is the same.

Anyway, as soon as I added the extra modules for the correct character sets then everything came to life and I could mount the USB disk. Happy days. So now I can play audio files from USB storage, which works great.

This is a journey into sound...

So I decided to try and get some sound out of my Vocore tiny linux box. It doesn’t come with any audio support, so I decided to add a USB sound card. I bought one of these, which I knew worked on the Raspberry Pi, so I knew had Linux support. They’re also pretty small:

I simply followed these instructions describing how to add USB Audio support to OpenWrt. At first, I chose to install sox as the player (I originally intended to use Madplay, but I don’t think it is supported on current versions of OpenWrt anymore).

I did not have much success playing mp3 files straight away, I kept getting sox WARN alsa: under-run messages when I tried to use the following command:

sox myfile.mp3 -d

Although sound was coming out … in chunks, so it was a partial success. But when I added a “-G” parameter to use temporary files (supposed to guard against clipping) like this:

sox myfile.mp3 -d -G

…then it worked fine for some lower quality mp3 files, but with a considerable delay before playing anything out the speaker. If I tried to play some better quality mp3 files then it was still stuttery. So then I started to experiment with reducing the sound quality, which sox can do easily. So I tried this:

sox myfile.mp3 -r 8000 -e unsigned -b 8 -c 1 test.raw

…which results in a raw file that can be played with the aplay command that comes with the ALSA soundcard drivers (specifically, the command above gives mono unsigned 8 bit with a rate of 8000 Hz). Whilst this results in a poor quality file, I found that the Vocore would happily play these raw files. In some ways it was kinda cool to hear my original mp3 file playing with a hiss in the background. But then I decided to see what happens if I create a file in “CD” quality, so I tried this:

sox myfile.mp3 -r 44100 -e signed -b 16 -c 2 test.raw trim 0 00:30

Note, that to save space on the Vocore I used the trim parameter to just convert the first 30 seconds. Playing that “CD” quality file on the Vocore also worked perfectly, with this command:

aplay test.raw -f cd

Passing the parameter -f cd means 16 bit little endian, 44100 Hz, stereo and is the same as passing -f S16_LE -c2 -r44100. Here’s a screenshot of a CD quality raw file playing on the Vocore with the aplay command.

So… what I’m doing now is using sox on another Linux box to convert the mp3 files and then just using aplay on the Vocore to play them. It should mean that sox is not actually required on my Vocore. It also means that playback is immediate and the quality is good. I will need to find a means to store the files, because they’re much bigger than mp3, but that’s a problem for another day… I’ll probably see if I can mount a USB stick or something.