Cross-compiling for OpenWrt with various programming languages

I have been messing about with the Nim programming language quite a lot recently. Because it compiles to C code, this means that it's very good for programming lots of devices - since most everything has a C compiler. I have to say that I am drawn to Nim's main goal of being efficient, because I dislike bloat. However, Nim has some disadvantages, and for me the biggest is that it has not reached a v1 release yet. So the language is not as stable as it could be, and I get nervous each time a new version of the compiler is released, wondering if my code will still work.

In fairness, Nim is quite stable, I don't want to be misleading. But until it reaches a version 1 release I will continue to be cautious. I would not use Nim for important work (yet) - although I am using it for hobbyist type stuff. And I've been enjoying it too.

But, in recent weeks I thought that it would be only fair for me to try out some other languages. So I've been doing some experiments on my linux box, mostly inside docker containers. Cross-compiling for embedded linux is high on my personal list of requirements, so I have been trying to target the latest stable release of OpenWrt (15.05.1) running on the Ralink RT5350F chipset. This can be done very easily with Nim. So, I tried that with a few other languages to see how I got on. Thank goodness for docker, it has made this process so much easier and when I'm finished it's very easy to tidy up my mess!

1. Go

At first, I thought that it was going to be easy with Go. It looked as simple as this command:

~# GOOS=linux GOARCH=mipsle /usr/local/go/bin/go build -v hello.go

Which looked all-good at first glance:

~# file hello
hello: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), 
statically linked, not stripped

But when I ran it on my device, I just got this 'Illegal instruction' message:

~# /tmp/hello 
Illegal instruction
~# 

I then found several issues on the golang GitHub repo about MIPS32 support. So it looks like the easy way is not so easy after all. Anyway, I did eventually get a 'hello world' program cross compiled for OpenWrt with Go. And this time I was able to run the executable on my target device. In the end I followed the first set of these instructions. However, the resulting binary was 1.8 Mb just for the hello world program. At this point I decided to give up, and by that time I had read posts from other people saying that cross-compiling Go code for embedded devices resulted in binaries that were big and slow. Maybe in the future things will get better. I hope so.

2. Rust

It appears that Rust would be easier for cross-compiling to OpenWrt if I was using the trunk version of OpenWrt, because support for the target mipsel-unknown-linux-musl used by OpenWrt trunk is much better than mipsel-unknown-linux-uclibc which I need for the stable release (which I'm running). So I expect that this will become easier in time. In any case, this seems to work:

~# rustup target add mipsel-unknown-linux-musl
info: downloading component 'rust-std' for 'mipsel-unknown-linux-musl'
 15.3 MiB /  15.3 MiB (100 %)   1.5 MiB/s ETA:   0 s                
info: installing component 'rust-std' for 'mipsel-unknown-linux-musl'

Whereas this version (with uClibc) doesn't:

~# rustup target add mipsel-unknown-linux-uclibc
error: toolchain 'stable-x86_64-unknown-linux-gnu' does not contain 
component 'rust-std' for target 'mipsel-unknown-linux-uclibc'

So, maybe I should just try that again when I've moved onto a later version of OpenWrt (or one of the LEDE releases perhaps).

3. Vala

I found the Vala language by accident, but it is apparently used a lot by the Elementary OS project - which gives it some kudos in my book. Additionally, the syntax is very much like C#, which means it should be somewhat familiar to me. Like Nim, it also compiles to C so I thought that support for embedded linux should be good. However, it depends on some libraries (like GLib2) which I don't have available on my device. I could probably make this work if I wanted, but I had little interest in doing that. So I didn't proceed any further.

4. Crystal

I had a very brief look at Crystal which does have some support for cross-compiling. It looked interesting, although I found some of the syntax to be a bit strange in places. However I don't think it supports the target device I am looking for. But I did get this far, purely out of interest:

~# crystal build hello.cr --cross-compile --target "x86_64-unknown-linux-gnu"
cc hello.o -o hello  -rdynamic  -lpcre -lgc -lpthread 
/opt/crystal/src/ext/libcrystal.a -levent -lrt -ldl -L/usr/lib -L/usr/local/lib ~#

I didn't go any further than that, but it was interesting to have a little play.

In summary...

It actually looks like Nim is a pretty sensible choice for what I'm doing. In comparison to other languages it's amazingly simple to set up for cross-compilation. So maybe I just need to put up with any minor instabilities in syntax that are likely to appear. But it is a good reminder that I need to try Nim on OpenWrt trunk (or one of the LEDE releases), because they've moved from uClibc to musl. I don't know if that will introduce any issues. I need to come back to that...

At least in the short term I am happy to keep on messing about with Nim. And I'm happy that I have at least tried some alternatives. I will try to keep my eye on Go and Rust, both of which showed signs of promise. When I have a device running OpenWrt trunk, or LEDE, then perhaps I will give Rust another try.

H2D2: Something to do with Pi

After having success with moving running H2D2 programs between platforms, I wanted to try running H2D2 on the Raspberry Pi. I was pretty confident that it would work, having already compiled H2D2 from source on Debian on another machine. What I wanted to do this time was run some H2D2 code on the Raspberry Pi and then transplant the program to Windows and allow the code to run to completion. So here goes:

...well that seems to work OK. But that was pretty much expected. You can see that I ran the demo code for 500ms this time. But the real trick is to see if the file which has been generated is binary compatible with other platforms. If I move the output file to my Windows machine, will the H2D2 program continue exactly where it left off? Drum roll please.

Yay! After putting all that effort in it's nice to know that H2D2 bytecode is Raspberry Pi compatible, I hope that it goes a long way towards my efforts at crossplatformness. Of course, this also means that I have a means to fork a running program, since I could go back to the file that I copied off the Raspberry Pi and start again from that exact point as many times as I like.

...umm and I also need to write a different example program, all these mandelbrots are starting to get boring again. Trouble is, it is quite a good piece of test code. Anyway, I'm off to implement number arrays in H2D2 now.

DALIS Reloaded: H2D2

I've recently been thinking that my DALIS programming language may have gone as far as I'm prepared to take it in its current incarnation. It was only supposed to be a proof of concept, so I probably went way past that stage some time ago. But some of the most recent features have been hard to implement because of decisions (and shortcuts) which I made at the start. Also, I've learned more about programming language implementation now, so I'd like to go back and change some things.

And since I've had the Raspberry Pi I've been reminded how much I really like programming in C, so I've decided that this time round I'll do the language in C and try to make it portable. I suppose there's an outside chance it may fit on a small microprocessor like one of the AVR ATMegas that I have lying around.

So... I've started again from scratch, bringing with me the things I learned from the DALIS proof of concept. This time round I'm not sure if I'll stick to the same syntax as the original DALIS or not. In my head the codename for this new language is H2D2.

Because the re-write is in C I'm hopeful that it will be easier to port and that it will be somewhat faster - even if speed isn't one of my main aims. At some point you *know* I'm going to render the Mandelbrot set as a benchmark don't you... be warned :-)

But many of the original ideas from DALIS are coming with me, programs should be able to be paused and resumed, and a paused program should be capable of being serialised before being restarted. This should allow me to continue running H2D2 programs on the cloud.

This time I'm going to generate syntax trees - my current thinking is that the source code can generate a syntax tree, and what I'm really building is a Virtual Machine which is capable of executing the syntax tree directly. The virtual machine can save the running state (the stack and the value of all variables) after any instruction which could become a resume point. The original DALIS didn't use syntax trees, it interpreted one line of source code at a time and used recursive functions to parse expressions. My new approach means that running a program won't need to keep parsing the source code, which will hopefully be more efficient.

So I've started all this from an empty project in the Pelles C IDE. I've not used any non-standard libraries, and I've written my own implementations of things like the stack and the tree traversal algorithm. So far I've implemented instructions like IF, ELSE, ADD, SUBTRACT, MULTIPLY, as well as something that will print an integer on the screen (which will eventually become PRINT or WRITE or whatever I call it). I can't assign variables yet though (but constant number values can be coded in the H2D2 source).

So currently I'm building the parser to see if I can construct the syntax trees from little bits of H2D2 source code. It's looking positive, I have some simple programs running, evaluating expressions and printing results, that type of thing. I've even been able to save a running program to disk and then resume the same program afterwards, which is cool.

The next big step has to be assigning variables because then I'll be able to try some more interesting stuff...

A simple DALIS server

A while back I wrote a simple web server and said that I could turn it into a DALIS server that would run outside of IIS. Well that's exactly what I did, take a look at this YouTube clip I've made:

In this clip I'm showing two instances of the same program running (one in IE the other in Chrome). Each program shows a kindof bouncing ball thingy in ASCII Art. This is not the kind of thing that I designed DALIS to do of course, but who cares. In the background you can see the DALIS server runnng as a console app. I've made it so that each thread servicing a web request shows up like a blinking light. This means you can see it is working - and you can tell that I still like blinking LEDs on things. Ahh, do you remember the days when you used to see blinky LEDs on the front of your modem? Good times, eh?

So anyway... whilst I have not said much about the DALIS progaming language recently, development has continued on background threads :-) I have added a few more features to the language. I have also tried to think of a better name for it, but DALIS has kinda stuck now. One day I might have the guts to tell everyone what DALIS stands for, or maybe not.

DALIS on the Fez Panda II

Now that I have DALIS running in the .Net Micro Framework, I decided that I should try it on some real hardware, so I bought a Fez Panda II to try it out on. Currently this only supports .Net Micro Framework v4.1, so I had to tweak a few things, but not much.

So here is a screenshot of it running in the Visual Studio debugger:

That's running on the actual hardware, with the output set to come out on the debug console. I'm still testing with the same temperature converter DALIS program. But it works, which is the main thing. The next task is to get the output appearing on the Fez Touch colour screen that I also bought. The end result might be to run DALIS programs from the SD card... or to download them from the web and always run the latest code.

Improvements to DALIS

After working on some improvements to my DALIS programming language, I decided to celebrate by actually taking the time to understand the algorithm that draws the mandelbrot set better. After some reading on wikipedia, I was able to refactor it somewhat. So, this is my current implementation:

LOOP d=0, h=0, e=1.1
   LOOP b=-2
      LOOP n=0, h=127, r=0
         r = r*r-n*n+b
         n = 2*d*n+e
      REPEAT d=r, h=h-1 IF ((r*r)+(n*n) < 4) AND (h > 32)
      WRITE (h)
   REPEAT b=b+0.04 IF b < 1
   WRITE RETURN
REPEAT e=e-0.1 IF e > -1.2

I think it's much better. As you can see, it's now possible for several variables to be assigned on the same line and there is a RETURN keyword to write line breaks easily. Drawing the mandelbrot set in ten lines of code is pretty good, surely?

At the moment I'm working on a means of getting user input, which is going well, and in the background I'm thinking of ways to build simple webservices with DALIS... At this rate I'll have to start writing a book entitled "The DALIS programming language" :-)

DALIS - experiments on other platforms

I decided to give my new (experimental) programming language, codenamed DALIS a try on some other platforms.  This one is obvious I guess - the good old iphone:

DALIS on iphone

...but I was pleasantly surprised to see it working OK on my PSP as well: 

DALIS on PSP

The fonts on the PSP don't come out the same, but it seems to run OK, which is the main thing.  I would be fun to try a Kindle, but I don't have one (yet).  The advantage of a cloud-based language like this is that you only need a browser, so it should work on all kinds of things.

DALIS - proof of concept

For the last week I've been writing an experimental cloud-based programming language in the evenings (as you do).  The language has just reached the point where it can actually do something... So I’ve ported my ASCII Mandelbrot drawing program from C, take look at these screenshots:

DALIS code view

That's the source code, and ...

The output of the program

... that's the output it gives. 

Not bad, I think.  The language syntax needs lots more work, as does the parser, but it’s good enough to prove *something*.  I’ve set it to do timeslices; the code runs for so many milliseconds (whilst rendering the page on the webserver) and then returns the results, with the program state hidden inside the html.  The next page request picks up where it left off, so you can actually watch the Mandelbrot as it is being drawn.  It’s not fast, but it does actually work.  I’m sure I’ll optimise it.  Well it’s an idea ... I'm still not sure if it’s a good idea though!  But it has been fun, so I'll spend a bit more time on it at least.