First steps into the world of Nim

I was recently doing a bit of reading up on the Rust programming language, but a stray comment somewhere about the Nim programming language sent me off on a bit of a tangent. The thing that really got me interested in Nim was that it compiles to C, and I noticed that this brings quite some options for portability. One of the reasons why I like programming in C is that you can run the code on all kinds of machines, from tiny embedded devices to supercomputers. But that does come at a cost, because it takes longer and you often have lots more typing to do because of all that boilerpate stuff.

But it looks like Nim would allow you to be quite productive, whilst the resulting executables should still be efficient and fast. To get some ideas, I went off to Rosetta Code and found the Nim code for a simple webserver which looks like this:

import asynchttpserver, asyncdispatch
 
proc cb(req: Request) {.async.} =
  await req.respond(Http200, "Hello, World!")
 
asyncCheck newAsyncHttpServer().serve(Port(8080), cb)
runForever()

Being able to create a webserver with just a few lines of code, and with the potential for the code to be portable and run on all kinds of devices seemed very tempting! By this point, I'd forgotten about Rust and wanted to explore Nim a bit further.

So whilst Nim has not reached v1.0 yet, it certainly looked very compelling. The compiler documentation made it look like cross-compilation was not that difficult, so I decided to try and cross-compile some code for a router running OpenWrt. Before going off and spendng a lot of time learning a new language, I wanted to see that I *really* can write portable code. I was very happy to see that in just a few minutes I had the example webserver code running on my TP-Link WR740N, as shown here:

To my amazement and joy, this really wasn't that hard. After installing Nim, I had to edit Nim/config/nim.cfg to point to my cross-compile toolchain, so I ended up with these lines:

mips.linux.gcc.exe =
 "/home/openwrt/openwrt/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/
bin/mips-openwrt-linux-gcc"
mips.linux.gcc.linkerexe =
 "/home/openwrt/openwrt/staging_dir/toolchain-mips_34kc_gcc-4.8-linaro_uClibc-0.9.33.2/
bin/mips-openwrt-linux-gcc"

Then, after doing that I simply needed to pass the correct parameters to the Nim compiler, like this:

nim c --cpu:mips --os:linux webserver.nim

Which cross-compiled the example webserver ready for it to run on my TP-Link WR740N. That actually seems pretty awesome. I think that I need to learn more, perhaps I'll even go and buy the Nim in Action book. All this stuff worked perfectly without any trouble, which is pretty rare, so I am left feeling very impressed so far.

Drawing spheres with H2D2

Since I wrote this blog post, I have been looking for a better demo program to show what I can do with my H2D2 programming language. I needed something more interesting than "hello world" ... and which didn't involve drawing the Mandelbrot set.

I started trawling through the Rosetta Code website for inspiration and came across this sphere drawing program in C. I guess it's a simple ray tracing algorithm really. It seemed like a reasonable candidate.

So I set about porting it from C to H2D2, which was an interesting experience. Interesting because H2D2 is not very forgiving when you make mistakes - since I have not (yet) made the compiler error messages very user friendly. So it makes you reason things out very carefully in your head, rather than rely on the compiler to spot problems with your code.

I have managed to get it to work though, and the resulting H2D2 example program shows a few of the language features (like a function call and arrays for example). However, it also highlights a few things I should work on, like creating a syntactic shortcut for assigning values to an array. Anyway, I can now create output like this:

             !!...!!!***Bo               
         ..............!!**Boo           
      ..................!!**BBooe        
    .....................!!**BBooee      
   .......................!!**BBooee     
  ........................!!**BBoooee    
 .........................!!**BBoooeee   
!........................!!***BBoooeeee  
........................!!***BBooooeeee  
!......................!!***BBBoooeeeee  
!....................!!!***BBBoooeeeeeo  
*!.................!!!***BBBBoooeeeeeeo  
B*!!!...........!!!!****BBBooooeeeeeeoB  
 B***!!!!!!!!!!!*****BBBBoooooeeeeeeoo   
  oBB*************BBBBBoooooeeeeeeeoo    
   ooBBBBBBBBBBBBBBooooooeeeeeeeeooB     
    eoooooooooooooooooeeeeeeeeeooo*      
      eeeooooooooeeeeeeeeeeeeooB*        
         eeeeeeeeeeeeeeeeeoooB           
             oeeeeeeooooBB

But to show it in action, I made this recording, which shows it off in realtime in the browser:

Anyway, I am really pleased that it works so well. I can go back to fiddling around with the language now.

Hello World, H2D2 style

Now that my experimental programming language, H2D2, includes the all important server part, I thought that I'd better record an example of the traditional 'Hello World' program:

Although these programs are very trivial, it shows that I can now do H2D2 programming from my browser. It's the server part which compiles the source into bytecode and then executes the program in timeslices. So the actual code is running inside the server and we're just seeing the output in the browser (alongside the source code of course). Since I recorded this in a coffee shop before work this morning, the H2D2 server was running locally inside Xcode on my Mac, for convenience. But I would be able to do the same thing even if the server was running elsewhere, like on my TP-Link router.

Considering that the whole H2D2 language is written in C, I think that's pretty good going. It seems reasonably fast and I don't even think I have the compiler optimisations turned on. Obviously there is a little bit of JavaScript doing the HTTP posts between the browser and the server. But I really do need to write some better code examples, which should be things that don't involve the Mandelbrot set for a change...

UPDATE: one of the "better code examples" I mention above can now be seen here.

The joy of tests

Since my H2D2 project has quite a lot of code in it now, I've been using .Net interop as a quick way to write some unit tests for the H2D2 programming language.

I have written a C# unit test project in Visual Studio which tests much of the existing H2D2 language syntax. This means that I can add new features and quickly check that I've not broken any of the existing code. So I can very easily make sure that programs which used to run continue to work as expected.

Recently, I've been making H2D2 support function calls, and so it's been handy to write tests for various different types of function call (passing parameters, reading return values, allowing functions that don't return a value, etc). The unit tests have made it much easier to ensure that everything works after I've made changes.

Since I've written more unit tests in C# than anything else, I thought it would be the quickest way to get my tests up and running. And because I had already compiled H2D2 into a windows dll (the unmanaged kind), it is not very hard to run H2D2 programs from .Net - even easier because I had already written some basic wrapper code for all the interop stuff. It's not much harder than putting some H2D2 source code into a string and calling a Run() method. Obviously all this has been done 'just for fun', since it's only one of my hobbyist projects anyway.

But when I mentioned to one of my colleagues that I had spent some of my free time writing unit tests for fun he said that I was clearly stark raving mad. There's a possibility he's correct.

However, when I think of how I'd feel if I broke some of my existing H2D2 code and had to go back and unpick where I went wrong, I reckon that these unit tests are really less about having fun and more about preventing a severe loss of fun if I ended up scratching my head and trying to figure out what I'd done to make something break.

So I hope that's a suitable explanation for why I've been writing unit tests in my free time. It's the best excuse I've got.

Although ultimately, I may end up looking for a unit testing framework for C so that I only need one IDE loaded to run my tests... The .Net interop solution may just be a stepping stone.

Running H2D2 cross platform

So one of my goals in developing DALIS/H2D2 was to make it possible to run a single instance of a program on multiple platforms. Since the H2D2 virtual machine can run code for a timeslice and then persist the entire state of the running program, it should be possible to put an executing H2D2 program into hibernation, move it to a different platform (ie a machine with an entirely different type of OS or processor) and then carry on running the same program. No matter what point the program was frozen at, the code should be able to carry on where it left off.

Well I've now gotten to the point where I can test that theory. The first experiment is to run a program on Windows for a few milliseconds and then complete the execution on Linux. To make this work I needed to make sure that all my data was persisted in sizes that are the same on different platforms, so I need to use int32_t instead of int, that type of thing. Since I've written my code in a cross-platform way and since I'm using data sizes that will be the same on different platforms everything should just work. So here we go, I'm running my mandelbrot program on Windows for 200ms:

...so that outputs a file called 'demo.hby' which is the H2D2 bytecode including its persisted state (all the program instructions, the call and data stacks and the values of all variables). Now I need to move that file to my Linux box and run the code from where it stopped. On the Linux machine I have already compiled the H2D2 virtual machine from source using GCC of course. Here goes:

Awesome! It works! I guess it's not much more than a neat trick at the moment, but I think it's an achievement of sorts. If you had some kind of long running process, it might be handy to be able to wake it, run it on whatever machine was available, and then put it back into hibernation. Okay, you can't start re-writing all your business logic in H2D2 just yet... but it's early days. This is why I always imagined DALIS / H2D2 to be a cloud based language, where you don't care what type of platform or processor is being used from one moment to the next.

So the next obvious experiment is to do the same thing, but on the Raspberry Pi... maybe I'll do it in reverse, by starting the program on the Raspberry Pi and then finishing it on Windows.

More work on H2D2

Since I've been writing the parser for H2D2 - the continuation of my work on DALIS, I thought that it might be useful to see the syntax trees that are being generated. So I found a nifty tool called phpSyntaxTree which is a website that will draw a graphical version of a tree from bracketed notation.

So I added some code to H2D2 which will output the syntax tree created by the parser in bracketed notation, and now I can paste the resulting text into phpSyntaxTree and see what the results look like.

Whilst the H2D2 language doesn't have much of a syntax yet, I have been able to parse things like this example code:

an = 99
PRINT an
mynum = 88
PRINT mynum
PRINT 1 + 2*3
IF 1 PRINT 1 PRINT 0
PRINT (1+2)*3

That program tests the majority of things that I have written so far, simple things like branching, expression parsing and assigning numeric variables. This is how that program comes out:

You can see the difference between the two expressions, one with brackets and the other without, and when I run the program, the first result is 7 and the second result is 9, so it seems to be working fine. When H2D2 programs are executed I'm using a non-recursive algorithm, which means that the code can be frozen at any point and then resumed - even if the code was part way through evaluating an expression there's no problem in carrying on exactly where the code was paused. So far I've only tested all this in Pelles C on Windows, but I will put a quick makefile together soon so that I can compile it with GCC (on linux), I want to keep the C code for H2D2 as portable as I can. The next thing that I'm going to try and build is a loop, which is kinda important.

A trivial Neural Network in DALIS

Recently I've been working on adding arrays and functions to my DALIS programming language. To test this stuff out I decided to write a very simple Artificial Neural Network. I've hard-coded the weights between a 5 neuron feed-forward 3 layer network. It's a classic XOR example with 2 inputs and one output. The output should be 1 if the two inputs are different, otherwise the output will be zero. This is the output from the program:

Trivial Neural Network Example
Input 0,0 Output 1 is: 0
Input 1,0 Output 1 is: 1
Input 0,1 Output 1 is: 1
Input 1,1 Output 1 is: 0

...and this is the DALIS source code that I used:

:<b>Trivial Neural Network Example</b><br/>
ann(a=0, b=0)
ann(a=1, b=0)
ann(a=0, b=1)
ann(a=1, b=1)

FUNCTION ann()
  sizeL1=2, sizeL2=2, sizeL3=1
  L1[sizeL1]=0, L2[sizeL2]=0, L3[sizeL3]=0
  W2[sizeL1*sizeL2]=0, W3[sizeL2*sizeL3]=0
  W2[1]=2, W2[2]=-1, W2[3]=-1, W2[4]=2
  W3[1]=2, W3[2]=2

  L1[1]=a, L1[2]=b
  WRITE "Input "+TEXT(a,0)+","+TEXT(b,0)

  LOOP j=1, size = sizeL2
    LOOP i=1
      w=weightindex(b=i,f=j,s=size)
      L2[j] = L2[j] + L1[i] * W2[w]
    REPEAT i=i+1 IF i<=sizeL1
    L2[j]=output(x=L2[j], threshold=2)
  REPEAT j=j+1 IF j<=sizeL2

  LOOP j=1, size=sizeL3
    LOOP i=1
      w=weightindex(b=i,f=j,s=size)
      L3[j] = L3[j] + L2[i] * W3[w]
    REPEAT i=i+1 IF i<=sizeL2
    L3[j]=output(x=L3[j], threshold=2)
    WRITE " Output "+TEXT(j,0)+" is: "+TEXT(L3[j],0)+EOL
  REPEAT j=j+1 IF j<=sizeL3

VALUE = 0

FUNCTION weightindex()
VALUE = (f*s)+b-s

FUNCTION output()
  IF x>=threshold
    retval=1
  OTHERWISE
    retval=0
  END
VALUE = retval

So I'm now starting to think about genetic programming in DALIS, there's no real reason why DALIS programs couldn't write other DALIS programs and run them by posting them to the server.

A load balancer written in DALIS

I mentioned to a friend of mine that I wanted to slap together a quick load balancer to demonstrate that DALIS programs can bounce between different servers without the need for any shared state. Being able to do that was my original aim for writing a cloud based language, and a demo showing DALIS in action would be nice to have. Of course, I had intended to simply extend my C# webserver example and make a simple load-balancer out of that. But my friend happened to say “why don’t you write the load balancer in DALIS?”. Damn him, putting ideas in my head like that. Still I got my own back when I showed him the Esoteric Programming Languages site and he spent way too much time reading it :-) So ... here is my (very simple) load balancer written in DALIS:

TRANSPARENT
IF ISASSIGNED("arg0")
   IF (RANDOM >= 0.5)
      newUrl = REPLACE arg0, "DALIS", "DALIS1"
   OTHERWISE
      newUrl = REPLACE arg0, "DALIS", "DALIS2"
   END
   IF ISASSIGNED("arg1")
      newUrl = REPLACE newUrl, "balance", arg1
   END
   IF RAWCONTENT = ""
      WRITE WEBGET newUrl
   OTHERWISE
      WRITE WEBPOST newUrl, RAWCONTENT
   END
END

I had to add a few new things to the language to make this work (it is great when you’re the boss of the programming language and you can add whatever you like):

  • RANDOM – generate a pseudo random number between 0.0 and 1.0
  • RAWCONTENT – allows your program to access the data in the raw http request
  • TRANSPARENT – tells DALIS that you don’t want it to add anything to the output
  • WEBPOST – does an http POST with the data you specify
  • REPLACE – replace a value in some text with some new value

Anyway, this seems to work a treat, it means that you can run 3 separate DALIS servers (called DALIS, DALIS1 and DALIS2 in this example), one being the load-balancer, along with two others. You can then run a DALIS program via the load-balancer and it will route it randomly to one of the other two DALIS servers. Using the TRANSPARENT keyword in the load-balancer code is critical, since it allows stuff to simply pass through without any interference.

The trouble is, I’m actually starting to enjoy programming in DALIS now. What have I done?