Linux on Microsoft Azure

I've recently been messing about with Microsoft's Azure platform; and I already think it's brilliant. I didn't expect the support for Linux Virtual Machines to be quite so good. Literally with just a few clicks I created a Linux box on the cloud (I went for Ubuntu), I can't really see it getting much easier. I suppose I should have known when I recently saw that Microsoft's CEO Satya Nadella had presented this:

Anyway, the first thing that I did with Azure was test my Linux VM out by copying over my C webserver code and compiling it. Everything went perfectly. But I quickly realised that it would be useful to be able to leave my code running in another terminal session when I disconnected from the console. That's where I tried out GNU Screen which allowed me to start my program in a session and leave it running after I had disconnected. GNU Screen is a nice simple solution, which means that I can keep my code simple. I like this :-) And at the same time I noticed a couple of bugs in the webserver code which I had not spotted before, so I was able to fix them too.

It's great to think that I can write some C code on my MacBook; cross-compile and test it on a small router running OpenWrt, then move it to the Azure platform without any worries. This is one reason why I still like programming in C (and Linux).

Next I decided to copy my H2D2 server across to Azure (of course). Again, everything worked perfectly. So I'm currently testing H2D2 on the cloud, let's see how stable it is. I also took the opportunity to tidy up the html, to make it look more presentable as well:

For some reason it came out in grey. I don't really know why.

Since I'm expecting to see a few crashes, I have used a simple shell script like this one which will restart my server program when something goes wrong.

Originally, I started with the most basic type of VM on offer, with a shared core. But then I did an upgrade to an 8-core machine, which gave a noticeable performance boost (as you'd expect). It was nice to see that Azure can upgrade your VM really easily, yet again it was all done with a couple of clicks (and a reboot of the VM). But now, to save CPU I've gone back to the shared core again which will be less of a drain on my account.

Drawing spheres with H2D2

Since I wrote this blog post, I have been looking for a better demo program to show what I can do with my H2D2 programming language. I needed something more interesting than "hello world" ... and which didn't involve drawing the Mandelbrot set.

I started trawling through the Rosetta Code website for inspiration and came across this sphere drawing program in C. I guess it's a simple ray tracing algorithm really. It seemed like a reasonable candidate.

So I set about porting it from C to H2D2, which was an interesting experience. Interesting because H2D2 is not very forgiving when you make mistakes - since I have not (yet) made the compiler error messages very user friendly. So it makes you reason things out very carefully in your head, rather than rely on the compiler to spot problems with your code.

I have managed to get it to work though, and the resulting H2D2 example program shows a few of the language features (like a function call and arrays for example). However, it also highlights a few things I should work on, like creating a syntactic shortcut for assigning values to an array. Anyway, I can now create output like this:

             !!...!!!***Bo               
         ..............!!**Boo           
      ..................!!**BBooe        
    .....................!!**BBooee      
   .......................!!**BBooee     
  ........................!!**BBoooee    
 .........................!!**BBoooeee   
!........................!!***BBoooeeee  
........................!!***BBooooeeee  
!......................!!***BBBoooeeeee  
!....................!!!***BBBoooeeeeeo  
*!.................!!!***BBBBoooeeeeeeo  
B*!!!...........!!!!****BBBooooeeeeeeoB  
 B***!!!!!!!!!!!*****BBBBoooooeeeeeeoo   
  oBB*************BBBBBoooooeeeeeeeoo    
   ooBBBBBBBBBBBBBBooooooeeeeeeeeooB     
    eoooooooooooooooooeeeeeeeeeooo*      
      eeeooooooooeeeeeeeeeeeeooB*        
         eeeeeeeeeeeeeeeeeoooB           
             oeeeeeeooooBB

But to show it in action, I made this recording, which shows it off in realtime in the browser:

Anyway, I am really pleased that it works so well. I can go back to fiddling around with the language now.

Hello World, H2D2 style

Now that my experimental programming language, H2D2, includes the all important server part, I thought that I'd better record an example of the traditional 'Hello World' program:

Although these programs are very trivial, it shows that I can now do H2D2 programming from my browser. It's the server part which compiles the source into bytecode and then executes the program in timeslices. So the actual code is running inside the server and we're just seeing the output in the browser (alongside the source code of course). Since I recorded this in a coffee shop before work this morning, the H2D2 server was running locally inside Xcode on my Mac, for convenience. But I would be able to do the same thing even if the server was running elsewhere, like on my TP-Link router.

Considering that the whole H2D2 language is written in C, I think that's pretty good going. It seems reasonably fast and I don't even think I have the compiler optimisations turned on. Obviously there is a little bit of JavaScript doing the HTTP posts between the browser and the server. But I really do need to write some better code examples, which should be things that don't involve the Mandelbrot set for a change...

UPDATE: one of the "better code examples" I mention above can now be seen here.

I love it when a plan comes together

So recently, several of the different things I have been messing about with have come together. This was always in the back of my mind, but I didn't really know how long it was going to take. Recently, I have been working on:

  • building my own webserver (dweb)
  • cross-compiling for Linux on a TP-Link router (with OpenWrt)
  • ...and my own experimental programming language (H2D2)

Now I have bean able to bring all that together, what it means is that I can run H2D2 programs on my little TP-Link router. At the moment, I'm doing it through jQuery Ajax calls. So the programs are sent to the router with an HTTP POST from an ajax request, the router runs the program remotely and then returns the output back to the browser. Nice.

Here is a screenshot showing it all in action:

Woo-hoo... it works! Oh, and apologies for the partial ASCII Mandelbrot, I really can’t help it. The next experiment will probably be to do what H2D2 was actually designed for, namely to run a program partially on the router and then finish off the same program instance on some other platform. Then, I should be able to bounce a single program between the router, a Raspberry Pi and a desktop, which will be a neat trick.

Extra speedy H2D2 on the Mac

I remember there was a time I was very happy because H2D2 was running my Mandelbrot test program in about 650 milliseconds (whereas the original proof of concept, DALIS, took nearly 12 seconds to do the same thing).

Since moving over to my new MacBook and compiling the H2D2 source using Xcode, just look at the result now:

Whoa! 72 milliseconds! Now that's impressive. This new MacBook is pretty fast, but I'm sure the Apple LLVM compiler is also playing its part in achieving that level of speed. I have done some optimisation work in H2D2 recently, but I don't think I can take the credit this time.

It really pays to do this type of 'system level' programming in C (plus I enjoy it of course). Right, I'm off to work on parsing string expressions now...

H2D2 … now on Mac OS X

OK, first of all, I've bought a new laptop, and it's a Mac - a 13" MacBook Pro to be precise, the one with the Intel Core i7 processor, 8Gb of RAM and 750Gb hard disk. After having great success with running things as VMs at home, I've decided to do the same on my laptop. But the old Toshiba T130 was just not powerful enough to do that (it seemed like quite a powerful laptop when I bought it, but that was over 4 years ago I think). Besides, the Toshiba was feeling quite slow anyway.

But I came to realise that if I was going to run all my stuff as a VM, then I didn't need Windows as the host OS. And getting a new shiny Apple piece of kit seemed like a pretty good choice. I've virtualised the old (Windows) laptop and now it's running happily as a VM in VirtualBox on my new MacBook. Pretty seamless really.

But, I then went and downloaded Xcode (Apple's IDE) and quickly realised that it will happily support plain old C programs. So I copied the H2D2 codebase from Pelles C on Windows into Xcode on the Mac. It compiled and ran first time - even I was surprised. But that is what is supposed to happen when you write portable C code - you can run it natively on just about any platform.

So, I'm going to carry on developing H2D2 in Xcode for a while and see how that goes. My initial impression is good, I'm still learning the keyboard shortcuts … but then I'm still getting used to the new Mac keyboard as well. I was surprised to realise that there is no hash (#) key, at least not on the UK Mac keyboard. Maybe twitter isn't as popular amongst Apple users as I had thought. Or maybe that's just because I think of Steven Fry when I think of a typical Mac user :-)

Anyway, I'm still finding my way round the IDE at the moment. It's probably too soon to draw any final conclusions. Maybe I should buy a book on Xcode...

Some H2D2 progress, whilst messing with Run Length Encoding

So progress on my programming language, still codenamed H2D2 continues on background threads :-) I've not given up, but I haven't made many posts on the blog about it recently. It was probably getting boring anyway, and recent work has been tidying things up and writing unit tests - which does not make for interesting reading.

One thing that I wanted to try, was to compress the H2D2 bytecode so that it doesn't take as much time to serialise a running program between machines. I believe that Java JAR files are zipped, so I thought that I'd do something similar with H2D2. But I also wanted something that I could write myself that didn't take an age. Since I'm doing this for fun, it's nice to start with a blank page and go from there. So anyway, my web wanderings brought me to here. That page talks about Run Length Encoding - something that I implemented once myself back in the 1980s (or maybe the early 90s) - not that I have the source code anymore. But I read about the RLE PackBits algorithm and thought it might be a good way to go. Not a massive processing overhead and reasonably simple to implement. I chose not to look at the example souce code provided ... and just to write it from scratch myself. It sounded like fun.

It took me about an hour whilst drinking my morning coffee to write the compression code, and an hour or so in the evening to decompress the data back to the original values (I also wrote some unit tests to check for any obvious problems I could see). It seems to work OK: when I compress H2D2 bytecode I'm getting about a 30% reduction. It's not as good as zip compression obviously, but it will do for now.

I'm sure my code won't win any prizes for optimisation or style, but at least it works. I guess it might also come in handy if you were trying to squash some data on a microprocessor without needing a massive library or lots of processing power. So here is an example C program:

// Example RLE compression with PackBits
// ...just an example, not production code :-)

#include <stdio.h>
#include <string.h> // for memset

void fgrow(char*, char*);
void fshrink(char*, char*);

int main(int argc, char *argv[])
{
  if (argc==4 && strlen(argv[1])==2 && argv[1][0]=='/')
  {
    switch (argv[1][1])
    {
      case 'c':
        printf("Compressing... ");
        fshrink(argv[2], argv[3]);
        printf("done!\r\n");
        return 0;

      case 'u':
        printf("Uncompressing... ");
        fgrow(argv[2], argv[3]);
        printf("done!\r\n");
        return 0;
    }
  }
  
  printf("Usage: RLE { /c | /u }  \r\n");
  printf("   use the /c switch to compress or the");
  printf(" /u switch to uncompress\r\n");
  return 1;
}

void fgrow(char* inpath, char* outpath)
{
  char ch=0, hdr=0;
  FILE* op = fopen(outpath, "wb");
  FILE* mp = fopen(inpath, "rb");

  while (!feof(mp))
  {
    hdr=fgetc(mp);
    if (feof(mp)) break;

    if (hdr>=0) // copy bytes verbatim
    {
      for (; hdr>=0; hdr--)
      {
        ch = fgetc(mp);
        fputc(ch, op);
      }
    }
    else // copy a run of bytes
    {
      ch=fgetc(mp);
      for (; hdr<=0; hdr++)
      {
        fputc(ch, op);
      }
    }
  }
  fclose(mp);
  fclose(op);
}

void fshrink(char* inpath, char* outpath)
{
  char buf[128];
  memset(buf, 0, 128);
  int n=0, hdr=0, runlength=0;
  FILE* op = fopen(outpath, "wb");
  FILE *mp = fopen(inpath, "rb");

  while (!feof(mp))
  {
    buf[n++] = fgetc(mp);
    if (feof(mp))
    {
      n--;
      break;
    }

    if (n==128)
    {
      // buffer is full
      hdr=127;
      fputc(hdr, op);
      fwrite(buf, 1, 128, op);
      memset(buf, 0, 128);
      n=0;
      continue;
    }

    if (n >= 3 && buf[n-1]==buf[n-2] && buf[n-2]==buf[n-3])
    {
      // last three bytes are the same
      hdr = n-4;
      char symbol = buf[n-1];
      char tmp = symbol;
      if (n > 3)
      {
        fputc(hdr, op);
        fwrite(buf, 1, n-3, op);
      }
      runlength=3;

      while(!feof(mp) && tmp==symbol)
      {
        tmp = fgetc(mp);
        n++;
        runlength++;
      }
      hdr = 2-runlength;
      fputc(hdr, op);
      fputc(symbol, op);

      // put the different byte back for next read
      if (!feof(mp)) ungetc(tmp, mp);
      memset(buf, 0, 128); 
      n=0;
    }
  }

  // flush the buffer if not empty...
  if (n > 0)
  {
    hdr = n-1;
    fputc(hdr, op);
    fwrite(buf, 1, n, op);
  }

  fclose(mp);
  fclose(op);
}

Obviously this compression will only shrink files which have runs of the same repeated byte value in them. Otherwise it may increase the file size. You can try it from the command line and pass /c as the first argument to compress a file and then use /u to decompress the file back. The next two arguments need to be the names of the input and output files respectively. This example doesn't have any error trapping, so if the file fails to open, or something like that it will go bang. But that would not be hard to add.

Of course, in my H2D2 code, I'm using the same algorithm, but I've adapted the code to work on blocks of memory rather than on files. But in H2D2 I have made a VFILE struct to work alongside the C FILE pointers and I've added some functions that work the same as the C IO functions but taking my VFILE pointer instead. This means that I can easily swap code to operate either on disk or in memory...

Unit Tests in Pelles C on Windows

I said before that I might try and do some unit testing in C, rather than use the features in Visual Studio. It would mean that for H2D2 I can do everything inside Pelles C and won't need to have two IDEs running. But I've never done any unit testing in C before (at least if I have it was so long ago I've forgotten). So I needed to go off and find some means of doing that.

So I found the libtap library which is a unit testing framework for C programs. Well, it is something that implements the Test Anything Protocol which you can read about here. I'm no expert on the Test Anything Protocol, this is the first time I've used it, but for my purposes it looks like it will do fine. As long as I can run code and check for expected results and have the Passes and Failures shown in some sort of output then I'll be happy... But for the moment it looks ideal. I've had to make some minor modifications to get it working in my project (in Pelles C for Windows) but it was not very much effort really. I've just made my own version of tap.c and included it in my project. I had to include an implementation of vasprintf() which I found here rather than rolling my own, but that seems to work quite well for me.

I found libtap after searching for C unit testing on stack overflow in case you're wondering how I found it in the first place.

What I've ended up with is a #define which tells the compiler which version of main() to include. One version is my usual code and the other runs my test suite. I haven't began porting my actual H2D2 unit tests back over from C# yet, but I do have a trivial example running just to check that I can run some unit tests. So this this is all I've got so far:

I've also set it up so that the content of tap.c is not included when I compile normally, so this means that my code isn't bloated with the test framework when I'm not using it.

I've resisted the temptation to mess around with libtap too much, I've only made enough changes to get it working in my project. Thanks need to go to Nik Clayton, who wrote the library and to whoever River Tarnell-7 is for posting his vasprintf() implementation. Thanks! In case it helps any other people using Pelles C who would like to add units tests to their project, here are the files:

Download libtap, adapted for Pelles C on Windows.

I'll probably try this out on the Raspberry Pi soon, because I think it may be useful to have a lightweight unit testing framework for GCC on the Raspberry Pi too. I'll post an update if anything comes of that idea.

The joy of tests

Since my H2D2 project has quite a lot of code in it now, I've been using .Net interop as a quick way to write some unit tests for the H2D2 programming language.

I have written a C# unit test project in Visual Studio which tests much of the existing H2D2 language syntax. This means that I can add new features and quickly check that I've not broken any of the existing code. So I can very easily make sure that programs which used to run continue to work as expected.

Recently, I've been making H2D2 support function calls, and so it's been handy to write tests for various different types of function call (passing parameters, reading return values, allowing functions that don't return a value, etc). The unit tests have made it much easier to ensure that everything works after I've made changes.

Since I've written more unit tests in C# than anything else, I thought it would be the quickest way to get my tests up and running. And because I had already compiled H2D2 into a windows dll (the unmanaged kind), it is not very hard to run H2D2 programs from .Net - even easier because I had already written some basic wrapper code for all the interop stuff. It's not much harder than putting some H2D2 source code into a string and calling a Run() method. Obviously all this has been done 'just for fun', since it's only one of my hobbyist projects anyway.

But when I mentioned to one of my colleagues that I had spent some of my free time writing unit tests for fun he said that I was clearly stark raving mad. There's a possibility he's correct.

However, when I think of how I'd feel if I broke some of my existing H2D2 code and had to go back and unpick where I went wrong, I reckon that these unit tests are really less about having fun and more about preventing a severe loss of fun if I ended up scratching my head and trying to figure out what I'd done to make something break.

So I hope that's a suitable explanation for why I've been writing unit tests in my free time. It's the best excuse I've got.

Although ultimately, I may end up looking for a unit testing framework for C so that I only need one IDE loaded to run my tests... The .Net interop solution may just be a stepping stone.

Implementing Function Calls

I've been busy lately adding the concept of function calls to my H2D2 programming language. It hasn't been as hard as I thought, but there has been a reasonable amount of code to write. I have added the notion of scope as well, so that when a function returns, any local variables go out of scope and the H2D2 virtual machine can free the memory allocated to them. Currently, functions can return a value (although it's possible in H2D2 to return nothing from a function too) ... but I can't yet pass parameters to functions, that's on my TODO list, I'll work on that next.

So here is the current test program that I'm using to see if it all works:

test()
loop(s=48)
  print 10
  print 13
  print s
  print 32
  x=test()
repeat (s=s+1 if s<58)
print x
return

function test()
  loop (s=65)
    print s
  repeat (s=s+1 if s<91)
return 42

This program simply outputs some ASCII characters, the result comes out like this:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
0 ABCDEFGHIJKLMNOPQRSTUVWXYZ
1 ABCDEFGHIJKLMNOPQRSTUVWXYZ
2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
3 ABCDEFGHIJKLMNOPQRSTUVWXYZ
4 ABCDEFGHIJKLMNOPQRSTUVWXYZ
5 ABCDEFGHIJKLMNOPQRSTUVWXYZ
6 ABCDEFGHIJKLMNOPQRSTUVWXYZ
7 ABCDEFGHIJKLMNOPQRSTUVWXYZ
8 ABCDEFGHIJKLMNOPQRSTUVWXYZ
9 ABCDEFGHIJKLMNOPQRSTUVWXYZ*

Simple stuff, but it demonstrates calling the "test()" function like a statement (ignoring the return value) and also assigns its return value to a variable too. I have also done some work allowing this type of code to be written out as labelled bracket notation. Essentially I'm treating each function like a seperate program, starting with its own root node, and then each call to that function simply points to the correct root node. This is how the above program looks if I render the labelled bracket notation generated from the bytecode:

Currently, when I parse the source code, I simply scan for function definitions and emit the code for them and then go back to a 'bootstrap' node which is the programs entry point. The starting point of the source code is pointed to from the 'bootstrap' node. It seems to work fine, and it means that programs always execute from node zero (which I'm making sure is always the bootstrap node).

In other news, I compiled H2D2 as a Windows dll, and used Interoperability to call it from C#, this worked a treat and was extremely fast. With the dll version, any output that the H2D2 program creates is returned as a pointer to a string, so after the program has been run the C# code needs to make an additional call the dll and tell it to free the string containing the output. But, my mandelbrot test program runs in about 200ms this way, since it doesn't have to muck about with writing to the screen. Once the C# program has all the output it can write it in one go rather than a character at a time, which is much quicker.

Now that I have installed Visual Studio 2012 I might be able to hide H2D2 behind a websocket interface. This should allow for a 'chatty' interface to H2D2 where you can ask it to run programs and get the results returned. This is also on my TODO list :-)