Go Packages and Debugging

 This post encompasses all the work done over the past few weeks. I’ve been restructuring the code into an easily deployable package that will allow for easier parallelisation. Building packages in general is supposed to be straightforward when following this tutorial but I still run into some problems. On a positive note, I learned of the exquisite ‘godoc’ feature which converts the comments left in the code into documentation. That is one of the coolest features I’ve seen in a language so far.

 

Structs and Interfaces

The last two weeks mainly involved converting the current work on the genetic algorithm into a  generalised user friendly package for easy deployment and use. This involved creating a struct for the individual chromosome and another struct for the population.  I then implemented an interface which in Go is very straight forward. You just declare an interface and include all the methods that are used by the interface and Go handles all the magic in the background.

 

 

Now anything that employs these methods will automatically be associated with the related interface type. More on interfaces can be found here , here and here. The reason for employing structs and interfaces is because it allows future flexibility when it comes to shipping individuals between populations and even populations between different machines should distributed computing become a requirement. It also allows the addition of parameters to all the individuals and to populations themselves which would be helpful in scalability, reliability and debugging. The code also becomes more readable. This view point is supported by Matt Welsh, a software engineer who works on large production systems. Although he does raise the issue in his post that implicitly implementing interfaces in the manner that GO does may cause problems when it comes to debugging. I did get a little lost on how using reference/value types in Structs and Interfaces works but this stackoverflow post offers some clarity on this issue.

 

Go really needs a Deep Copy Mechanism

This conversion was a painful because I changed all my functions to take in pointers to types. The reason for this was mainly readability. This way I can now call pool.calcFitness() to calculating the fitness instead of calling calcFitness(&pool).

I received an error stating something about index being out of range when I attempted to compile the new code.

 

The piece of code that was causing this error was this:

This error is due to my lack of experience with Go and structs as I learned that you can’t assign an individual to a slice that’s not yet initialised (noob error). I changed it to this:

and the error disappeared, only to be met with an old and familiar problem.

I thought there was something strange going on with the code since I was running roulette wheel selection but the fittest individual was not being duplicated in the population. More correctly, after the top individual was duplicated, it was replaced during crossover. This error became apparent when I set the crossover rate to 1.0 (or 100%) meaning that individuals would swap out their whole chromosome for their partners to create the child. Basically this means that chromosome of one parent should be replaced by the chromosome of the other and vice versa (basically cloning from one individual to another).

This did not happen for all the individuals as can be seen below (pay particular attention to what happens to the first and last individuals during the ‘mid arrangement crossover’ stage). Individuals 1 and 2 are supposed to swap genes; 3 and 4 will also do the same. We only care about the population before and after the crossover at this point.

Go’s lack of deep copying mechanism had made itself apparent again. I had to change my chromosome encoding from binary to decimal to really notice what was going on.

This was the piece of code in the roulette selection function causing this issue:

 

This was passing a reference around, linking these two individuals. To get around this problem, I created a new individual and assigned it to that position in the population slice:

This worked! The solution I came up with was to first sort the list according to fitness (fittest in position 0 ). Then I initialised a new chromosome, assigned it to the value of the fittest individual, then assigned the least fit individual to the new chromosome. I didn’t expect this to work since I thought it would just chain all the chromosomes together through referencing but hey, it is what it is.

 

Pulling Go packages from Github

To pull packages from github, I needed to re-install git with the option to allow git to run from the command prompt. Then Go stopped bringing up errors in relation to finding the path to git.

Using the advice passed to me by my supervisor, I managed to get ‘go get’ working in goclipse but for some strange reason, ‘go get’ doesn’t get packages that are being imported by imported web packages. Say for example your pulling a package from github that imports another package from github, which is also importing other packages from github. The last package from github in that chain is not pulled down. Furthermore debugging in goclipse is a pain to setup. I have yet to figure out how to debug go programs in eclipse.

 

LiteIDE to the Rescue

I have been playing around with LiteIDE as an alternative to goclipse. It’s a minimal, well maintained IDE built specifically for go that offers good support for debugging. It’s very straightforward to setup, just download the compressed file for your system and extract it to a folder ( I extracted it to my c:\Go directory) and then make a shortcut from the liteide.exe file in the ‘liteide\bin’ to your desktop, start menu or wherever you would like to access the IDE.

Liteide has all the features available in goclipse, supports the Go Playground (for quick prototyping) and is cross platform. But it does require you to setup the go folder structure manually whenever you start a new Go Project (src, pkg and bin) which goclipse automatically handled. If you don’t set it up, you will run into an error when attempting to run ‘go install’. I found the solution to this problem on this stackoverflow post.

After you choose your environment (mine is win64), you need to edit the win64.env file to include the gopath to the current project environment (e.g. GOPATH=c:\Users\name\Documents\GoProjects\testProject). You also have to include the path to the gobin folder you created inside that project folder (e.g. GOBIN=%GOPATH%\bin). This is where your executable is stored when you use go install. If you don’t you will run into the go install error previously mentioned (here is the stackoverflow solution for osx users).

While in your environment file (e.g. win64.env), you will notice that the path variable includes a path to mingw64 (PATH=c:\mingw64\bin). This is what you will need to get debugging to work, which I admit was quite a pain. The mingw.org site offers a tutorial on this but I couldn’t work out what was useful to me in relation to LiteIDE and go debugging. Fortunately, there was a stackoverflow post that directed me to download a file which did include the mingw64 folder I was looking for. Just extract this folder in the root of the C folder or wherever the liteIDE path is pointing to and debugging will work without issue. (tutorial for osx users)

LiteIDE also comes with integrated support for the gopm (Go Package Manager) which is a neat little package management tool that makes it easy to search, install, update and share Go packages from source control (e.g. github).  Go package management is a hot and important topic in the GO community at the moment. The team behind gopm are also building a very neat tool that allows you to compile Go Code from github through the browser and just download an executable for any platform.

If you get lost during any part of this process, remember to have a look over the related Go Documentation. One of the best tutorials on building Go Packages with LiteIDE is by W. Kennedies, it is targeted at osx users but is very easy to follow if you are running another platform.

 

Parallelising the GA

I have been looking at making the GA concurrent . Since Go threads are cheap enough to run literally 1000s of them at any one time without issue, why not parallelise everything. I am currently working through a goroutines tutorial and have been looking to parallelise all my loops. I’ll have to limit use of the rand function as it appears to slow down code. I also borrowed some code on how to always use the maximum number of logical cores available on a machine.

 

Discussion with Prof. B. Praechter

I had an interesting discussion with Prof. Ben Paechter in the AI class about my honours project (I later found out that he happened to be the Associate Dean of the University and works with Prof. E. Hart on Artificial Immune Systems ). He suggested the dynamic allocation of cpu resources based on population levels. This is interesting when using adaptive-hfc where the number of populations grows and shrinks over time. The basic idea is to have a set number of threads in total within the application but dynamically allocate them to different tasks based on the population size and count. So more threads could be dedicated to searching higher level populations for solutions and that is also a potential area of resource. The GA package should be built to be flexible enough to handle this dynamic allocation of resources.

 

Distribution

I have also been looking at already available message passing libraries that could be used for distributed GA processing.

Some interesting ones are Beego , NSQ , Web and simplequeu. If I have enough time, I could probably build my own using the networking tools available implicitly in Go.

 

Issues to Note:

An additional issue I noticed is that goclipse doesn’t automatically recompile the executable file for a local package when a change is made if you are calling the package like this (PGA package):

Instead you have to use

So basically you have to remove the ‘./’. This appears to be an issue with Go rather than eclipse so it is one to note if you don’t want to spend hours debugging code that is actually working fine. Ohh and don’t name your packages with capital letters like I have done. (Thanks to my supervisor for pointing this out to me)

 

Notable Pages

1.       Compile in the browser: gobuild.io

2.       Go Package Search engines: godoc.org and go-search.org

3.       Great Go Programming Blogs: GoingGo.net and gopheracademy.com

4.       Go by Example: gobyexample.com

5.       All Go Projects: Go Libraries

6.       Genetic Algorithms in Go: genomego, goga, go-galib and search engine results

7.       Deployment in Go:  golang.org/cmd/go and golang.org/doc/code.html

8.       An interesting comparison of Go to ruby: Parallelism for the Win

 

Notes:

  • ·         Getting used to using packages, Using “Go Run main.go” to get this working
  • ·         Created a struct for population as populations will be shipped around to different computers.
  • ·         Learning to use gopath and go install/get – neat features
  • ·         Go makes it a pain in the ass to use remote packages, go get requires git
  • ·         Go can’t efficiently use all cores even with maxprocs – so can’t max out processors. Change in direction of project and focus on concurrency.. Make everything that can be done concurrently… Get done concurrently. Loading populations, crossover, mutation, fitness.. everything from the smallest level to the biggest one. Concurrency is cheap in Go
  • ·         Initialising custom type structs requires {}
  • ·         To actually initialise values in the struct requires a construct abit more complex than {} – this is required for slices and complex data structures in the struct. Not needed for ints
  • ·         Keep getting problems and it turns out that nothing is being returned from my initPop function. After a lot of fumbling with pointers I get it working. I have to pass the value into initPop by reference and initPop itself must take a pointer back to the original value.
  • ·         Go really needs a deep copying mechanism. It’s too painful otherwise. Lack of deep copying is one of GOs biggest short comings. Makes it hard to do AI
  • ·         Need debugging and goclipse seems a little too overblown for go. Switched over to LiteIDE, a lot more versatile. Need to install mingw (to get gdb to work) to get debugging working but otherwise good. Also have to worry more about directory structure as goclipse seemed to take care of creating src, bin and pkg for you.

Comments   

0 #3 Emmanuel K. 2014-03-04 01:15
Quoting Stan Steel:
For deep copy, a trick I've used in the past is to serialize an object and deserialize it into a new structure. This is common in other langs like Java and Javascript where performance is not critical.


Hey Stan, Thanks for the advice. Seems like Go really doesn't have a deep copy function. It has several packages that allow for the serialization of objects (most notable one being the encoding/gob package).

I did find a manually implementation of a deep copy function in Go at : https://rosettacode.org/wiki/Deepcopy#Go

For anyone interested in serializing objects, I recommend taking a look at the Go "encoding" package and this stack overflow post. https://stackoverflow.com/questions/19762413/how-to-serialize-deserialize-a-map-in-go
Quote
+1 #2 Stan Steel 2014-02-25 17:04
For deep copy, a trick I've used in the past is to serialize an object and deserialize it into a new structure. This is common in other langs like Java and Javascript where performance is not critical.
Quote
0 #1 Jon Kerridge 2014-02-18 13:35
The way you set up the GO environment needs to be captured in an appendix to your report so that anyone following on knows how to set up LiteIDE!
Give the document to me to test!

There also needs to be another appendix on deficiencies in GO as discovered during the project. The appendix can detail work arounds which you can then refer to in the report.

HFC framework is the next goal so we can test the package

We now need a chapter structure for your report together with a paragraph describing the content of the chapter and then the sections within it described by a sentence.
Quote

Add comment


Security code
Refresh