Over the years, I’ve screwed with a lot of build systems. Both in the course of my own projects and other people’s, and I’ve come to a conclusion over the past fourteen years.

At best: you can reimplement make poorly. At worst you can reimplment half of autotools, poorly.

That’s pretty much what I’ve seen. Thus as time has gone on, I see it very hard to do better than good old Make. Especially when the GNU version has about five hundred pages worth of voodoo to appease even the worst masterbaters, and the need for autotools is kind of waning IMHO.

Enter ninja.

What I’ve generally found with Ninja is that it’s very simple. Like C: the little bit of syntax you need to remember is a small quantity. Opening a build.ninja file is probably enough to grok what’s going on if you’ve ever used an actual build system that involves editing files.

Likewise answers to questions that tend to make it easier to build a wonky, hellish, broken build monstrosity, tend to be “No, you can’t because that would make this slow”. And let’s face it, if you want much more than a relatively simple Makefile, you’re probably building a case for pain.

Based on the past year, I think ninja will be sitting next to vim and dump in my toolkit of loved and trusty computing companions.

When working off a local clone of code while I’m stuck sitting in a doctors office, I’ve got to admit, it would have been smart to have run make tags, and copied the resulting TAGS file over from the development environment to my transformer. Turns out it depends on more than just the usual tags generator >_<.

Oh well, I guess it’s time to nav it the other way!

Sometimes Linus is a genius

When you begin to do more advanced things with subversion (not my idea, trust me), I would have to conclude using tarballs and larger hard drives is more useful than subversion.

For the first 10 years of kernel maintenance, we literally used tarballs and patches, which is a much superior source control management system than CVS is, but I did end up using CVS for 7 years at a commercial company [Transmeta[11]] and I hate it with a passion. When I say I hate CVS with a passion, I have to also say that if there are any SVN (Subversion) users in the audience, you might want to leave. Because my hatred of CVS has meant that I see Subversion as being the most pointless project ever started. The slogan of Subversion for a while was “CVS done right”, or something like that, and if you start with that kind of slogan, there’s nowhere you can go. There is no way to do CVS right.[12]

source: Wikipedia on Git.

Please, for the love of sanity (and science), do not use Subversion…..you have been warned!

Right, premake4 is one of my favourite ways to build C/C++ stuff but tonight I’m thinking it has a moron involved. You can specify a project as being one of four kinds: ConsoleApp, WindowedApp, SharedLib, StaticLib. The documentation here states that this likely means /SUBSYSTEM:WINDOWS will be passed to link.exe, and indeed it is. That is how you say call WinMain and do any other I’m a GUI app magic for Windows; other wise you get a main and a command  prompt; simply put.

This FAQ entry on the other hand, is just retarded.

A case for autotools that I have only recently begun to understand

Like just about anyone who has ever had to install software from source has, I have /used/ autotools before. But like many barely ever scratched the surface. Lately I have been cuddling up with the autotools from a developer perspective, a lot more. To the point perhaps, that I am liking autotools better than I ever thought I would. Like anyone whose used more than 1% of autotools, I know you can (or are supposed to when the developer did it right) be able to run configure/make outside the source tree. Also I know about –prefix and most of the usual configure script goodness. Now, it’s far from the first time that I have mentioned it, but my “Holy Grail” of builds has long been a multi-tree build:  one tree for build files, one tree for distribution files, and trees for whatever source and data files are needed. Then because I may be doing a diverse set of platforms, this usually becomes a need to further grind down into having co-existing build/dist trees: for example to have FreeBSD, OpenBSD, Linux, and Windows NT builds in the same working copy. Including using different toolsets, such as GCC 3.x, 4.x; and MSVC 9.0, 10.0 on the Windows NT builds. Now that my main computer is an ARM, processor architecture will probably end up mandatory. I like stuff like Build/platform/toolset. Hacks to keep this sort of thing working under {insert random OS here} should be kept to a minimum. How well a build system supports helping me with this problem (and file system hier) is one way that I judge build systems. Over the years, I have tried…just about everything except ant and maven but hey, how many C/C++ projects do you see using those at home? Multi-lingual stuff is always idea. At present, my favourite build system is premake4 — it makes setting up such a build pretty painless. After that is probably Qt’s qmake, since it makes compiling fairly painless. With the GNU build system, it is pretty easy to do something like:


$ cd Build/Linux/ARMv7/gcc44
$ ../../../../configure --prefix ../../../../Dist/Linux/ARMv7/gcc44
$ make install

Which would give something well suited for testing, and there is a config file to save whatever configuration options I usualy test with in $prefix. The check and installcheck targets also do what I will typically do with a `make tests` or a `./tests.sh` in my own working copy. To top it off, autotools probably has one of the best tools for making a distribution: make distcheck. I do not need most of what autoconf can do, and usually prefer to skip it. The ability to have things fail at configure rather than compile time is handy. Being able to e.g. toggle between Deps/{pkg} and the systems {pkg} at configure time is great and something I already do with premake4. Automake can also pretty much do what you could get out of not having to hand write Makefiles. What REALLY shines however is libtool! We have all cursed at something or other involving autoconf, automake, or libtool. But the little secret is libtool is one of the best assets ever given to a developer. If you don’t think so, you should try to alcomplish the same thing in SCONS for a solid week, just to support three platforms with differing toolsets, unless much as changed in a few years: the amount of kludges is a recipee to be pissed off. libtool can pretty much do it all and for a lot of platforms. The real question is how well autotools would really play with doing a Windows NT build with Visual C++. I’ve never tried that. That is also the principal reason I’ve never used autotools on most of my projects lol.

Hmm, finally a chance to kick back and update my journal, before dinner ^_^. Something that’s been on my mind, is task management and record keeping, or more broadly the issue of “Productivity”.

When it comes to general coding or computing tasks, I stopped worrying years and years ago. I’m noted as a VIM user who loves its editing efficiency, and I would take something like vi or emacs any day of the week over a glorified Notepad, which is a class of code editor that excels at inserting efficiency. Likewise, I spend much of my time in a command line environment—which is far more productive for me than what most `normal` people use on their computer. Being at home in a terminal is also kind of a necessity where I work, hehehe.

So what does concern me, is at it’s essence Getting Shit Done and Keeping Track of Shit Done. Over the past couple months, I’ve extended my natural workflow to something more effective for a working environment. In the end, it’s still the same objective though, GSD and KToSD.

Traditionally, by project I would usually keep a set of notes: in the project root, either in a file called TODO or buf; which is a convenient name when your editing source code and want to write out a scratch buffer. Technical and personal notes would generally fall somewhere under ~/Documents with temporary notes and snippets in ~/tmp or ~/misc, depending on how long I need to cache it. Obviously, most of the interesting things would get converted to HTML and blogged here! My vim setup also defines an “Outline” mode that I utilize for brain storming and various other things that belong in a .outline file.

At the present, I’m utilizing a note taking program called TomBoy, which has become quite the useful notebooking tool! While it has a few things I find irksome or overly minimalist, it also has a few that I quite like, and fits the #0 rule: it just works. TomBoy is written in C# and works great on every platform I’ve tried it, including Windows. The main lack to it, is having to fumble about to find which window on screen belongs to my current note. I would like to find the time someday to write plugins for interfacing it with other software in my workflow but that’s more so for use outside of work. My systems all run the TomBoy client and utilize one of the many backends for synchronization, so I always have an up to date set, which is good because (*groan*) sometimes I may have to work from home.

What I do with TomBoy, is maintain a set of notes: the Current Action List and . Whatever is most relevant to the here and now gets scribbled into the current action list. So for example, if I’m implementing some change to a program I will have outlined the phases needed to do it as a set of bulleted lists in my Current Action List note, which is saved in a notebook named after the company I work for. During a `change of gears`, or as certain information becomes less associated to the Here And Now of what I am working on, it gets cut and pasted into a note named after the project in the same notebook—so I know exactly where to find things. If a note becomes overly concentrated, becoming more like a multi-section collection, the notes get split and prefixed by the project name. I also have a “Programming” notebok for things that are of a more general programming nature, like comments about APIs or how to use certain tools. Today I added a “Personal” notebook, which will probably gain a Current Action List note of its own someday.

An example of this workflow in practice, earlier in the month I was focused on writing code to model a problem and code to test that for correctness. Kind of like an individual Test Driven Development. But I didn’t have time to review things very well, yet I had gotten into this sweet work flow between coding, noting, and testing.  It got done and so did the code that needed to use it.

I outline what I need to do in TomBoy, using as little or as much detail as necessary, then I do it and adjust the note as necessary. It’s for insurance: in case doing X takes N*4 times as long, or I suddenly have to take a shit in the middle of something, I won’t forget to do the Xth thing. Out of convention, I prefix completed “Tasklets” with a ‘-‘ and highlight problem areas or WIP’s with a ‘+’ prefix, e.g.

     

  • – change A
  •  

  • – change B
  •  

  • + change C
  •  

  • change D

Where D is still to be done, C is giving me more overhead, and A and B have been hashed out.

This is helpful because for example, you might think of something Important(tm) that you don’t need to focus on *now* but will need to _soon_. Or at least, it is helpful to me. I’ve been doing this for a while now and it’s really meshing well. When a subject get hammered out, like A, B, C, and D all completed, tested, and committed; then I either delete the note text or transition it out of Current Action List and into Project Name. Just depends on whether or not the info needs to be retained.

Something that proved the utility of all this to me, was when I decided to take a step back and focus on examining the work I had done. So I took a couple hours to go over the programs code base, subject by subject, and making corrections as I saw preferable. To do that, I created two indented sections in my Current Action List: one for notes on application code and one for enumerating what the test cases actually cover. For each significant element that had changed during that cycle of development (probably ~50 commits), I did two things: review the test cases from the outside in, writing down what they actually tested (the assumptions), and an occasional (highlighted) comment about what needed to be tested for or changed. Then I opened the associated source code, and went through it and made any practical comments, in a form of super-short-review. This resulted in a Current Action List that looked like this:

Subject A tc ->
  * tests quux domain
    + includes ...
    + excludes ...
    + how it deals with X possiblity
  * and so on as appropriate, covering:
    + domains and ranges
    + state assumptions
    + error
    + and dealing with 'wtf was that?'
Subject B tc ->
  * ditto

Subject A ->
  Short, concise reviews
  * including comments with pointing out
  * and areas for change

Subject B ->
  ditto.

Looking at the assumptions the test cases make rather than the code it tests, showed me areas where the tests needed to expand, which showed me places that needed to be changed. To keep myself “Fresh”, I paused after each subject/module to play “Fix it” to the flaws that I found in examining the tests and code, then update the note contents accordingly. Then I moved that chunk out of Current Action List and into Project Name. So if for example, before moving onto compiling the stuff for B, I would make the corrections to A that compiling the notes had found, then cut/paste it into the note for the project. I think this is a good thing, kind of like the old UNIX guys. I think it was DMR, who once commented on writing the BUGS section of a man page, only to go fix it rather than release it.

Doing that check the tests, check the code, fix the code cycle, allowed me to cover a lot of ground fast without having to worry about keeping track of minute details. I also didn’t space out from reading to much code and writing to little, and still have it saved in case I need to reference it later.  After everything was done and I was ready to start the next phase of what I was working on, I transitioned all that stuff out of my Project Name note and into a note named “Project Name Test and Code Review”, and left a line in “Project Name” to see that file for that. Clear, clean, and simple as eating.

Two reasons I love TomBoy: it has *excellent* support for lists and indentation. It totally and completely blows away software like gmail and Word in that regard, and provides the ease of use that my outline modelet in VIM was created to fill. It is incredibly easy to manage and manipulate info in a TomBoy note, and adapt it using list or indentation structure to organize a note in such a way that you can parse it at a glance. I generally use indents for sectioning, lists for enumerations. The second thing I love, is that whenever you write the name of a note or a file path it becomes a clickable hyperlink, very much like a wiki. But unlike the majority of wiki’s it is tied to the name of an actual note (entry) rather than a SillyNamingConvention. There’s also great support for renaming notes.

The thing that TomBoy is really Not So Good At, is managing TASKS instead of NOTES. It just wasn’t designed to do more serious task stuff then a sheet of paper. Yet it is pretty damn good at the whole notebooking thing.

There is a program called Tasque but for my needs, it is much to simplisitic to be useful. In fact, Google Tasks in GMail is probably as good (and sucks). I gave Tasque a wurl because it had both a TomBoy plugin and a RTM backend. In testing that out, I found the TomBoy side of Tasque really ain’t that spectacular either. In point of fact, just integrating RTM back into my work flow as proven useful. My usage of Remember The Milk has totally gotten overhauled, and it would be worth while to someday write a TomBoy plugin to help associate their notes/tasks.

I’ve basically replaced what fixed (and standard issue) lists I had with purely “Smart” lists, letting tagging and inheritance do the organizational leg work for me. This suits me, because I can set myself a task with tags that will make it show up in each list I want it to, so e.g. a task can be in both my list for work tasks and my list of people to contact, be readily searchable, and have a way of associating notes, due dates, and all sorts of handy stuff with it. Basically all the good stuff for tasks that TomBoy lacks.

That’s the thing of it though,

  • Just Work without Kicking.
  • Let the software do the house keeping.
  • Allow me to focus on working rather than structuring
And I’m quite happy, lol.

Well, while I’m waiting on a subversion command to finish (on another project), I may as well flush my train of thought, in between wondering how any large project can bare to use subversion lol.

One of my open loops is a component called the “Data Browser”, it’s meant to provide a view into data extracted from a projects source code, and present that data to the user: while bridging that browser interface into the rest of this programs peers. In less abstract English, it’s a tags browser. Go figure.

Something I love about programming, you can often express a notion in 3 words of code, what would take 10 words of English to describe. How that works? You can reduce the English word count with the use of insider jargon, but being a programming language, outsider is redefined as those who can’t read the language rather than those who don’t comprehend the associated tech speak. There fore the word count falls significantly.

My present train of thought however, is concerned with how the data should be presented: what is most suitable for the user. The fact that the program is designed first and foremost for my own convenience is aside the point :-o.

In search of the holy grail of user interfaces: I’ve found this the most optimal method.



+---------------------------------+
|  [-] Classes                    |
|      [+] Foo                    |
|      [-] Bar                    |
|            someMethod()         |
|            ....                 |
|      [+] ....                   |
|  [+] Functions                  |
|  [+] Macros                     |
|  and so on                      |
|                                 |
+---------------------------------+


where as much information about the individual items has been omitted for the sake of brevity ^.

Properly expanding the tree for a given type of data should display information unique to it. I.e. what is most pertinent to *that* type of data, rather than a common subset that applies to everything.

Columns for a methods display might look like:

| name | signature | return value | visibility |

where as the columns for classes, as opposed to ‘Bar’, might look something like this:

| name | visibility | in namespace | 

.

In a perfect world we could do this over a sandwich without much coding. Using the GTK+ TreeView widget and friends, grepping the manuals suggests that life is just going to be waaay easier if each element (Classes, Functions, Macros, etc) of the tree becomes a separate tab holding a specialised treeview as part of a Notebook widget! I’ll look into it closer when I have more time for that.

Now of course the tree view could simply show the lowest common denominator for info, and rely on a “Properties” button to show the individual details for the currently selected item, or we could (barf) just have expanding the trees spawn a new window customised to that type. But nether are to my taste.

Enough rambling, time to get a move on while subversion continues to (ab)use my networks bandwidth.

Today I’ve had to get used to living with Subversion again, as much as I simply love git, setting up a repository is easier with svn then git-svn :-S. Aside from the obvious problems that chop up from their different mental models it’s not that bad.

The lights flickering as the thunder blares outside on the other hand, tell me that I should probably *WAIT* to merge this vendor branch into the trunk – and go look up how atomic svn copy src-url … dst-url actually is in terms of network and power failures! If I was using git I would just say screw it and do the merge, because I know already that git doesn’t phone home to the repository, because the working copy is a repository.

Oh well, to the manuals!

Quick & Painless C/C++ header installations using rake

In wrapping up and refactoring my new rake driven build system, I’ve written numerous utility functions. Most stuff specific to compiling crap is in a custom “Builder” module that provides a templated interface to the current OS/Compiler kit, via environment variables and methods like make_object, make_shared_library. My top level rakefile also has a neato inference rule rigged together for handling my triple-tree and quad tree build patterns.

The real fun of the day however, is this baby:


#
# A magic function that generates file tasks that copy all headers from
# directory ind # to directory outd.  Any missing directories are created as
# needed.
#
# The return value is a list suitable for the right hand side of a task, e.g.
#
#   task :headers => header_magic(src, dest)
#
def header_magic(ind, outd)
  dirs = []
  hdrs = []

  find_files(ind, /.*.h[hxp]*$/) do |p|
    outdir = File.dirname p.sub(ind, outd)
    outfile = File.join(outd, File.basename(p))

    directory outdir
    file outfile => p do
      cp_r p, outdir
    end
    dirs.push outdir
    hdrs.push outfile
    CLEAN.include outfile, outdir
  end

  dirs+hdrs
end
find_files is another custom function, it simply takes a block and yields and file names matching the specified pattern. (In this case, more liberal than the usual header extensions but sufficient for my needs)
I call it header_magic, because it after all the other stuff I played with, it’s the closest to Clarke’s third law. Which is exactly what I want!
footnote: I just have my top level clean task nuke the entire build tree; so the CLEAN part hasn’t been tested.