Programming

Today at work, I was thinking over my next agenda with tmk, namely implementing the checksum based method of checking whether or not targets are up to date, and recipe caching.

Implementing the checksums are actually pretty easy, the hardest part is just adding a command line option and a method for changing the checksum algorithm to be used.

Doing the cache on the other hand, is a bit more ‘thought’ required in the solving. Because tmk currently does variable expansions on rules, it’s impossible to correctly cache for any rule or variable assignment involving an environment variable. The principal reason for the variable expansions, was so to help ensure the uniqueness of a rule. So obviously, the proper thing to do is to delay the variable expansions until they are needed, there by making the data store completely cacheable.

The obstacle of course, is that rule definitions can no longer have the chance to be a way in which the rules become unique, which rules out the possibility that variable assignments can occur after the top level: which encourages me to push the spec in the direction I had originally planned. Another option, would be to just leave it as is, and note the issue around environment variables in the manual. Of course, a third possible solution is making the processing step more aware of line numbers.

Since I’d rather have it that way, I’m opting for the first: delaying the variable expansions so that the entire result of the parsing phase can be cached.

I also feel like a chicken with it’s head cut off atm :-S

I’ve just fixed the bug about tmk reporting the wrong filename when an included file contains an error. The thing that makes me smile, is because I designed the recipe parsing and processing code reasonably well, making that work correctly was trivial. At worst, I expected that I would have to modify the Recipe class to incorporate a separate data stack into the parser, but not even that was necessary. Most changes were (as hoped) just refining the data structures used for storage.

tmk is pretty simple, it does two passes at a recipe:

• First it parses the recipe into an internal data store, most serious (e.g. syntax) errors are reported here. A minimal level of evaluation is done, namely we need to do some expansions or you can’t use variable substitutions when defining a rule.
• Secondly, it walks through the data store at processing time (e.g. doing the magic), conducts final expansions when needed, and carries out its mission in life.

I’m going to be dead tired before noon even approaches,  but I’m smiling now! The focus of my day, has been on getting tmk up to snuff enough that I can use it as a general purpose solution to my problem: cursing at the present ‘generation’ of such tools.

I worked on getting a base set of magic bound variables and teaching tmk that certain rules may be skipped, if a set of pre-conditions hold about the files involved. Getting that done was easy enough. The checking code is now more robust, properly handling an arbitrary set of input/output names, in as much is humanly possible ;). I’m smiling, because I spent most of the night being annoyed every which ways up, on top of an already splitting head. Ended up having to quit coding for a bit, and just hit RvS for a couple hours.

About two hours sleep, and still plenty of hours until sunrise, I woke up and got back to the codin’ and now it’s done!

So far, tmk is put together in a rather short amount of time, even if it’s been on my dancing list for a few months; finally it’s almost beta quality. Only show stopper that’s come up in testing, is it fails to handle unexisting tmk variables correctly, but that’s a one LOC fix. An outsanding issue, is that tmk variables with whitespace in them are improperly expanded, e.g. $(foo bar) expands to bar) even when a variable named ‘foo bar’ exists. That however is because of how the tokenization feeds the parsed data into the variable expansion system. Although for the sake of simplicity, I planned long ago to make the specs dictate such variable names as invalid whether or not tmk actually accepts them, so it’s a lesser issue. The include processor directive, also reports the wrong (e.g. parent) filename but correct line number (e.g. from the included file), yet that bug can be fixed in a few minutes; so I haven’t bothered yet.

Two features that remain to be done, is making rules relational (by dependency) rather then executing them in sequence, and to giving tmk the option of using checksums rather then modification times for minimising rebuilds.Which also comes into part of the leg work, for implementing a cache, hehe

Most of what needs doing, is some light polish and adding more builtin directives. Been thinking about making tmk understand a simple plugin system, that would allow it to load reasonably trusted bits of python code into part of the program, thus allowing new directives to be added at will, as well as replaced. I’ll worry about that later though.

Yesterday I setup tmk to understand how to include files, by using a simple pre processing directive, making the syntax for tmk fairly simple:

variable = value

rule lhs -> rule rhs:
        command directive! word arguments

Where variable expansions and file name globbing is widely used as part of the design, while keeping the recipe parsing code fairly trivial. I designed it this way, both to be familiar with Make (although lhs/rhs are reversed), fairly obvious at a glance, and to be easy to implement. Now tmk also understands the concept of a (pre) processing directive. Since directive! is used within rules, the processing directives use the inverse, i.e. !directive. I believe that makes a better distinction then using a different symbol, especially since you can not use a command directive everywhere a processing directive is permitted. Right now there is only one directive, !include which slurps up it’s argument list as the name of a recipe file to include within the current, before continuing of course. Some form of conditional will probably be added later on; even though the rule syntax gives a way to define certain looping behaviour.

Today’s goal is to basically implement the concept of if the outputs (lhs) are more up to date then the inputs (rhs), the rule can be skipped. I don’t expect that to take to long, since most of it is just screwing with the rule expansions in order to properly stat the files. If I can get it done expeditiously, I’ll also have time to setup a few late binding `special` variables that will be useful, in proper Make fashion lol.

At which point, tmk will be effectively as good as a generic make implementation; and I can work on the parts that need to follow on. Namely the parallel support and portability aspects. The latter of course, being the main reason why CMake, SCons, and most every other such tool I’ve tried, has been rejected as more trouble then it’s worth.