Projects/Neo Ports Manager

Old regexese mastery revisted

by

For keeping my mind off things, and passing time while more up to date ports compile, I’ve spent some time working on neo (a replacement for my private updater.sh; itself a replacement for portmaster/portupgrade). When I had last left off, I was trying to integrate code lifted fro NPMs pkg_info wrapper, used for splitting canonical package names into the packages name and version numbers; when I noticed comically that the old kludge didn’t quite hold water. Kind of ironic, considering it was one of the most throughly tested snippets of code I have ever written lol. The principal problem was making it deal with names like libsigc++, pcre++, and libxml++—which contain regex meta characters (the +’s) and dealing with ()escapes breaking the algorithm.

So I just sat down and rewrote it, *properly* this time around, and it works like a blasted charm, in half the amount of code and a fraction of the execution time to boot. The only downside is between development and testing, I’ve spent several hours on it, and I’ve got to get up for work early tomorrow :-(. But hey, it’s not like I could have gotten a decent nights sleep anyway…. and I would rather have something like this catch my focus, then leave my train of thought to it’s usual ends.

neo is basically meant to replace updater.sh, but at the moment, I’m focused on replacing portversion—to make it worth using. Basically, I gave up on portupgrade and portmaster, and wrote a shell script that is more to my taste (updater.sh) some months ago. Using it is a simple thing:

# portversion -o | grep '<' | awk '{ print $1 }' > /tmp/pu && vi /tmp/pu
# /home/Terry/updater.sh /tmp/pu

leaving me to manually select and sort the list before starting the upgrade. Now I could easily insert filters to handle that kind of stuff in a highly flexible way, but if we’re gonna do that…. may as well ditch the portversion dependency, and that means ditching /bin/sh :'(. Making the current dilemma, converting the wacko zacko formats of version numbers, into something comparable in order to tell if it’s up to date or not. Technically, a straight compare could be done, since it obviously isn’t up to date unless it matches the version saved in the INDEX, but that precludes handling common edge cases.

A little $HOME, NPM, and EPI fun

by

Been cleaning up my home directory a bit, particularly ~/Projects and the dumping ground in ~/2besorted lol. I’ve managed to find a lot of old notes, even have imported a couple saved items into del.icio.us. While I’ve mostly dropped all my bookmarks, there are still a few files with old bookmarks in them. I merged them, using the old `cat files | sort | uniq > newfile` method ;). Sometime I’ve gotta shift through them and see what’s worth keeping. Since ma.gnolia’s great lossage, I’ve still have been maintaining the habbit of using Google and my brains limited RAM as a substitute for bookmarking >_>.

I’ve also taken some time to sort out the change over from Subversion to Git on NPMs source forge page. Last night, I started work in converting my updater.sh to Python… figured to dub it ‘neo’ and incorporate it as part of NPM. Hey, I’ve always wanted to remove portupgrade from the picture… lol. The neo script is basically meant to be comparable to portmaster or portupgrade, and whatever century time brings me back to Neo Ports Manager, will likely serve as the backend. However, I must admit, unlike NPM, this part is somewhat of a selfish agenda at heart: it will be able to do more then my updater.sh can without help.

Finally found the time to add a few pages to the wiki being setup with the Encapsulated Package Installer project. The stuff there is still mostly bare, because I’ve been focused else where’s the past couple of days. Mostly the content added was centred around development tasks, going to have to settle down and plan out what changes need to be made. All in all, things have been moving foreward. When it comes to my personal projects, EPI has the lead seat, and the rest are enqueued, in terms of programming.

That being said, of course I have my strokes of working on other things when shorter periods of time are free: the odds and ends, hehe.

the nights antics

by

I was rather happy to see that SourceForges SVN services were back up to a decent speed, other wise I probably would’ve gone to bed instead of getting back to work :

I managed to finish the work on the project file, I had inhaled the qmake manual before dinner ;-). I knew exactly what I wanted, test works all been done in a ‘one lump sum’ directory… Which is the style that I really hate, to much crap to visually grep through when looking for parts of the project.

The build format that I desire is basically like:

Makefile -> top level, non recursive make
doc/ -> documentation files
dist/ -> files ready to install (all done)
obj/ -> object files and crap from moc / building docs, etc
src/ -> source code files
src/module/ -> source code files for module

Which is basically what I’ve setup, maybe I’m just spoiled by the layout of FreeBSDs source tree and build system… hehe.

I was quite happy to find that as usual, Qt makes life easier not harder xD. It only took the fine (if short) manual and a little bit of trial time to get more or less what I wanted out of it, all the better! Without having to worry about a recursive trail of makefiles. When it comes to recursive makes, I have no problem with the concept, I just don’t care much for it.

I figured that if I couldn’t get what I wanted out of qmakes project files that I would have to do things manually. I don’t really have a problem with hand crafting makefiles but I really prefer not to have to maintain them, let along risk leaving something behind that needs a Bachelors of GNU Makeology to understand fully.

I did manage to get one module (essentially) imported into the SVN tonight and I’ve almost got another ready to go in. But before I commit it, I want to get the things refresh() slot sorted out. To do that, I just need to clear the m/v I guess but that can wait until tomorrow’ish.

As I often have said, laptops make poor pillows !

There’s about 5 more modules to import, as soon as they are adapted and ready to roll that is. Right now I’m in advantageous territory code wise but as usually, my working environment is, uhh… “Hostile” to say the least to productivity.

Maybe if anyone in this fuckin’ family knew an ounce of computers, let along programming, the bastards would let me work in peace rather then in pieces !!!!!!!

Source Forge SVN slow !!!

by

Bah humbug !!!

Tried to checkout a working copy of Neo Ports Manager over SVN, that took twenty minutes or so… Created an independent branch and the starting directory structure for the work I need to get imported, plus copied a few files from the trunk that I also want there.

That took at least 10 minutes to commit..

Looking on Source Forge, I can see that two other people have filed about the SVN services being so dang gum slow.

Looks like it might be awhile before I finally get my latest stuff under revision control +S.

Another commit to NPM

by

Well this was a fun one, for the first night in quite awhile I’ve spent some code-time on one of my own projects instead of [SAS] related business.

After someone on daemonforums pointed me at a few make targets (depends-list, showconfig). I put it on my to do list to refactor NPM to make use of such things.

As it turns out, depends-list is not documented and doesn’t seem to work on every port. The included makefile that implements showconfig however, does list a lot of make targets that I can count on (more then the ports manual page), including suitable *-depends-list targets.

NPM’s way of handling port build options in the trunk, I’ve never been very happy with… Because although it works, the whole thing hinges on a method of pulling out the OPTIONS variable from a specified ports Makefile. The code to do it, should we say gives me a queezy feeling when I think about it.

I also had planned on writing a small module to handle calculating a programs dependencies through the ports INDEX file but never finished it. I had gotten as far as processing the index and scratching my head over a couple of the rarely used segments of the file. The various m/.*-depends-.*/ targets solve that problem already, so we’re in cake land 😉

It’s much easier to just read the information through a pipe and process it into something manageable. I’m to exhausted atm to keep working on things in a sound frame of mind… Else I would probably be awake for another 3 or 4 hours, passing out around 0700~0800.

Some stuff I would like to do on the next round of commits,

process data received from the pipes while reading them, not after reading it all (not that this is likely to be a noticeable improvement in response time for big jobs)

figure out exceptions to be raised

get it finished

get it tested

get the program using it

test that

pass out again.

The return of NPMs Coding Spider

by

My first commit to Neo Ports Managers SVN since March… Sheesh I’ve been AWOL on this ain’t I?

Revision 61 – Directory Listing
Modified Fri Apr 4 07:22:12 2008 UTC (5 minutes, 2 seconds ago) by sas_spidey01

searchlet module now uses an internal searching code by default with the ability to use a separate external program for searching ports.

It assumes calling convention is ‘your command’ ‘what to look for’ and the following output parsing convention:

portname description

any white space will do.

ability to set external command not yet implemented [in the GUI I meant]

It’s been so long I had to change my password in order to login to SVN and the websites SSH successfully loool. Oh well, back to work I go!!

My official todo list:

Terry@dixie$ cat ~/code/Python/src/npm/trunk/src/todo.txt                  7:26

searchlet config in settings pages:
 external command
 fields to search for with internal command

finish the GUI for configuring port build options.

GUI for updating via portsnap/c*sup

help browser

   ... older notes
Terry@dixie$                                                               7:30

There is not a lot that actually has to be done before a beta can be released, it’s just it has to get done. Most of the rest is stuff I *want* done first…

I’m usually happier when I’m working on some thing then when I’m groaning about, and honestly I’d rather concentrate on code then my mind wandering off along past explicits.

by

Well, I have a todo list stored away on Neo Ports Manager and one of the things on my todo was to remove the dependency on the psearch program, namely because it would be faster to update our display as we find stuff in ports rather then parsing the output.

I think I might make another page in the settings to allow one to choose what fields to search in and possibly which to display as details. I plan on embedding the code to search ports right in NPM, but if it doesn’t clog things up I don’t see why not to allow the ability to use an external program. The best reason being if the index file ever changes or is ax’ed and I get hit by a bus, no one has to learn Python to fix NPM lol.

A couple months ago I sat down and started trying to figure out the format of the index file in ports, it was pretty easy and I wrote a small script to search it and print out all the cells of information in each entry of the index that had a matching package-version in it.

This is a quick test script I wrote tonight that searchs for a port and displays a short info on it, being a test I wrote the port to look for in the file test rather then passing sys.argv over to save time. I plan to adopt NPM to using a variation of this for searching ports now.


     1: #!/usr/bin/env python
     2: 
     3: # This is as much of the format of /usr/ports/INDEX-* file as I have
     4: # been able to reverse engineer by looking at it and a few other things
     5: # that didn't come with a nice fat manual page: the source is the documentation!
     6: #
     7: # Each program is listed on one line with each field separated with the pipe
     8: # symbol (|), list values are delimited by a space.
     9: #
    10: # pkg-ver|                   -> as listed in pkg_info
    11: # location in ports|         -> /usr/ports/editors/vim
    12: # install prefix|            -> /usr/local
    13: # short descr|               ->
    14: # path to pkg-descr|         -> /usr/ports/editors/vim/pkg-decsr
    15: # maintainer e-mail|         -> who@foo.com
    16: # listed in categories|      -> list of cat, e.g. cat1 cat2
    17: # dependencies to build|     -> list of pkg-ver required to build this port
    18: # dependencies at run time|  -> list of pkg-ver required to run this program
    19: # website|                   -> the programs website
    20: # |                          -> unknown
    21: # |                          -> unknown
    22: # XXX: the unknowns may relate to internal port opts, e..g use zip, etc
    23: ####
    24: 
    25: 
    26: import re, os.path
    27: 
    28: INDEX="/usr/ports/INDEX-6"
    29: 
    30: # find keys 1,2,4 in index row via regex
    31: def search(pattern):
    32:     idx = open(INDEX)
    33:     for line in idx:
    34:         keys = line.split('|')
    35:         k = (keys[0],os.path.split(keys[1])[1], keys[3])
    36:         for each in k:
    37:             if re.search(pattern, each):
    38:                 print 'found'
    39:                 mypp(keys)
    40:     idx.close()
    41: 
    42: def mypp(table):
    43:     print "PORT INFORMATION FOR: %s" % table[0]
    44:     print 'location: %s' % table[1][11:]
    45:     print 'descr: %s' % table[3]
    46:     print 'category: %s' % table[6]
    47:     print 'website: %s' % table[9]
    48: 
    49: 
    50: if __name__ == "__main__":
    51:     search("linux-flock")
    52: 
    53:


So close, yet so farfetched.

by

I’ve begun implementing Style 1 for NPM’s user interface but it is much less functional then style 2. I think style 1 is a lot more conventional, it’s based on an idea I had a long time ago. After I installed Qt3 bindings for Ruby I thought about writing a ports front end with that kind of layout.

I personally like style 2, it has a ‘orthodox’ file manager like style to it, since I’ve never gotten to use such a file manager a lot that might be why I like it lool. I stil need to write the loading code but that won’t be difficult in python.

It is however the most obvious thing in the world to me that as soon as my family is up and about there is not even a snow balls chance in hell of getting further work done.

Which as this case proves, means it’s time to kill some time until bed time… So I can get back to coding in piece and start work on my PC-BSD test machine.

Yet that dream still naggs at my mind….

by

I figure for today, I can probably work on updating my PC-BSD test machine to using FreeBSD 7, for updating the ports that will likely have to wait for later. With luck I can leave it running over night.

I want to spend some time working on the user-interface for NPM, I know what I want but I am not sure what value it will have. Essentially my idea is for the user to select from several implementations of the main window including the ability to use custom made modules for it.

I also want to implement an option by which each module is checked against a given list of checksums, if the module fails the checksum it won’t be allowed to load. With this method one could theroectically restrict any non-NPM supplied window layouts from being loaded but it’s probably a useless feature but one I’d like to tinker with just for the fun of it lol.

I also know if I don’t get a few things out of my head I’m going to pop.. Maybe coding will help :.

NPM going Beta soon !!!

by

I’ve spent the last 3 hours or so hashing out changes in NPM’s sources.

I’m not totally satisfied with the look of the settings dialog, I don’t think it looks professional enough for a proper release… Maybe it is just me. This is one reason I’m axious for the Qt4/KDE4 port because I’d like to get my arms around some of the changes in layout but I guess it can wait.

I still need to come up with icons for the menu to replace the text labels (ports, logging, network) but I think this is a good enough settings front end for now. I took a quick screen shot of each of the pages that show the preferences that I’ve setup so far:

[click to enlarge]
Free Image Hosting at www.ImageShack.us

Free Image Hosting at www.ImageShack.us

Free Image Hosting at www.ImageShack.us

That should take care of the basic user settings for now. I really think it is time I start to focus on the back end slots. Because as much progress as I have made in the last several commits of getting the GUI to work with the settings system I never got around to writing code to get it to make setting changes persistent through the GUI :o)

Once that OptionsInterface and related code is done and interfaced with the code in portactions.py I can work on the next phases I need done before a beta release:

  1. Implement the GUI front ends for running portsnap/cvsup/csup, atm it should run but without user output and a start button lol.
  2. Write the about dialog and integrate access of the Options GUI (the above screen shots show it) into NPM_MainWindow (the main application window).
  3. Alternate NPM_MainWindow forms that can be loaded at user choice 🙂
  4. Tweak code for setting ports build options more

After that, I guess Neo Ports Manager will be ready to have a Beta Release.

The handling the dialog that stands in for make config, I know sooner or later I’m going to have to resort to calculating a full scale depends list of options much like make config-recursive does because doing otherwise would screw up during any operation that tries to do a make config: the display area is read only to the user, no I/O form user to portupgrade and friends. Fixing that will come later… I also need to get my tookus back to the German translation when I’ve got the time to spare.

Oh what fun it will be when NPM hits a 1.1 Release, I can finally have a coffee break of sorts <_< hehe.

I would do more tonight but work is in the morning… Thus time to go to bed when you consider it’s after 0330R [local morning], well at least this is the earliest I’ve gone to bed in awhile LOL.