Python

Todo: Toolkit and Library madness

by

Examine for portability the following toolkits under these OSes: Windows NT, Mac OSX*, Unix-like.

*I sadly don’t have a Mac, although I would really love one. So a OSX binary of some form will have to pass.

GTK+ — C, C++, Perl, Python, Ruby, PHP, C#.NET (Java bindings also avail but seem to be gnome-centric)

Qt4 — C++, Perl, Python, Ruby, Java (C#.NET and PHP bindings seem to be questionable and only C++/Python and possibly Java bindings can be depended on).

WxWidgets — C++, Perl, Python, Ruby, Java, C#.NET

Swing and SWT would be considered if they were available under more languages !!!

Evaluate portable standard/add-on libraries to each language for the following capabilities:

String handling (they all do well in the language standard)
Regular Expressions (Only Perl and Ruby get this great imho)
XML Processing
Network programming — both sockets based API and protocol support (IMAP, POP, HTTP/S, FTP)
Database handling — must support MySQL, SQLite, and some form of flat file.
Basic compression and archive format support (e.g. tar; zip; gz; bz2; lzma)
Inter-Process Communication (IPC) methods and related process control (e.g. fork(), exec(), signal(), kill() type routine-families).
Ease of use and deployment

Goals:

Build up a standard frame work of toolkits/libraries/etc that are portable across both my general operating environment and the various languages I use

Attempt to standardize myself on few languages rather then rubber banding between various programming languages + sh/awk/sed/friends

Ensure full development environments are available to me under FreeBSD 7 and Windows NT 5.1 (e.g. I can code from either machine and not change tools)

At least one language for scripting/prototyping and one for more efficient execution when implemented in that language (e.g. 1 interpreted + 1 compiled)

and try to stay sane along the way without writing a few libraries in the process >_>

Something that has been on my mind of late

by

DRAFT POST

I very much need to start standardizing myself on a smaller set of tools. Maybe tools is not the best choice of ord, o much as it is a question of languages and libraries. It’s actually kind of ironic, I really would like to learn several more languages (Ada, Fortran, C#, Erlang, and I never did wrap my head around Scheme).

C++

Pro’s:

can do low level stuff

can interface well with C code and libraries

Qt is natively C++ and supports everything I need it to run on (in Qt4 OSE hehe)

Fairly portable and standardized (at least much of the 1998 standard…) with several compilers available.

Mainly toolkits and frame works available, both native and from C.

Widely used and my first language

Con’s:

having to dig into lower level interfaces is very error prone when doing it with a headache the size of California.

Often not my first choice for building a prototype *quickly* but good for final implementation.

Java

Pro’s:

Inheritable portable between JVMs of the same implementation (e.g. Sun JVM for Win and Sun VM for Mac can run same code).

Simpler OOP syntax then C++, imho

The Swing GUI toolkit is fairly portable and SWT sufficiently supports the platforms I want to avoid skipping.

I like the way it handles exceptions, and usually like checked exceptions — when a class is designed appropriately.

It’s well defined if not perfectly standardized and compliant implementations are fairly common enough (Sun’s)

Wildly used and my third language.

Well documented (if a bit boring)

Con’s:

Everything is OOP….

I prefer native code to waiting on java to load

Requires a suitable runtime

Conventions expected by some tools can occasionally be irksome (to me)

C# (C Sharp)

Pro’s:

It’s not Java

It’s similar to Java/C++

Core elements are standardized

Less resistance to shooting oneself in the foot / doing stupid things then Java but not as much as C++.

Gtk#, the binding of GTK+ to C# is available on the platforms I care about.

Con’s:

Most C# applications are probably tied to Microsoft Windows implementation via .NET or through Gnome related interfaces

Requires a runtime (e.g. .NET, Mono, or Dot GNU)

Requires just enough learning of it’s differences from Java, for me to use it.

PHP5

Pro’s:

Portable and interpreted

Widely used across the WWW for server side scripting.

GTK+ bindings to PHP available and portable’ish

Simple language and member of the CBF.

The OOP increasingly resembles Java syntax and is just “enough” syntax to be useful without cramming.

Well documented

Con’s:

Local php.ini files can cause problems (how much can you assume?)

Needs a run time (php) of the right version (5)

php4/php5 is less common off web servers or developers systems

Python

Pro’s:

Great for writing prototypes

Several toolkits available that are fairly portable

Easy to work with and quit portable (and issues of portability well noted in the docs)

Large standard library

Implementations for the Java Virtual Machine and Microsoft .NET framework are available (if not as current as CPython) as well as the standard (C)Python implementation.

Con’s:

Requires a run time

I *hate* it’s handling of regular expressions after being so used to Perl…

It’s slower then native code

Interfacing with C /or C++ code can, uhh… Get interesting, from a portability perspective.

The program is the source

Ruby

Pro’s:

Handy pure OOP language

Great handling of regular expressions within the language itself, as opposed through importing an object oriented interface (i.e. as in Python/Java/C++)

Usually very “comfortable” to write.

Large standard library

Con’s:

It requires a run time and alternative compilers (e.g. xruby, ironpython) may be lacking in reliability or cause some features to be unavailable

The program is the source

Wishing Ruby 2 would come sooner….

Documentation can be irksome at times

It’s slower then native code and even slower then Python (Ruby 1.8.x)

Does Python or PyQt3 have a pox on C++?

by

This is illogical as I’ve seen Python get.

Terry@dixie$ cd code/C++/src/qsm                                           0:11
Terry@dixie$ python                                                        0:11
Python 2.5.2 (r252:60911, Apr 17 2008, 16:34:02) 
[GCC 3.4.6 [FreeBSD] 20060305] on freebsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> from qt import *
>>> QDialog
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'QDialog' is not defined
>>> 
Terry@dixie$ cd $HOME                                                      0:11
Terry@dixie$ python                                                        0:11
Python 2.5.2 (r252:60911, Apr 17 2008, 16:34:02) 
[GCC 3.4.6 [FreeBSD] 20060305] on freebsd6
Type "help", "copyright", "credits" or "license" for more information.
>>> from qt import *
>>> QDialog
<class 'qt.QDialog'>
>>>

Maybe there is a reason for this and I’ve just got to much of a headache from today to notice it, but WTF man? lol. It’s almost as if Python has a pox on running a script in that directory :

I wanted to test an idea for a dialog quickly, so I started a new python script. Only to have NameError’s whenever trying to reference a member of the qt module. So i copied over one of the py-qt3 example programs that came pre-installed and tried that, same barfing results.

Then I tried interactively with python, no go — changed directories to home and tried both, and they fucking work !!!!

I don’t know whether I should laugh out loud or start hex editing maliciously lol.

by

Well, I have a todo list stored away on Neo Ports Manager and one of the things on my todo was to remove the dependency on the psearch program, namely because it would be faster to update our display as we find stuff in ports rather then parsing the output.

I think I might make another page in the settings to allow one to choose what fields to search in and possibly which to display as details. I plan on embedding the code to search ports right in NPM, but if it doesn’t clog things up I don’t see why not to allow the ability to use an external program. The best reason being if the index file ever changes or is ax’ed and I get hit by a bus, no one has to learn Python to fix NPM lol.

A couple months ago I sat down and started trying to figure out the format of the index file in ports, it was pretty easy and I wrote a small script to search it and print out all the cells of information in each entry of the index that had a matching package-version in it.

This is a quick test script I wrote tonight that searchs for a port and displays a short info on it, being a test I wrote the port to look for in the file test rather then passing sys.argv over to save time. I plan to adopt NPM to using a variation of this for searching ports now.


     1: #!/usr/bin/env python
     2: 
     3: # This is as much of the format of /usr/ports/INDEX-* file as I have
     4: # been able to reverse engineer by looking at it and a few other things
     5: # that didn't come with a nice fat manual page: the source is the documentation!
     6: #
     7: # Each program is listed on one line with each field separated with the pipe
     8: # symbol (|), list values are delimited by a space.
     9: #
    10: # pkg-ver|                   -> as listed in pkg_info
    11: # location in ports|         -> /usr/ports/editors/vim
    12: # install prefix|            -> /usr/local
    13: # short descr|               ->
    14: # path to pkg-descr|         -> /usr/ports/editors/vim/pkg-decsr
    15: # maintainer e-mail|         -> who@foo.com
    16: # listed in categories|      -> list of cat, e.g. cat1 cat2
    17: # dependencies to build|     -> list of pkg-ver required to build this port
    18: # dependencies at run time|  -> list of pkg-ver required to run this program
    19: # website|                   -> the programs website
    20: # |                          -> unknown
    21: # |                          -> unknown
    22: # XXX: the unknowns may relate to internal port opts, e..g use zip, etc
    23: ####
    24: 
    25: 
    26: import re, os.path
    27: 
    28: INDEX="/usr/ports/INDEX-6"
    29: 
    30: # find keys 1,2,4 in index row via regex
    31: def search(pattern):
    32:     idx = open(INDEX)
    33:     for line in idx:
    34:         keys = line.split('|')
    35:         k = (keys[0],os.path.split(keys[1])[1], keys[3])
    36:         for each in k:
    37:             if re.search(pattern, each):
    38:                 print 'found'
    39:                 mypp(keys)
    40:     idx.close()
    41: 
    42: def mypp(table):
    43:     print "PORT INFORMATION FOR: %s" % table[0]
    44:     print 'location: %s' % table[1][11:]
    45:     print 'descr: %s' % table[3]
    46:     print 'category: %s' % table[6]
    47:     print 'website: %s' % table[9]
    48: 
    49: 
    50: if __name__ == "__main__":
    51:     search("linux-flock")
    52: 
    53:


Beware of Pythons

by

One sad side effect of using Python so much the past few months..

I actually had to invoke GDB’s ‘help’ command when debugging some C code tonight :

I know several languages but I tend to do things in “the language of the moment”, if I’m working on some thing in C, most of what I do on the side will usually end up in C. Like wise if its a project in Ruby, ruby scripts start filling up lol.

Maybe my brain could use an extra 256K of memory?

Proof of Concept: Randomize a list

by

Well, the idea was essentially that RvS allows up to 30~32 maps or so all stored as a pair of assicated arrays loaded from the servers configuration file. One array that holds the game types (e.g. tango hunt, hostage rescue, misson et. al.) and the other the actual maps to load, i.e. GameType[0]=first maps gametype and Maps[0]=first map to load.

I took a moment the other day (before my gfx card went fuzzy) to write a small program that would generate a new configuration file with a maplist specified from another file, the prototype works fine but I didn’t know how I could archive my real goal: randomly populate the maplist from that file.

Here is a proof-of-concept algorithm that takes a list of items and returns a new list containing the same values but sorted into a ‘random’ order.


#!/usr/bin/env python

import random

def unsort(list, index):
    "return a copy of list reordered randomly"

    new_list = []
    table = []

    while len(new_list) != index:
        r = random.randint(0, index)
        try:
            if r not in table:
                table.append(r)
                new_list.append(list[r])
            else:
                continue
        except IndexError:
            pass
    return new_list

if __name__ == "__main__":
    li = [1,2,3,4,5]
    print unsort(li, len(li))

This took me about 10 minutes of thinking at work today and about 12 or 14 hours later I’ve made a test script in Python since that is the language I’ve been using for stuff latly. Basically the problem is the only way to randomly map items from one list into another is if you access elements in the list at random. But if you access a list at a random index and push it into a new list you can get duplicates. So a table (list) is used to store random numbers we have already generated. If the current number is not referenced in the table we can assume that we have not copied the element at that index in the list into the new list, if it does exist we need to try again. One bad thing is the algorithm is designed to run until completion, which can take an arbitrary amount of time to ‘unsort’ any given set. In theory, it could loop forever! However it would be trivial to modify it to abort the randomization process after a given time frame (such as >N iterations or seconds, which ever comes first) and just append remaining elements onto the end of the new list.

Here is a few test runs:

Terry@Dixie$ ./unsort.py
[3, 5, 4, 1, 2]
Terry@Dixie$ ./unsort.py
[3, 5, 2, 1, 4]
Terry@Dixie$ ./unsort.py
[4, 1, 2, 3, 5]
Terry@Dixie$ ./unsort.py
[5, 4, 1, 2, 3]
Terry@Dixie$ ./unsort.py
[5, 2, 4, 1, 3]
Terry@Dixie$ ./unsort.py
[3, 2, 1, 4, 5]
Terry@Dixie$ ./unsort.py
[4, 5, 3, 2, 1]
Terry@Dixie$

Odds are the PRNG is being seeded with the system time by default or making use of /dev/random on my FreeBSD laptop and that is good enough for me. And these results are random enough for me because I dunno how to write first class pseudo random number generators hehe.

snoring through logging

by

been working on a simple logging module for npm, so far I’ve set it up to handle 3 different outputs, the usual STDERR for big problems, a regular log file for every thing but debugging, and a developers log file for use during debugging.

stderr
CRITICAL: critical message
ERROR: error message
WARNING: warning message
run log
2007-12-27 07:27:45,156: CRITICAL -:- critical message
2007-12-27 07:27:45,157: ERROR -:- error message
2007-12-27 07:27:45,157: WARNING -:- warning message
2007-12-27 07:27:45,158: INFO -:- info message
dev log
2007-12-27 07:27:45,156 root CRITICAL: logger.logger.py.73
         critical message
2007-12-27 07:27:45,157 root ERROR: logger.logger.py.74
         error message
2007-12-27 07:27:45,157 root WARNING: logger.logger.py.75
         warning message
2007-12-27 07:27:45,158 root INFO: logger.logger.py.76
         info message
2007-12-27 07:27:45,159 root DEBUG: logger.logger.py.77
         debug message

The formats are just a matter of what I find easy on the eyes, think the middle one could stand with a shorter date format.

I might ditch the idea of a second log file and just follow my original idea of being able to specify the level of logging to use. It’d also be less work then

Either way yee slice it, I need some freaking sleep… because I can’t think no more.

:wq!

A short look

by

Well, almost missed my narrow window of code time… Was passed out on the couch with a full stomach before bed until nearly 0100 xD

Between My mom, my sister, the bird, and my nephew (a louder bird)… And helping with baking there is really not much I can do during the day.. Any attempt at even thinking about seriously trying to read or write goes out the window and in comes a headache. So I have to work at night, when every one else is alseep… Until I crash or the clock reaches a point where I need to go to sleep in order to be @ work on time.

I completed the mock up of a simple dialog for simulating make config. And I’ve almost finished a working prototype for the module but I don’t have time to handle writing the slots and associated code to deal with the check boxes right now. And I need to get that done and tested before I can incorporate the prototype unto the module it belongs.

I posted a screen shot of the mock up awhile back. I’ve fixed the display a bit (the leading variable name from the makefile is removed). And added an ok and cancel button with the beginnings of a more key-board friendly behavior when it comes to using the keys instead of the rat to use it hehe. I’ve also set the caption on the dialog to contain the ports name as category/program.

In doing this, I figured the most simple way was to just muck around with a string of our target (e.g. /usr/ports/net/samba3 and /usr/ports/www/links/ in my tests). Although now that I think of it, the same routine I wrote to get a list of options to create check boxes for. Could be used to look up the category and portname variables from the makefile instead because it’s not tied to looking up any specific variable. — TODO: compare both methods for speed
after the prototype is ready to be moved to alpha group.

In trying to figure out how to get the desired result from a string version of our working path, I remembered that most languages offer some way to obtain the basename of a file or directory and set to look for it in Pythons os module. But doing that would mean we’d have to cut off the trailing slash and refeed the path to the library routine then join them into a new thing. Not exactly my idea of fun when writing a quick helper subroutine. I was very happy that help(os.path.split) revealed a method that takes care of most of that legwork itself.

Python also has some thing called list compression and which allows the mapping of the contents from one list into another based on given criteria. I’ve generally avoided list compressions out of disfavor for some of the (large) examples I’ve seen before. But for this, I actually found it yielded both quick to write and easy to understand code.

p = name # '/usr/ports/category/program' for example
prog = os.path.split(p)
cat = os.path.split(prog[0])

path = ["%s/%s" % (c,p) for c,p in zip(cat, prog) if c and p]
return path.pop()

The above snippet splits name into two lists of two elements, each a string. Basically the 2nd line returns a list of [‘/usr/ports/category’, ‘program’]. So obviously line 3 likewise splits /usr/ports/category into a list of two elements, the last of which is category. I really love how os.path.split() speeds up reading this, it takes only a glance to read it without having to double check it.

The 5th line maps any elements of the two lists (cat and prog) into a single a list of strings, i.e. some thing like path => [ ‘/usr/ports/category’, ‘category/program’ ]. The test for each element being true could be ommited in this case but I wanted to show the possibility.

Since path is a list and we want to display a string (without using str() on the result!), we just pop() the last element off path, which is exactly the element we want ‘category/program’

mm, now for some sleep… It’s after 5am here and I’m exhausted.

Enter the Dragon

by

Well, so for work goes well for day one.

I’ve spent most of the day working on the configuration system, at this point it would be a lot easier to have a configuration file, so I set about to work with Pythons SafeConfigParser class and the format of the config file and options, e.t.c. There is still more to do but it is a great start, especially considering that I am not entirely comfortable with OOP under Python

I’d say greater then half the code for installing and updating the ports tree is done enough for further testing, the only issue right now is the reaction to errors sup’ing or portsnaping.

I am some what tempted to try and have a wrapper of sorts that will abstact the issue of using QT or KDE specifics but I don’t think there would be any point. I need to dig deeper into the PyQT/PyKDE, QT, and KDE documentation soon so I can work more gui_error.py.

widets ? qt : kde;

It would be nice to use as many elements of the KDE bindings as possible, so it fits better with systems running KDE, yet it would also be nice to keep to the QT bindings more strongly so it is less tied to KDE.. In the end it will probably be which ever I’m more comfortable with.

My primary goals in the coming days is to finish the options and errors modules while I start work on the ‘searchlet’, which should handle searching through ports. Until a concreate mock-up can be made of the entire main window, I think I should keep the search stuff as far away from the rest of the program as I can.

Work on the searchlet as I call it, shouldn’t be to hard to keep self-contained since I prefer to work on smaller pieces and properly prototype things when I can, I’m not a real fan of monolithic masses =

Basically 7 tenths of that battle will be wrapping my head around working with the KListView and QListView widgets. Once I’ve figured that out, it’ll be simple enough (I hope) to figureout how to make it work with displaying a search through ports. It’s just I need to get a grip on the ListView widgets before I can do that lol.

Either way, I think I have done enough for one day… 0700 Zulu Time and it would probably be nice to have a little sleep before work tomorrow 0.o

A little horsing around before bed, not pretty but considering I don’t know QT from a hole int he ground… And I’m doing this with C++ documentation for Python, I think it’s a nice ‘Hmm, how do you make a QListView object?” test:

#!/usr/bin/env python
# Neo Ports Manager (NPM) -- refer to the LICENSE file for terms and conditions

"""

"""
import sys, os
from qt import *


def main( argc, argv ):
    a=QApplication( argv )
    listViews=QListView()
    listViews.resize(640,480)
    listViews.setCaption("Qt Example - Listview") # Sets window title
    # Add some columns to the list view
    listViews.addColumn('Qualified name') 
    listViews.addColumn('Namespace')
    # element is how to create a new list view item for display
    element = QListViewItem(listViews, 'qName', 'namespaceURI') 
    # Now lets populate an array of items into view
    els=[ [listViews, 'dir1', 'descr1'], [listViews, 'dir2', 'descr2'],
          [listViews, 'dir3', 'descr3'] ]
    for j in els:
        QListViewItem(j[0], j[1], j[2])
    # And sort it descending by col 1
    listViews.setSorting( 1, False )


    a.setMainWidget(listViews)

    # draw & exec
    listViews.show()
    a.exec_loop()


if __name__ == "__main__":
    main(len(sys.argv), sys.argv)

I don’t like that QListViewItem(j[0], j[1], j[2]) line but I was trying to translate from the C++,

for ( int i = 0 ; i < attributes.length(); i++ ) {
    new QListViewItem( element, attributes.qName(i), attributes.uri(i) );
}

And I’ve yet to figure out how to better cook this up, still reading the docs here…

backlog

by

This one made my day 🙂

[15:11] [SAS]_LCpl_DUKE: Did you know: The Navy takes now only not-swimmers.
[15:11] [SAS]_RSM_Spidey01: ?
[15:11] [SAS]_LCpl_DUKE: Yeah, they defend the war-ships longer…
[15:11] [SAS]_RSM_Spidey01: LOL
[15:12] [SAS]_LCpl_DUKE: the newest joke i got 5 minutes ago… 😉

I’ve been brushing up on Python for a little projectI’ve got in mind, so I’ve been cramming like a student at finals week. Python is a nice language but one I never put to use when I originally started learning it.

The thing I love the most about Python, is the doc strings, it’s such a great idea if you ask me.

--- def test(params):
...     """ Simple test
...
...         More detailed info about it"""
...
...     print params
...
--- test.__doc__
' Simple testntnt    More detailed info about it'

A poor example yes, but in actual usage the doc strings can be very useful. I also like it because it can be used to move inline-documentation (some times aka comments) into the body of the function, rather then being above it. And they are accessible at runtime with the __doc__ special method.

One thing I do wish Ruby had, is Doc Strings for methods, b/c then in IRB I could do a simple obj.methods.sort and follow up with checking a methods docstring rather then an intermixing of playful testing in IRB with looking it up in TFM.

With Python, it’s quite easy to go to interactive mode and use dir(obj) to get the methods an object responds to and any available __doc__ special methods to learn more about it, before we go RTFM hehe.