Python

by

As a young programmer, I think it was probably around 2.4 or 2.2 that I started to use Python. Today it remains one of my favorite languages.

I remember thinking quite highly of the Zen of Python at the time. Much time as gone on since then, and to be frank, I think it even more beautiful and dead on balls accurate today than I did then. The older I get as a programmer the more accurately it reflects the reality.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

Solving the wrong problem?

by

Programming language Python’s ‘existential threat’ is app distribution: Is this the answer?

I kind of can’t help but wonder if this is really about solving the wrong problem.

In dealing with the developer side of things: pip and venv really aren’t that bad compared to some of the squirrelly means of distributing software the world has known. But much beyond ‘type pip install xxx and cross your fingers’, I wouldn’t really call it a user oriented system. It works well enough for Python developers but is not catered to Joe Average User or twelve year olds who just want to blow stuff up.

To make things ease on end users of course: you have to solve the actual problem. Linux has a good rule about not breaking userspace–but userspace doesn’t care about you! Personally I think that is the real pickle.

Over in NT land it’s pretty simple. You build some shit and the system ABI is probably the same across a bunch of Windows versions if you’re not too crazy, and most of the baggage is the problem of your install creation process. Whether that’s some fancy tool or some rules in your Makefile. It’s impressive when you load up a video game that’s old as hell and it just works, despite being almost old enough to buy a beer. It wasn’t made to be efficient: it evolved to become stable. It grew up in a world where people shipped binaries and pushing changes to users was costly.

Now by the time you have an actual Linux desktop distribution: all bets are pretty much off. A decent one will usually maintain a viable ABI for a major release number but that doesn’t mean everything will be compatible forever, nor does it mean the binary dependencies you require will remain in xyz package repo for the next fifty years. Some of this lands on distributions and how they deal with package management to squeeze binaries into nixie norms of hierarchy. Some of this also lands on developers, who may or may not know what an ABI is from a hole in the ground because they’re used to recompiling to APIs and configuring ten thousand build time knobs and don’t care that changing something impacts binary compatibility between their library and the works of others.

There are reasons why things like AppImage and Flatpak exist. Many of these I think owe to the source centric nature of unix systems. Different communities have different norms of sharing and reuse.

When I began learning unix systems, I chose a source centric flavour that would let me learn how things worked under the hood. The kind where you waited three and a half days because a new version of KDE or GNOME landed and many a dependency in the food chain needed to be rebuilt. The kind where you learned to diagnose linker problems and grumble knowing that changes to library X meant recompiling half your environment if you wanted to be sure your applications didn’t combust quietly in a corner just waiting for the day you actually needed to launch them again, or curse at some major framework linking to some piddly library that triggered same.

In the end my times with that system were dominated by two things: stability and compile times. But I didn’t chose that in order to have an easy one click and done system. I had chosen it because I wanted to learn how computers worked and develop the means of figuring out why the fuck programs broke. Today if you use that flavour of unix, you can pretty much live a purely binary world that wasn’t so easy when I was a padawan.

By contrast an acquaintance of mine back then, ironically a Python programmer, had chosen a more widely known distribution that focused on having the latest binaries available without having to compile all the things. One that’s still quite popular about ~15 years later. Let’s just say the usability of binary focused distributions has improved with time despite the warts that is binary distribution in *nix land. Or to summarize it thusly:

When it came time for a major version upgrade: I spent a few days compiling before getting back to work. He spent a few days cursing and then reformatted, lol.

The STAMAN Project: Phase IV,

by

Having thought of tasks and storage formats, it’s now time to figure out an implementation language, i.e. what programming langauge am I going to write the task manager in.

O.K. based on what we’ve got so far, it is easy to infer the following is worth having:

  • Portable between systems—a must for me 😉
  • Easy access to SQLite—usually trivial.
  • Better tools than gmtime().
That means this page is rather useful, for what languages can be ruled out. In my répertoire this means Go (aww), Scheme, AWK, and shell languages can be skipped. Reason being the portability of Scheme bindings in general, and the others lacking sufficient portability (for my taste) at this time. That still leaves about 13 languages, lol. PHP, Java, Lua, JavaScript, and X86 assembly are easy for me to rule out. Reasons for that can all be easily guessed; at least if you remember how much I enjoy Suns Java tools. JS/Lua are great choices but I don’t want to screw with the bindings and stuff.

I’m not very interested in compiling SQLite in C/C++ on Windows, or the CLI binding everywhere. So this effectively makes the choice Perl, Python, or Ruby. Out of those three, none is perfect either: perl doesn’t come with the required database code, it just has the definitive interface for databases everybody mimics. Python and Ruby on the other hand, come with SQLite bindings—which many distributions separate out into separate packages. It’s just a lose, lose situation when you think about dependencies, but it does beat writing your own everything for every program. Sometimes. Setup with these three dynamic languages would be easy though, in so far as we’ve gotten with the above.

Time handling is another issue. Perl has fairly minimalist handling of time built in, but on the upside, if you need it, it’s probably three abreast on CPAN. Time::Format and the core Time::Piece module each come to mind. What isn’t built into Perl, often comes with it or can be added to it. Ruby provides a simple but effective Time class, that makes for more natural code than you may expect. More complex operations will require Googling for a Ruby gem, or hand coding it on demand. Python on the other hand provides a comprehensive datetime module, and supporting time and calendar modules, all out of the box! I would say Python takes the lead here.

Rule one of getting work done: know how to leverage libraries.

In terms of programming languages, Perl, Ruby, and Python are generally equal enough for most tasks, so long as you don’t shoot yourself in the foot. Some subtle differences that personally irk me:

  • Perls autovivification can be almost as much a miss-feature as it can be a convenience. You’ve just got to learn the damn language :-P.
  • Ruby functions are not first class objects! Some things can also be weird if you’re not used to Ruby.
  • Python doesn’t always stand up well to typos, especially if they involve indentation o/.

Because of how many lines of code I’ve done in Python over the years, I am more familiar with it’s set of “Irks” than Ruby’s, like wise I know Sh, C, and Perl more intimately than other languages, so I really know their irks. For perl, it’s mostly thin things that get in inconvenient when combing the warnings pragmata with the nature of perl syntax. They spiritually conflict at times. Under Ruby, I mostly find gripes that have a bigger place in programmer culture. My issues with Python generally have to do with trade-offs that I disagree with as a matter of my convenience, even if it usually results in a Good Thing overall. It comes from a C-oriented background meshed with a love for the Perl programming language.

This is a fact: you will always be irked by your programming language, if you use it enough. What can I say, nothing is perfect. Shoot!

For this particular application, there’s some things worth noting also: language portability. If the machine doesn’t run perl, it’s not a real computer. Most systems you’re likely to care about will run Ruby and Python, and there’s probably a crusty old version of Python for those you don’t (nor directly should). In contrast however, Perl is often a lower level of “Cross platform” behaviour than Ruby/Python. You’ll find this highlighted well in the Camel book. One reason I use Python frequently, it always behaves as expected without so many subtle hiccups.

How much this pertains to the current matter, i.e. implementing STAMAN. Perl is the most universally available language, and I’m more prone to need such a feature than most people. A plus over Ruby is no crappy 1.8.x/1.9.x porting issues…! Of course however, I have a camel to ask about minor details, hehe. In my experience the Python 2k/3k thing is less issue than Ruby’s for writing code yourself, more of an issue in leveraging existing code.

So I reckon, that means Perl or Ruby is best called for here. I exclude Python, because I just use the frick’n thing to often.

My dumbest python moment ever….

by

In cleaning up tmk’s cache related code for a fresh commit, I wrote an expand2str() method that encapsulates the issue of dealing with expand() returning a list of expansions, and a properly converted string being desired anyway.

Suddenly I noticed this backtrace:


Traceback (most recent call last):
  File "tmk.py", line 929, in 
    process_recipe(recipe)
  File "tmk.py", line 742, in process_recipe
    recipe.parse()
  File "tmk.py", line 276, in parse
    if not self.eval_pproc_directive(p):
AttributeError: Recipe instance has no attribute 'eval_pproc_directive'

Which is totally ridiculous, because eval_pproc_directive and parse are both methods of the same class. While the former is defined after the latter, by the time the instance (self) exists, the class is fully defined. Making a short test case, proved that the act of one in a billion hadn’t changed Pythons rules about this stuff.

In poking around to see what changed introduced during this commit, may have popped the magic cork, I noticed removing the reference caused the same type of error, successively on methods defined after they were invoked.

Then I saw it!

I had accidentally indented the expand2str() method one short, there by making it a function rather then a class method, and there by doing like wise and making them nested methods inside expand2str().

Sometimes Python really irks the typoist in me!

This is so recursive, it hurts

by 

# this is so recursive, it hurts
for topdir in args:
    for root, dirs, files in os.walk(topdir):
        for file in files:
            for regexp in patterns:
                if re.search(regexp, file):
                    file = os.path.join(root, file)
                    if verbose:
                        sys.stdout.write("file name: %st" % file)
                        p = os.stat(file)
                        a, m, t = (
                             time.strftime("%Y-%m-%dT%H:%M:%S",
                                        time.gmtime(p.st_atime)), 
                             time.strftime("%Y-%m-%dT%H:%M:%S", 
                                        time.gmtime(p.st_mtime)), 
                             time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime()), )
                        sys.stdout.write( "atime: %stmtime: %stnow: %sn" %
                             (a,m,t))
                    else:
                        print file

What a huge, ugly, monster, I’m glad the finished code doesn’t look like that lol.

Slithering around an aging question

by

Hmm, it seems if I can sort a suitable SQL-interface (beyond the standard issue sqlite3 module) that Python will become my primary language.

It seems, no matter how much I try to circumvent it, Python is the best choice all around : And I’ve tried to knock it down to size lool.

Pythons str and list classes, I actually quite like; although I don’t care much for dict (Pythons associative array class), that might change come Python 3. Most of the stuff built into Python, I like using. The regular expression support, well what can I say… I don’t like any stuck on OOP or procedural interfaces, after having used Perl. The =~, s//, [m]//, and related $symbolic_vars in Perl are just to useful!!!! But Pythons ‘re’ module is good enough for stuff, I guess. Between the built in xml.* and the third-party lxml interface to libxml2, that routes pretty much a snap. I get along well with Pythons IPC offerings about as well as I do with posix’s C offerings. The zlib, gzip, bz2, zipfile, and tarfile modules are enough for the compression/archive formats I may eventually need to operate on. For network related operations, Python provides a lot of stuff plus a familiar sockets interface; I don’t have no problems with sockets programming (although I rarely get a chance to use it in C), I just don’t have time to say, implement FTP! For GUI toolkits, basically offered is TKinter, GTK+, WxWidgets, and Qt. And it looks like GTK+ will probably be my new default kit… The epydoc program also seems to provide the style of documentation I’m used to (Javadoc-style) with good options hehe.

Not to mention the best parts of using Python as my standard language (I and II)

This is how I relax…. lol.

by

Having taken some time to analyze the mailcap and mime.types files, which map media types to handler programs and file extensions respectively. I’ve decided to implement something, uhh…. more “fun” out of it, for my own usage. I’ll use mailcap and mime.types to setup a suitable mime/file associations to desire, and a mixture of m4 and environment varables to make it more useful hehe. I’ve also setup a small wrapper around Pythons mailcap and mimetypes modules, so I’m happy !

Since I need a handler that includes a web browser, rather then creating an if x is probably running … script, to just use $BROWSER. But since, there is a shitload of apps that don’t understand it, I’ve created quick python script to handle it. Thus, mailcap will execute this script, which will “figure it out”. Once I realized how much I could use such a quickie, I added a little more abstraction to it; making it usable with other variables. And used getopt for a simple interface to it.

#!/usr/bin/env python

import sys, os, getopt

# default values for -e and -s options
envvar = "BROWSER"
envsep = ":"
# skip executing program 
noexec = False

# accepted options, usage statement, and manual page
shortopts = "hme:s:n"
longopts = [ 'help', 'man', 'environment=', 'separator=', 'no-exec' ]
def usage():
    print """
usage: %s [options] file ...

    Parse a field delimited list of programs, executing each in turn, until the
    specified application has opened the indicated file(s).

Options:

    -h,--help    display this useless message
    -e,--environment the environment variable to look up, default to BROWSER
    -s,--separator    the field separator for -e VAR, defaults to :
    -m,--man    view this manual page
    -n,--no-exec    do not execute any programs, useful if parsing $PATH

    --long-options may be abbreviated.

Exit Status: 

    The return status of the last executed program handler. If no program was
    executed, return with a non-zero exit status.

Caveats:

    A non-zero exit status is considered a successful execution.

    Programs found by parsing the environment variable, are passed onto the
    system shell UNMODIFIED. This is not secure, but it is flexible.
""" % sys.argv[0]
    sys.exit(1);


try:
    opts, args = getopt.getopt(sys.argv[1:], shortopts, longopts)
except getopt.GetoptError, e:
    print(str(e))
    usage()

for opt, arg in opts:
    if opt in ('-h', '--help'):
        usage()
    elif opt in ('-m', '--man'):
        # XXX works on unix, dos, winnt, osx
        os.system("%s -h | $PAGER || more" % sys.argv[0])
        sys.exit(0)
    elif opt in ('-e', '--environment'):
        envvar = arg
    elif opt in ('-s', '--separator'):
        envsep = arg
    elif opt in ('-n', '--no-exec'):
        noexec=True
    else:
        assert False,"unhandled option: %s" % opt

# exec handlers until success, abort, or exit
try:
    browser = os.environ[envvar]
    apps = browser.split(envsep)
    for handler in apps:
        # just print it, don't execute it
        if noexec:
            print handler
            continue

        ret = os.system(handler)
        if not ret == 0:
            print("Bad handler -- %s" % (str(handler) % " ".join(args)))

except KeyError:
    print("No %s set!!!" % envvar)
    sys.exit(1)
except Exception, e:
    print("Un-handled exception! -- %s" % e.message)

One of the common questions of a language, is how to parse command line options? I’ve tried several interfaces in differing languages, but I know one thing… The getopt(3) routine in C on unix is awesome for short options. Give something more portable and with (easy) support for long options, and I’ll have a big grin on my face >_>. But if I have to wade through 20 pages of documentation for a comparable OOP-Interface, I’ll be snoring before I get to writing usage().

Of languages, libraries, and tools

by

I have been contemplating my standard language/library issue with care, although I’ve yet to finish most of the tests…. Courtesy of work, laziness, and to much work on the website lol.

So far, it seems to me that language wise my best thoughts:

Python, I don’t really ‘like’ python but I can do a tremendous amount of work in it compared to other languages, time wise. And it’s fairly easy to implement and test stuff — worst thing I can say about it is it’s interpreted and the regular expression support is done via a standard library module. Although jython (compile to byte code, but compares to a smaller version of an old CPython release) and ironpython (python for .net clr) should help the former, and the api for regex ain’t that bad, but sucks compared to Perl lol (like most languages seem to, that don’t steal Perls syntax or likewise make it a core element of the language syntax). Python also can handle interfacing with other languages well enough between the CPython, Jython, and IronPyhon implementations.

Java, great syntax and fairly logical. The byte code is portable enough between Sun JVMs and it even has the portable Swing GUI toolkit, although I’d probably end up using SWT or the WxWidgets bindings. The bad things about Java, although you can compile to native machine code, ya need to get the GNU Compiler for Java (GCJ) working first. Regular expressions are done via it’s libraries rather then syntax, and. Well the only _nice_ way of saying it, most of the tools with the JDK eventually piss me off, but I like the language aside from that. The only major con is it is as OOP-centric as I’ve ever seen, and it’s that way by design.

C++, lower level then any of the others and offering the best level of interface to C code of any of them around ;-). I personally prefer Java’s syntax over that of C++ but it still gets the job done. It’s just a question of how often you want to press Shift while typing… One of the best things, the STL is quite nice and well enough supported now to be worth using. Although, in my experience when working around C interfaces, the STLs value can go out the window, but maybe that’s just how long I went before using the STL. One plus, unlike Java or Python, you get a more traditional pointer, not to mention a few interesting things in the std::namespace and boost libraries. Not sure what shape exceptions support is like under various system compilers, but I don’t think that I’ve ever seen C++ code using exceptions outside of examples or perhaps the FAQ :.

Useful points of interest:

WxWidgets for the GUI — portable to target platforms of most interest and in all three of the above languages (and a few more too).

libxml2 for XML — portable to target platforms, but seems to only have C++ and Pyhon bindings easily available, libxmlj (for java) I think would need compiling and I haven’t tried that under FreeBSD or Windows yet.

Semi-Native Regular Expressions — Boost libraries for C++, Pythons own regular expression module, and ditto for Java (not that I like them).

std::string, str, and String classes — basic string handling in C++, Python, and Java. Boost and WxWidges also provides wxString.

MySQL bindings — database operations; quickly available for C++, Python, and Java (JDBC based). Using SQLite in C++/Python is also easy enough but I’m a bit leerly of Java-related connections for SQLite3.

As to the other stuff that was on my list:

C++ and Python provide suitable IPC systems, never have looked at Java in that light. Most of the stuff I need, is basically provided by the C libraries under POSIX systems and Windos, Python has a suitable interface to it, so I’m happy hehe.

Python has good enough built in support for common compression and archive formats. Whatever can’t be done via java.util.zip hopefully could be done via JZlib, I’m not familiar with anyhing ZLib under C++, but there is always the traditional interface ^_^.

I haven’t considered the network side that much. Although I sometimes find it a bit lengthly, I don’t really mind working with the C-style sockets library I’ve used here, using the windows sockets library shouldn’t be to far off from the ones used on *BSD and GNU for my needs (aside from headers/linking). Python has support for a number of protocols (and pwns Ruby on documenting the classes for most protocols IMHO), Java well enough out of the box, and both Python and Java provide usable interface to TCP/UDP. WxWidgets also has a few things as well but more limited, that might be worth looking at, since I’d rather like to use a similar interface in each language, while still leaving code that can be read.

As to the ease of use and deployment, that is always a tough question. Under FreeBSD and Debian at least, they are all ‘easy enough’. Windows of course, always has to be the pain in the ass. But for total portability this is an AoR where Python excels. When it comes to deploying stuff, it gets tricker. Windows lacks a package manager and you eventually have to draw a line between what you’re stuck bundling and what your stuck making a dependency. One of these days I need to look at py2exe and see how it would work out for what I’ve a mind to setup.

Big pluses:

Qt4 for C++ and (generally) Python is great and works on all my systems, QtJambi I’ve never tried on FreeBSD yet, but would be interesting if it works. The only thing I hate about Qt, the commercial license costs an arm and a leg :-(, but most of my needs fit within the (free) Open Source Editions license terms with a smile.

WxWidgets seems to be less painful then GTK+ under Windows while still supporting many languages, although GTK+ would probably be more fun to learn in my case. It also has nifty things, including string/regex classes.

MySQL and SQLite are generally portable and have bindings to other languages quick & easy enough. I’ve never used SQLite on a project, since I run MySQL on one of my own systems to service all my SQL-related needs 😉

When it is all said and done, I really think Pyhon is the best choice… I dunno why or how, but it seems that way! Hmm, to re-dive into using Python for most tasks, or to get C++ crazy… Or live with using both side by side? For my own sake, I think I should sake fuck Java! But it is a great language in it’s own right.

Oh well, maybe in a few months I’ll have figured out which path to take. I can always make use of other languages and tools, but I very much need to find a “primary” set to work with!

think:
    switch(choice) {
        case PYTHON_ONLY:
            // Python becomes my primary language
            break;
        case CPP_ONLY:
            // C++ becomes my primary language
            break;
        case PYTHON_AND_CPP:
            // use both side by side
            break;
        default:
            goto think;
    }