Projects/tpsh

Now if I didn’t have to be awake in ~4 and a half hours, I could get the . / source commands implemented, so handling tpsh_profile / .tpshrc files on startup could be done… and have more of the manual page written out :. Hmm, that reminds me another big thing I need to work on is the shell special variables ($0…$9…, $*, $@, $#, $?) and the rest of ${parameter expansion} syntax.

If I could get anything done during the day here, and actually sleep at night… instead of coding odd hours, now that would just be heavenly lol.

$ git log | sed 's/my name/<snip>/g' | sed 's//<sed@removed.it>/g' | vim -

commit 4835c4091e59af282ee98952568f0e9ac91c8d09
Author: Terry <snip> <sed@removed.it>
Date: Sat Mar 21 09:27:41 2009 +0000

do_getop() made generic, `set [options]` now works.

The do_getopt() function now takes an array ref and hash to pass onto
Getopt::Long::GetOptionsFromArray, and twists it in place.

set_bin() now calls do_getopt() with it’s argument vector and a global
hash; startup code now calls do_getopt() with @ARGV and that same hash.
Because of that, `set -x`, `set -o xtrace`, and such work but `set +o
xtrace` and such can’t be used to turn the option off yet lol. (+o atm
is just an alias for -o.)

commit 90bf68e8b5f0aed867ae4a9fd2eca2c8d99dc1fd
Author: Terry <snip> <sed@removed.it>
Date: Sat Mar 21 08:16:39 2009 +0000

tpsh -o longname now works

further more as an extension, ‘-o logname1,longname2’ also works but is
not documented yet.

commit dcffa41bb57d70c82f502b08ea8c76ebeb559b1c
Author: Terry <snip> <sed@removed.it>
Date: Sat Mar 21 07:49:28 2009 +0000

which built-in command added and documents

see Built-in Commands and CEVEATS & BUGS in tpsh.1.pod for details.

It is really time to make do_getopt() a generic wrapper around
Getopt::Long, the current functionality in do_getopt() needs to be moved
into set_bin() anyway…

commit 6f94f751ac76177a23138a1ad014b4af1af50de7
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 18:29:43 2009 +0000

%Aliases renamed %Macros

because it stores all named macros, whether defined with function, macro, or alias

commit 2a83cbc45daf78cb41cba40c28735252e521f965
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 18:24:55 2009 +0000

is_macro() and is_builtin() subroutines added

commit 1853b61eff14d35ddf1f89282ed26a8f3957d373
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 08:43:50 2009 +0000

type built-in documented

documented the type built-in command, and tweaked type_bin; it now
prints macro instead of alias, and will stop at the first match found.

commit 39a9824ca29d7beea6c5a054f970b86968861568
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 08:30:12 2009 +0000

pwd built-in added and documented

commit 488aa51ed2c430848aa80fb26ad093ccec604237
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 08:08:04 2009 +0000

function and macro built-in commands documented

commit 273b7f8bbbf0b3db8759053969e854f1de963003
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 08:02:41 2009 +0000

pipes and I/O redirection can be used inside a named macro definition created with the macro built-in

commit 7aa70dd6873e147f3bfdec7460312dc54482696b
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 07:53:06 2009 +0000

macro_bin now reports `macro name()` as an error, correctly

commit e52aed3dec5455c6488dc4465aa75d20aca4e696
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 07:22:39 2009 +0000

hash built-in documented

commit 843a0de91b3c2df2b1fed8b9416426bc0e7c01af
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 07:21:22 2009 +0000

hash built-in now has an -r option that nukes the %Path hash table

commit 27526bb013cc1ab7b425ee1f3b685585746a0160
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 07:17:36 2009 +0000

hash_bin now displays an error if the item to be deleted does not exist

commit 45c71f7c54faa81106de5e68290884150823e10c
Author: Terry <snip> <sed@removed.it>
Date: Fri Mar 20 07:12:50 2009 +0000

hash_bin can now remove elements from %Path

now if only I had time to do things tonight… really what I need to get done, is move much of what’s left in main_loop() further into a subroutine, so I can implement source_bin() for the . / source built-ins and set it up to handle .tpsh_profile/.tpshrc files on startup through init_sh.

a few other serious things that need doing: hook our completion function into Term::ReadLine::Perl (I’m using Term::ReadLine::Zoid on my workstation; which is very easy to hook up to an app specific completion function); finish coding the completion hooks; implement `tpsh -c “cmds”`, in fact `(cmds)` could probably be expanded to that for sub-shell grouping, lol; write eval_bin(), the code needed for an eval built-in command is there, just needs the sub to bind it together; make `backquotes/backticks` needs to be handled by a pipe opened sub shell, rather then running the command directly with a pipe; and also support for an arbitrary number of pipes in commands, the place holder code in exec_pipe() only allows ‘lhs | rhs’; adding ${} expansion to variable expansion, $TMPDIR works but ${TMPDIR:-/tmp} and things don’t yet, but that’s trivial to code.

Really the only thing that worries me, is how to implement job control :. Setting up the built-in test command, [ which is gonna be interesting of course but actually doable, like control flow: fun but simple. The job control though, I’ve no freaking idea how to implement in pure Perl under unix/win32, yet…

# last stream of commits; html fixed

$ git log d59b9f864e916cd914be55b223702ec618c1ec4f..HEAD | sed 's/.../<snip>/g' | sed 's/.../sed@removed.it/g' | vim - 

commit 9450eca02b4dffcce6802b8fd72ba4e307ff7b9b
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 10:12:25 2009 +0000

exec and exit commands documented

Also CAVEATS & BUGS section updated about argumentless cd/chdir behaviour.

commit c24135e3748b9dbe8520b5794311af92e3affb36
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 10:06:20 2009 +0000

tpsh now exits with $? >> 8 if no exit status was given to the exit built-in

commit a9138f5991142960114f6898e7d57185eff00256
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 09:59:59 2009 +0000

cd and chdir commands documented

commit 3145e1a5672330fc913e93b3992b7c6591ff58a8
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 09:53:58 2009 +0000

CDPATH handling skipped if path starts with / or .

commit e6f8690d05308789a31fb906a28b1dbeba304ec4
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 09:20:22 2009 +0000

builtin command documented

commit b789721176eeb7d157d7c42e38044b1c111fa3dc
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 09:16:44 2009 +0000

bugfix: alias expansion turned off for builtin command

noticed that after `alias echo foo`, that `builtin echo …` would
expand to ‘builtin foo …’

commit a6a7eeddb5309af903569aee31bb5f15ceef9d6e
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 08:46:20 2009 +0000

alias name now displays ‘name’ => ‘definition’

Also alias expansion is now disabled on the alias command as well as
unalias. This prevents expanding the lhs (name) of an alias when name is
given, but definition is not (e.g. `alias name`)

commit 99a57b1b88d58f4ad6b318c181668cf5f5e9eeda
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 08:21:19 2009 +0000

bugfix: rhs of alias definition is no longer sh_eval()’d by alias_bin

alias_bin has ran sh_eval() before storing the macro since commit
2da5a0e38e3d14c29b0fb93ad89797826a41ddac, which was causing expansion of
things like `alias lah=’ls -a $HOME’ to be stored as ‘lah’ => ‘ls -a
/home/Terry’ instead of ‘ls -a $HOME’; as the quotes were pre-expanded
on the command line, alias would receive and expand the environment
variable itself.

commit a9177482c3398a6272382672e45ea9d5e8254960
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 08:11:23 2009 +0000

alias built-in documented

commit 8296bb37be75e1e7c301c1ed316db13aa6c9c26d
Author: Terry <snip> <sed@removed.it>
Date: Thu Mar 19 07:34:16 2009 +0000

bugfix: SIGPIPE now handled by exec_pipe()

Previously a construct like `ls | head 1` would cause tpsh to terminate,
due to receiving an unhandled SIGPIPE on unix. tpsh now traps the signal
and will report an error if desired.

Display of the error message can be turned with the ‘reportpipe’ option,
which defaults to off.

commit b44b8f3b4cf6adedf972bf719f4de1546fc8a8b1
Author: Terry <snip> <sed@removed.it>
Date: Tue Mar 17 23:44:38 2009 +0000

noclobber mode is now implemented

commit 842d3a308e2663c15dc75600c19eb7f096664341
Author: Terry <snip> <sed@removed.it>
Date: Tue Mar 17 23:21:58 2009 +0000

+flag is allowed, i.e. +a

commit 289837932bf4af21c41c538e32e2d886d6f9a03b
Author: Terry <snip> <sed@removed.it>
Date: Tue Mar 17 09:26:30 2009 +0000

added set built in command and command line arguments to the manual

commit e1badaaf1e873ae0808eecb79b1b3a07219752f5
Author: Terry <snip> <sed@removed.it>
Date: Tue Mar 17 08:18:07 2009 +0000

the verbose option now works

xtrace now also prints to stderr, IAW with ash

I really need to give tpsh the ability to use $HISTFILE to save/restore the ReadLine history between sessions…

 commit 6eda7f33de6bb7505fc64986a5bd31211d68acc5
 Author: Terry ... ... <...@,,,>
 Date:   Fri Mar 13 05:36:45 2009 +0000
 
     initial implementation of the function and macro built-ins
     
     'function name() definition' creates an alias named 'name' with evaluation
     'definition'
     
     'macro name( definition )' creates an alias named 'name' with evaluation
     'definition'
     
     Eventually macro will probably define the alias wrapped in a (grouping),
     once (command groupings) are implemented. function will allow {
     groupings } when that is also implemented; but will not make the
     implicit.
     
     Currently variables and quotes are pre-expanded by the line processing
     code, before function/macro receive the text to store in a macro. While
     this is the desired behavour for the alias built in, it is not desired
     for other methods of defining a command macro! It is however possible to
     escape variables and things, e.g.$ macro e( echo hi '$USER' ); so that
     the expansion occurs when 'e' is executed, as opposed to when 'e' is
     recorded. This will hopefully be fixed in the future.
     
     The manual page is in continual development, and will reflect the
     desired "end state" for macro behavour.

Basically tpsh has no real concept of an alias or function per say; only a simple macro recording facility.

  # bourne style alias
  alias x=y
  # csh style alias
  alias x y

  # bourne again style named function
  function x() y

  # tpsh specific syntax
  macro x( y )

  # each make x expand to y (with appropriate context grouping to be added later)

Generally where things differ enough between sh and csh, to warrant a check of the fine manual, like alias and the export/setenv thing; tpsh endeavors to be more permissive about muscle memory.

Internally there is no real difference between an alias and a function, it’s just a macro with a name. sh doesn’t use the function keyword, but bash does (which zsh also permits, hehe). It’s much less bugger to implement functions with the keyword then not at the moment, and it’s really the macro keyword I’m interested in. The macro built in is meant for simple commands, where as the bash function syntax is meant for compatibility and to make using a complex command as the definition readable. The sub shell and same shell command grouping syntax used in sh, of ( commands ) and { commands } I am expecting will be treated like an anonymous macro.

Eventually I’ll probably alter sh_eval() to “relax” its evaluations of a function definition so that environment variables and quotes (etc) are expanded when used, instead of stored. Atm the only “don’t screw with this” case in sh_eval, is not performing alias expansion when one executes the unalias built-in, since you can’t specify the macro to delete if it gets expanded first ^_^.

Personally, I like ‘macro foo( bar )’ more then ‘alias foo=bar’ or ‘alias foo bar’, feels more natural. tpsh after all is meant to reflect MY preferences rather then being purely sh compatible, lol. Even though there is little difference in tpsh (yet) between ‘alias la=ls -a’ and “alias la=”ls -a”‘, the implicit grouping inside the () is just more to taste:

$ macro la( ls -a )
$ macro ll( la -l )
$ alias
'la' => 'ls -a'
'll' => 'ls -a -l'
$ 

most Bourne inspired shells that support alias display them as `la=’ls -a'”, but the alias command is fairly new in sh, and I prefer ‘la’ => ‘ls -a’, thus that is what tpsh uses.

Hmm, what pipe was I smoking when I thought it would be possible to get a measure of consistency between 3 different readline implementations?

Hmm, my recent pet project (tpsh) is actually shaping up quite nicely. It is almost to what I would consider a usable interactive state, the only big irk left I guess is the pipe handling and a few odds and ends. Development started like last month (first commit in my master branch is dated 2009-02-28), and has been a very on/off thing in my spare time; a lot remains to be done, but progress is quite nice.

I rather like the korn shells, but I find it annoying to see what implementation of korn from who behaves in what way. OpenBSDs customized pdksh being the best of the lot in my opinion, the only problem with the korn *family* being consistency in the wild… cmd.exe just barely counts as a command shell, better for scripting then for interactive use. I can’t help but think, cmd.exe’s filename completion was either written by a 5th grader or a hold over from command.com (I haven’t used command.com in years). Currently I use zsh most of the day on FreeBSD, ksh on OpenBSD, and cmd.exe on Windows XP. So it is really important to me to have a single and SANE shell.

I used to use tcsh a lot, but after getting into Bourne-style shell scripting (Nota bene: I know nothing about csh scripting) I got pissed and sought out a shell that would allow me to test sh snippets at the prompt. To be honest, I do not have any issues with the GNU Bourne Again SHell (bash); some distros default init files aside. Yet just like tcsh/zsh — I have no big desire to cart bash around with me everywhere (and I do not like cygwin).

The solution? Create a shell that behaves more or less consistently across platforms, does what I want, and is not a pain in the ass to port along for the ride.

Line editing is completely delegated to Term::ReadLine. Someday writing something like readline or curses on an alien OS for terminal XYZ would be one thing, having to implement shell line editing and be portable between the UNIX and Windows NT environments: I wouldn’t do it if you paid me. (and even if drunk, I would still likely use normal libraries!)

On the interactive level, line editing for me is an absolute must. Command completion I can live without for awhile. Unix names are usually short and I like fairly terse and structured ones myself, winsucks is a bit of a different story. The child like way cmd.exe does command completion, it’s almost better to do without completion on windows haha!

Right now basic expansions work, ${VAR} and $VAR, line editing and history is via Term::ReadLine (I’ve been using Term::ReadLine:: Zoid and Perl implementations). There’s only rudimentary handling of quoting for the time being, something that needs labouring with in order to handle nested quotes (e.g. echo “`ls ‘C:/Program Files’`”) but currently does the job well enough for right now. The ‘single’ and “double” quotes mostly work as expected, The `backticks` work as expected in simple cases, but since there is no proper support for shell script or -c “command” yet, things like `find /some/where -name foo -print` will work as expected, but `find /some/where -name foo -print | xargs whatever` will not work. Once things are more evolved on the scripting side, it should be trivial to implement backticks correctly (nested quotes aside). The idea being to pass the backticks onto a child shell. Right now the only big catch 22 with quotes is only one set can be used and they do not nest yet; fixing that will be fun.

I/O Redirection and pipes also work for starters, but not completely. The >, >>, and < operators do work, but don't accept an optional file descriptor number yet (e.g. foobitz 2>/dev/null) or other tricks of the trade. Basically just the I/O redirection I use a lot is implemented for right now; since the focus is on interactive use by yours truly, it will probably stay that way for a while. Command1 | command2 also works well enough right now, but I’ve yet to implement pipe lines properly (e.g. cat f1 f2 f3 | sort | uniq | less; doesn’t work yet); just wasn’t enough time to shift from prototype to finish without sleep lol. Only one use of pipes currently is supported, until I have the time for coding unlimited segments in a pipe line.

Aliases even work ok at the moment, but no (damn!) addictive runtime parameter expansions have been added yet (e.g. alias qux=”xuq -f $1 -x $2″). That will have to be done someday, hehe. In point of fact, $*, $@, and the things related to positional parameters are not even implemented yet.

One benefit of my development environment, is I can use I/O redirection to run a simple test suite. I created a ‘test-sh’ script that strips the environment of all but the interesting stuffs before invoking tpsh. Then I have a test file with commands written out.

 $ ./test-sh < file

and I can watch the results of the test file fly across my screen, as if typed interactively (I freaking love unix shells). It is not quite a unit test, but it is enough to be able to run the test file and monitor the results for unexpected regressions or fuck ups, before pushing the code to vectra. Currently shell globbing is limited to Perls glob(), probably to be implemented directly with File::Glob::bsd_glob() in the future, and someway of letting an environment variable specify the desired GLOB_* flags. (perl 5.8s glob() seems to do what I want). I have yet to decide how to handle tab completion across the Gnu, Perl, and Zoid ReadLine modules but coding the completion functionality itself should be fairly easy; the issue is just how to plug it in properly. Exactly what the globbing behaviour will be like, I dunno yet: because to be honest I have never noticed any big difference between (t)csh, perl, and zsh when it comes to using globs (but I also admit that I rarely do more then abuse * and {x,y,z} on a daily basis). If at all possible, I would like to use File::Glob rather then roll my own (and modern perl glob() does use File::Glob in the background afaik). I believe that as soon as a full pipe line implementation is done, I’ll be ready to start coding it from a tpsh session lol. Goal wise, where am I trying to go? It is implemented in Perl, because perl is fun, perl is less pain to port then C/C++, and I’m not going to use Java or C# period lol (my zsh setup starts up slow enough as is). Python, Ruby, or PHP would also have been good choices, except I’m trying to avoid rubber banding between languages. (Also note: I _hate_ writing C code on Win32.) It should generally behave like a traditional Bourne shell most of the time, but not be a true sh knock off. Whenever in doubt, mimic ash / sh behaviour. It’s not meant to be POSIX compliant, anymore then it is meant to be a real sh: based on, not clone of. It is supposed to behave as close to identical under unix and winnt as possible, with “pains of installing” meant to be a working ReadLine module. A good example is perl -e ‘code’ should work on Windows without having to adapt cmd.exe quoting rules. (really I think cmd.exe has shitty documentation for such an essential program). One reason I like languages like Perl and Python, they mostly behave the same on whatever OS, with a minimal of pain involved (especially Python) It should be easy to use, providing all the expected features — without massive bloat. It behaves like _I_ expect, not what someone else expects (good example: unix style shell, not dos style shell) It is not an interface to Perl. There’s at least 2 different perl shells out there that I know of, but I do not want a shell script that behaves like perl or even does configuration in perl; I want a shell written in perl that behaves like a regular shell lol. It’s also an easy project to take my mind else other matters in life right now… tpsh’s builtin eval command will work more or less like sh, not CORE::eval !!! Maybe I will implement a peval or pleval builtin that works like shells eval, but instead executes it as perl code rather then shell commands. (basically type `peval perl` code instead of `perl -e ‘perl code’`). Accessing perl code through tpsh just is not a priority, when I want to talk to a perl interpretor, I’ll invoke one. Eventually I want it to be able to execute shell script in some form, the goal of which (obviously) will be handling my shells initialization files (~/.${USER}_shrc and ~/.site_shrc) without going belly up. To pull that off, a Bourne-style shell, a which program or built in, and support for functions / aliases is required. (My init files handle sh, bash, zsh, and several flavours of korn; plus several OSes from one main file + a site local settings file). My init file is rather elaborate in design compared to most others I’ve seen, yet very conservative in what it demands of the shell: the most advanced sh features required being function support lol. Basic idea of feature/priorities for tpsh:

 a usable shell for daily use (basically done)
 tab completion (filenames, then command; then pluggable, then 'smart' completion)
 code the remaining builtins and allow a coloured prompt ;-) (etc)
 expand the expression syntax into a more usable (read normal) form
 do shell functions
 handle job control (basic (i.e. bg, fg, &), then rest)
 allow shell scripting in a conventional way, rather then tpsh < cmdfile
 grow the control flow and things for shell script (if, while, for, ...)
 take POSIX into closer consideration for the minor details

the only part that worries me, is implementing the [ builtin lol. (mm, maybe feed [ stuff ] into a syntax tree that we can build a corresponding perl statement to eval())