Terry Poulin

back to part II

The getmail documentation said that was the best way to automate it, and its bloody better then buling up getmail with ‘daemon mode’ code. But also like the documentation said if we want to ‘stop’ this daemon like mail checking we need a script for cron to run getmail through and program it to not run getmail if a file is pressent. Now we can do this many ways, heck we could set a enviroment variable if we want.

I’ve written a little script that can be run with a users cron jobs and skip over all mail or only the given account. You need one rcfile per account and you can tweak it to follow any conventions you want. My RCFiles follow the convention of getmailrc-shortaccountname, hence getmailrc-bell and getmailrc-sas for my bellsouth and sasclan accounts. This script should work on any system that has a bourneshell compatible /bin/sh. Just edit the shebang if you need to run it as another shell (such as /bin/ksh).

#!/bin/sh

# Use getmail to check my E-Mail account using the RC variable
# This script has the advantage that one can save it as another file, change
# one variable and set a cron job to check each account at different times (1
# cron job per script). Then not only use a file in their .getmail folder to
# stop all the scripts from running getmail or use a file to stop one script
# from checking its account. It also keeps a log which will be trimmed by
# another script

# Name of the getmailrc file to use
RC=getmailrc-sas

# log to a file when mail is checked
LOGFILE=${HOME}/.getmail/cronjobs.log

#
# If a nomail or nocheck file is found in ~/.getmail/ exit without checking
# else look for a no file. Where  is equal to every thing that
# comes after the getmailrc- in $RC. If none of these files exsist check mail
# using the $RC file.
#
if [ -e ${HOME}/.getmail/nomail ]
then
        LOG=$(echo "Skipping mail for $RC")
        exit 1
elif [ -e ${HOME}/.getmail/nocheck ]
then
        LOG=$(echo "Skipping mail for $RC")
        exit 1
else
        DIE=$(ls ${HOME}/.getmail| grep $RC | cut -d '-' -f 2)
        if [ -e ${HOME}/.getmail/no${DIE} ]
        then
                LOG=$(echo "You have desided not to check this mailbox - $DIE")
        else
                LOG=$(echo `date` "checked mailbox with $RC")
                getmail -r$RC
        fi
fi

# Update log with the result
echo $LOG >> $LOGFILE 2> /dev/null

if you want to use the script copy and paste it into a text file and mark it executible. I saved them as ~/.getmail/check-.sh and chmod’d them 0700

Ok, let us make a cron job, because this is allready a big long post that took me forever to write with the way my house is. I’m not detailing cron(8) so please read the handbook or read the fine manual.

I created this crontab file to run my scripts to check my accounts every 5 and 4 hours respecfully and to ‘trim’ my log file every week.

# rstf's crontab
SHELL=/bin/sh
PATH=/etc:/bin:/sbin:/usr/bin:/usr/sbin
HOME=/usr/home/rstf/
MAILTO=""
#
#minute hour    mday    month   wday    who     command
#
# check sas every 5hr
5     *       *       *       *       rstf    ${HOME}/.getmail/check-sas.sh
#
# check bell every 4hr
4     *       *       *       *       rstf    ${HOME}/.getmail/check-bell.sh
#
# trim log @weekly
0     0       *       *       0       rstf    ${HOME}/sh/trim-getmail-cronlog
#

The trim-getmail-cronlog script is thus

#!/bin/sh

# Rotate my logfile for running get mail via cron

LOGFILE=${HOME}/.getmail/cronjobs.log
TMPDIR=/tmp/

if [ -e $LOGFILE ]
then
        tail $LOGFILE > $TMPDIR/gmlog.$$
        rm $LOGFILE
        mv $TMPDIR/gmlog.$$ $LOGFILE
else
        exit 1
fi

To load my crontab file into crons system i.e. /var/run/tabs/${USER} all I have to do is run a simple command.

crontab ~/rstf-contrab

Back to Part I

It should’ve taken like two minutes to install or so. The getmail program is very nice and it follows the concept of do one ting and do it well + allows for a great level of modularity. We can use it with a number of protocols and other softwares like procmail, spamasassion, clamav e.t.c.

Setting up getmail is very easy and theres great documentation so far I’ve been very happy with it. Lets go to our home directory, we do *not* want to be the root user for this.

I’m gong to use a console because thats how i like it, so I’ll short how to do it via that way. If you want to use a GUI app like Konqueror to do it be my guest, you should be able to easy enough.

Make a directory your home directory called .getmail and set the permissions so that only you have access.

mkdir -m 0700 ~/.getmail That ammounts to the owner having read, write, and execute but no one else but the root user being able to enter the directory. Lets cd over to our ~/.getmail folder and create a rc file. By default getmail reads the ${HOME}/.getmail/getmailrc file but we can create multiple rc files and have getmail use the one we choose.

getmail -rRCFILENAME

if its not in our .getmail/ folder we need to supply the path to the rc file, if its in .getmail we can skip it.

The syntax of the file reminds me alot of .ini files on Win32, to be perfectly honest the getmailrc file syntax is the easist I've seen. Heres a copy of one of the RC files I use complete with anotations of what the ooptions do. After this I'll go into more detail about the options to help you get a owrking rc file.

[retriever]
# This file is getmailrc-sas which is for checking my @sasclan.org account
type = SimplePOP3SSLRetriever
server = mail.host.tld
username = my_emailaddr@sasclan.org
password = My_Password

[destination]

# This destination is just for my e-mail not the systems local mboxes.
type = Maildir 
path = ~/Mail/
user = rstf
filemode = 0600

[options]
# Note that '0' means disable for all integar values.

# (int) 0-warn/err only, 1-retriv/delete only, 2-every thing
verbose = 1
# (bool) true = fetch all, false = only what getmail has not seen before
read_all = true
# (bool) true = delete messages after downloading, will override delete_after
delete = true
# (int) delete messages from server after int days
#delete_after = 1
# (int) max server reported message *total* size to download per session
max_bytes_per_session = 0
# (int) do not retreve messsages larger then this number of bytes
max_message_size = 0
# (int) max number of messages to process
max_messages_per_session = 0
# (bool) adds Delivered-To: header field to the message.
delivered_to = true
# (bool) add received: header field to the message
received = true
# (str) log to file
#message_log
# (bool) use system logger
message_log_syslog = false

as a reminder so I wouldn't have to check the documentation in /usr/local/share/doc/getmail/ or online. I put comment notes in the file briefing discribing what each option does and the type of setting, namely bool (i.e. true/false), int(eger) i.e. 0 1 or 435 e.t.c., or str(ing) likethis. Basically you need to have a [retriever] and a [destination] section. Under retriever we tell getmail what type of protocol to use, taken from the documentation heres the options.

• SimplePOP3Retriever
  — for single-user POP3 mail accounts.
• BrokenUIDLPOP3Retriever
  — for broken POP3 servers that do not support the
  UIDL
  command, or which do not uniquely identify messages; this provides basic
  support for single-user POP3 mail accounts on such servers.
• SimpleIMAPRetriever
  — for single-user IMAP mail accounts.
• SimplePOP3SSLRetriever
  — same as SimplePOP3Retriever, but uses SSL encryption.
• BrokenUIDLPOP3SSLRetriever
  — same as BrokenUIDLPOP3Retriever, but uses SSL encryption.
• SimpleIMAPSSLRetriever
  — same as SimpleIMAPRetriever, but uses SSL encryption.
• MultidropPOP3Retriever
  — for domain mailbox (multidrop) POP3 mail accounts.
• MultidropPOP3SSLRetriever
  — same as MultidropPOP3Retriever, but uses SSL encryption.
• MultidropSDPSRetriever
  — for domain mailbox
  SDPS mail accounts,
  as provided by the UK ISP Demon.
• MultidropIMAPRetriever
  — for domain mailbox (multidrop) IMAP mail accounts.
• MultidropIMAPSSLRetriever
  — same as MultidropIMAPRetriever, but uses SSL encryption.

Odds are if you don't know what you need, its probably SimplePOP3Retriever. If you've ever set up a mail client before you should know it, your ISP or webhost should be able to tell you as well. Next we gotta tell getmail what server to fetch mail off of with the server option. If your ISP is some thing like charter, its probably mail.charter.net. I don't have charter but all the mail servers I've seen have been mail.ispname.topleveldomain lol.

We need to set the username and password so the server knows its us and which mailbox we want. Other wise it will tell us to go 'eff off.

Now we need to tell getmail what to do with our mail once it checks the incoming mail server. This is what the destination section is for. You basically have two big options here, Maildir or MBox. I've always used mboxrd since thats what Mozilla Mail&Newsgroups, Mozilla Thunderbird, and Seamonkey Mail&Newsgroups used. Plus the systems local mailboxes are mbox format as well. Theres various 'variations' of mbox and probably maildir but compatible enough for our needs I'd say. Other options for 'type' allow us to use an

External Message Delivery Agent (MDA) like procmail.
Mutilple Destinations, using multiple maildir/mbox/externMDAs e.t.c.
Mutiple message sorting
Sort mitple messages into geussed destinations
And to use qmail-local to deliver messages as instructed in the .qmail file.

The exact specifics and how to set getmail to use these features are in the manual, go read it if you want to know more. I suggest ether mbox or maildir personally.

Maildir is pretty simple there is a folder containing new, cur, and tmp directories full of e-mails. If you plan on checking e-mail often or automating it (as I do) this is probably for you. Each e-mail gets its own file in one of those directories which I personally think makes it better suited sharing messages but bad for FAT32 file systems (i.e. many small files).

We can make a mail directory like this on the command line, assuming we want ~/Mail. Or just make four directories in a GUI file manager.

mkdir -p ~/Mail/{new,cur,tmp}

You'll need to set the path to the mail directory as well to use Maildir. You can also set the user and file permissions to use. You've got to use the unix octal format, i.e. 0755 instead of u=rwx g=rx o=rw or some thing.

[destination]
type = Maildir 
path = ~/Mail/
user = rstf
filemode = 0600

Setting filemode to 0600 means only I and the root user have read-write permission to my mail.

Now if we want to use mboxrd we have to specify the type and path to the mbox file as well. The user option works here too. Also you need to consider the locktype option. It takes a string argument and you have a choice of 'lockf' which uses fcntl locking or 'flock' default as of this writing (getmail v4.7.0) is lockf.

[destination]
type = Mboxrd
path = ~/Mail/inbox
user = rstf
locktype = lockf

We can also set up filter sections to use stuff like clamav & spam assasion on our e-mail. Considering that the odds of a virus or trojan that can invect a windows machine through being ssh'd into a freebsd box that is using mutt to view mail fetched with getmail from a server that filters spam (optional /w my ISP) and scans for viruses (nice ISP). I don't blood ythink I need to filiter things through an Anti-Virus ! But if you like go read the documentation on how to set that up.

I think I'll be looking into spam assasion for my ISP account though so maybe I'll have some thing topost there.

The options section I don't think is required but I'd suggest you set your read_all and ether delete or delete_after options.

My suggestions

[options]
# fetch all mail on the server
read_all = true
# then delete it after its in our Maidir or Mboxrd destination(s)
delete = true

[options]
# fetch mail getmail has not seen before
read_all = false
# then delete old messages after 1 day
delete_after = 1

The bottom option deletes the messages you download today from your mail server (not your destination) the next time getmail checks for mail and sees that the old messages are '1' day old. Any integer number will do but not a floating point number. i.e. 4675 will work but 2.43 will not.

I have two getmail rc files one for each account

$ ls -R ~/.getmail                                             20:21
getmailrc-bell
getmailrc-sas
oldmail-mail.host.tld-110-username
oldmail-mail.host.tld-995-username

So I can run getmail and tell it which file to use so Ican deside which mailbox to check. I've made a pair of shell aliases in my shells rc file to save typing.

alias gm-bell='getmail -rgetmailrc-bell'
alias gm-sas='getmail -rgetmailrc-sas'

Ok, lets run getmail (I'll skip the alias), it will take a few seconds bu t if it takes a really long time you might want to make sure your system is configured correctly to resolve the hostnames.

rstf@Vectra$ getmail -rgetmailrc-sas                                                       20:22
getmail version 4.7.0
Copyright (C) 1998-2006 Charles Cazabon.  Licensed under the GNU GPL version 2.
SimplePOP3SSLRetriever:My_EmailAddr@sasclan.org@OurMailServer:
  0 messages retrieved, 0 skipped

Looks like I have no new mail in the account.

Ok, lets try some automation we can set cron jobs to run getmail -rRCFILE when ever we want on one or all of our files.

This is a short how to for kicking sendmail in the buttocks and setting up a micro-replacement for send-only usage on a FreeBSD (post 5.x) system. Plus using the getmail utility to check for new messages. I spent long enough screwing with it after I miss-read some documentation (fbsd handbook/ssmtp).

The goal is to be able to use utilties such as mailx, mutt, and other MUA’s dependant on sendmail (or exteneral) MTA(s) and be able to check mail easy and automaticly. I assume you at least know how to setup a mail client such as Mozilla Thunderbird or Outlook Express (or can get the relivent info) and know how to edit files as root when needed.

First we have to kill sendmail. For this we need to edit the /etc/rc.conf file, since I’ve cleaned mine up to place various options in related ‘sections’ heres the relivent one from my rf.cofing. Note all the sendmail_* options at the bottom.

#########################################################################
#                               SERVICES                                #
#########################################################################


#cupsd_enable="YES"
#background_cupsd="YES"

samba_enable="YES"

ntpdate_enable="NO"
ntpd_enable="YES"
ntpd_falgs="-c /etc/ntp.conf -f /var/db/ntpd.drift"

# Completly kill sendmail
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

Save rf.config with those sendmail lines in it and we can stop sendmail from working. Sendmails probably the worlds most used Mail Transfer Agent but for a desktop we don’t really need it. While its well known for its history of security problems according to some OpenBSD people, at least they still patch it. Now with sendmail gone we have two problems.

1. Daily run messages are usually mailed to root and this might break that a tad.
2. MUA’s dependant on external MTA or utils dependant on sendmail may require configuration changes or stop working

Now to fix this we want some thing small that will allow us to send e-mail through some thing sendmail compatible’ish. I’ll assume that you have ports installed or know how – so go do it and update them. Personally I try to update my ports tree every few days or just before building a port if I rarly install stuff on the machine. I also prefer cvsup/csup 🙂

Ok, now that you have ports ready to go lets install some software. Open a shell (or use a GUI) to cd into /usr/ports/mail/sstmp/ so we can build it. Run the commands

make install replace clean

it shouldn’t take long to install so no need to go on coffee break 😛
In case your wondering what ssmtp is heres the pkg-descr:

A secure, effective and simple way of getting mail off a system to your
mail hub. It contains no suid-binaries or other dangerous things – no mail
spool to poke around in, and no daemons running in the background. Mail is
simply forwarded to the configured mailhost. Extremely easy configuration.

WARNING: the above is all it does; it does not receive mail, expand aliases
or manage a queue. That belongs on a mail hub with a system administrator.

WWW: http://packages.debian.org/testing/mail/ssmtp.html

Basically it doesn’t handle fetching or checking mail or reading it but it lets you send it. While some thing like sendmail, qmail, or postfix should at least provide send/recieve if you want to go bugger setting up a full blown mail server be my guest 🙂

sSMTP is really s send-only app that tries to emulate sendmail, most of the command line options to sendmail are accepted but many are just ignored. Some MUA’s can check mail for us, mutt springs to mind but I rather like using getmail so far. At first I thought I would use sendmail/fetchmail/ but I saw an app called ‘getmail’ in ports and checked it out. Its not as buggy as fetchmail is *said* to be, its got great documentation, and its really simple. So if your going to use getmail once ssmtp is done building start on /usr/ports/mail/getmail/ while we open another terminal to configure ssmtp while getmail compiles.

Using a shell or file manager like konqueror cd over to /usr/local/etc/ssmtp/ . We can ether use the .sample files here or create new ones from scratch. We to create a ssmtp.conf file that tells ssmtp how to send e-mail to our out going mail server. The config file syntax is a cake walk, its key=value and # for comments to end of line like many a language or file has.

# Your E-Mail _address_
root=email_username@isp.net
# Your outgoing mail server, normally the TLD is .net, .com, or .org
mailhub=mail.isp.topleveldomain
# Where mail looks like it came from, just stick in your e-mail address.
rewriteDomain=email_username@isp.net
# Your hostname, your e-mail address should be fine if your not a mail server
hostname=email_username@isp.net
# Set this to never rewrite the "From:" line (unless not given) and to
# use that address in the "from line" of the envelope.
FromLineOverride=YES

save as /usr/local/etc/ssmtp/ssmtp.config and that takes care of that. You might want to nuke sendmail or reboot of course for this to take effect. Now since we do want our daily run stuff to still work we will set up an alias so mail sent to our user account locally will actually be sent to our e-mail address. We’ll do this with the revaliases file in the same directory.

This ones a little bit og a bug but its not once you figure out how. Basically the syntax is like this

your_login_name_on_this_computer:your_email_address_to_send_to:the_out_going_mailhub_to_use

# Alias e-mail to the system root account to go to my private e-mail
root:myemailaddr@myisp.tld:mail.myisp.tld
# Alias my_username to my email address so I get my cron-job reports
my_username:myemailaddr@myisp.tld:mail.myisp.tld

This way the daily run and cron e-mail messages will still work and they will be in my e-mail rather then /var/mail/${USER} where I often forget to check 8=)

I’ve yet to figure out how to set up ssmtp for multiple outgoing mail servers but I’ll figure it out later & post it.

Ok, now lets get back to getmail