For a while I’ve been using the GNOME Evolution address book with Mutt, my favorite email client. I use a script called mutt-eds-query, which consults the default evolution-data-server (EDS) address book to find contacts whose name or email address match a substring. I have a couple thousand contacts still stored in my EDS address book, so it’s really useful for me to be able to consult them. When I type an address in Mutt’s To: field, for example, I give a few letters and then hit Ctrl+T to see a list of completions.
Unfortunately, the script stopped working when I moved to Fedora 18, because the new release contains the newest Evolution and EDS (3.6 versions). The new EDS cleans up a number of deprecated functions and has a more regular interface for querying data sources. Obviously that means the mutt-eds-query needed changing too. I used the extensive GNOME developer documentation to find out what had changed, and updated the script I found on the Mutt wiki.
So I hacked in some changes today. I’m not a born C programmer, so I’m sure there are bad style uses and other stupidities in what I did. Nevertheless, I posted the results alongside the original on the Mutt wiki page. I’ve also put my files up on my Fedora People space. Feel free to grab and try it, and let me know if it helps you. (You’ll need the evolution-data-server development libraries to compile it.)
In the future I’d like to update this script to aggregate and query my GNOME online accounts as well. There’s a very robust set of functions for doing that, and documentation, so if I could just find the spare cycles, I think I could probably make this happen. Alternately, I wonder if that might make a good addition to the folks-tools package?
For some reason this week I feel like blogging about some of my time-saving tools. I’m certainly not a superstar when it comes to eliminating wasted time. But when I was FPL I had to figure out ways to be more effective so I could spend more time on the tasks that truly required it.
If you run meetings online, you probably already know there are cool projects like Debian’s meetbot to make the process easy. You can give instructions to the bot during the meeting, which are recorded in the minutes, using commands like #info, #agreed, #action, and so on. We use meetbot in Fedora as well, for just about every meeting we run. We even use it to log hackfests and other groovy online get-togethers.
But how do you get those minutes out to subscribers of the team’s mailing list? I’ve seen some people encounter problems doing this quickly. Those problems cause stress, because you want to do a good job for your teammates. If the process is hard, it’s tempting — heck, sometimes it’s necessary — to put it off, so you can do other priority tasks. Then you feel guilty about it later when the minutes aren’t out on time. Wouldn’t it be great to eliminate that stress and guilt?
Now, I have zero doubt that someone could automate a no-time-required solution, and maybe some folks out there use such a system. But in my case, I do like to look over the minutes first, and sometimes prepend a little text at the top. For instance, I might want to add an explanation or extra pointer for context, or a note about something that went wrong mechanically. If you’re in a similar situation, or just not ready for full on automation for some reason, here’s how I do minutes very quickly. Maybe it will help you in the future:
I have my IRC client and my email client, Mutt, up in separate windows.
When the meeting ends, I send the command #endmeeting, and meetbot outputs information about where to find the minutes and log.
I immediately start a new email in Mutt, with the subject “<Name of meeting> recap <date and time>”. This calls up Emacs for me, but it doesn’t matter if you’re using a different email client.*
I copy the three lines (Minutes, Minutes text, and Log) that meetbot outputs, usually with a mouse. (If you’re in screen or something, use its copy function.)
I paste the lines into my email and align them properly if needed.
I use the mouse to copy the link address for the text-format minutes.
In Emacs I run a shell command using Alt+1, Alt+Shift+!.** The command is curl -s -o – <paste the link address from step 6>. This command retrieves the text-format minutes straight from the internet into the buffer.
That looks like a lot of steps, doesn’t it? But since almost everything there is copying and pasting, the actual time to complete this is under 2 minutes. (If you’re a fast typist and good with your editor, it’s more like 30 seconds.) Thus the title for this post!
Many thanks to Kevin Fenzi and the Infrastructure team for providing meetbot functions for our use in Fedora. They’re a big, big help every week for me personally.
* Well, it does matter a little. When I used Evolution, there wasn’t a way to insert the output of shell commands easily into my compose window. That’s totally sensible because Evo isn’t designed for the 0.1% of people who like running shell commands. It’s barely more work to just use a terminal and then Evo’s function to insert file content.
** The Alt+1 means that the following shell-command, run by Alt+Shift+!, will dump its standard output into the current buffer, which is where the email’s being composed. If you were using a terminal with some other email program, you could do curl -O <paste link address> which would retrieve the minutes to a file. Then you could paste the file into your email compose window.
If you’ve been reading my blog for some time, you’ll know that I’m a big fan of using offlineimap in cahoots with the Mutt mail reader and postfix as a local server. Joe writes about using msmtp as a sending utility, or mail transfer agent (MTA) as mail geeks say, but I prefer postfix because it gives me the ability to do slightly more sophisticated deferrals.
If I’m on a plane or otherwise offline, I like to be able to simply hit send without worrying about my connection status. When I connect to a network later, my configuration allows my email to flow to the proper relays as needed. If my system can reach GMail, that account’s email gets sent. If my system can connect to Red Hat’s VPN, email from my Red Hat account gets sent.
In July 2009 I wrote more about this setup here, here, and here.
That’s the longest break I’ve ever taken from blogging: 36 days without a post. At first I was just too busy for a week. Then as time went on, I had more and more things I was thinking about blogging — the pile of ideas kept getting bigger. Finally the sheer size of the pile and number of possible choices became overwhelming. I was a victim of too many choices.
I decided to break the stalemate by putting aside all of the ideas and starting with one new one, which is to talk about the tools I use to be more efficient and productive at work. What I use might not be right for everyone, but perhaps you’ll see some idea here that can help you with your job or your hobby, and be happier with your effectiveness. I was inspired by John Poelstra’s recent post on The Drug of Distraction to write about this subject.
Email is the lifeblood of most information-oriented jobs now, and especially one revolving around open source or a globally dispersed company. Both of these characteristics are part of life at Red Hat, of course! It’s really important to have a way of dealing with email that makes your work more efficient. For me that comes down to two major considerations. There are others, of course, but they’re not quite as important as getting these two right. That’s my email client, and filtering.
First is your email client. For a long time I was a relatively happy user of Evolution, the GNOME personal information manager that helps you manage email, calendar, tasks, and addresses. It’s a robust, full-featured program and I still think it’s a wonderful piece of work even if it has its issues. And honestly, what software doesn’t? However, what I found was that there were a couple of factors that were causing me to lose a lot of time in reading and processing email. By “processing,” I mean performing whatever task is appropriate for a specific piece of email. That might mean deletion, or it might mean responding immediately, or something in between.
Evolution allowed me to have several windows open at once where I could be drafting many different emails. I could make a start, then put the compose window aside to work on something else, or process another more important email before finishing. And for me, that turned out to be a drawback. It encouraged a lack of focus on what I was doing, so that by the end of the day I had a bunch of unfinished responses — and the concomitant feeling that I was a failure because I wasn’t getting the responses out. I was letting my tool usage affect not only my output, but the way I felt about my performance!
(You could of course say that the fault here was mine, not my email application. After all, why can’t I just focus better? And that would be a perfectly fair assessment. However, to change very deeply ingrained behavior takes a lot of time and effort, which I felt I could better spend simply doing work. And it’s probably arguable that changing my behavioral reaction to a windowed environment to optimize for how I deal with email at work wouldn’t necessarily be a net benefit in the general case of my overall computer usage. In any case, I don’t want this to come off as an indictment of Evolution, which I think is a great piece of software.)
Another factor that was hampering me was the way I was accessing my email through the client — IMAP in this case. Each time I wanted to read or reply to email, I was hitting the network. I have a great broadband connection at home using Cox Cable — around 10-12 Mbps downstream and something like 2 Mbps upstream. But even so, latency delays (seek and retrieval) on the server is more of a concern than transmission time, and there was a delay of seconds every time I accessed email. And with hundreds of email messages a day I was looking at — after filtering those I didn’t need to read — I was wasting a lot of time. With 500 email messages and, say, 3 seconds per email on a good network day between retrieval and response (this is a conservative estimate), I was losing 25 minutes a day assuming response time was good. Over the course of a week, that’s over two hours!
Two tools helped me wrangle my email to the point where I could reclaim every bit of those two hours and more. First was the Mutt email client, which is a text-based email reader. Like most text based programs, it has a bunch of keystrokes that you have to learn if you want to use it quickly. It took me about three days of use to get really accustomed to its interface, and most of two weeks to optimize my configuration. But once that was finished, I had conquered the focus problem in dealing with a GUI-based, windowing email client. I was responding to an email in a terminal as a single foreground operation, and when I was finished with my response, I would send it to return to the mail reader. Not seeing other email messages waiting for me (or arriving) as I composed helped me focus on the task at hand, finish it quickly, and return to processing other mail if needed.
The second tool — or rather a combination of tools — that helped me immensely was offlineimap, for syncing email to local storage, and postfix for sending out email from my local system. Using this toolset meant that my network delays were a thing of the past. Recalling email from its stored location is instantaneous, and when I send an email, the response from the server is instantaneous as well. The email gets sent out as a background operation and if anything untoward occurs I get notified as with any other email service.
The other major benefit of offlineimap is like other integrated clients in that it gives me a choice of when to sync my mail. I can either run the synchronization as a periodic background process, or as a one-time operation. I’ve come to greatly prefer doing this as a one-time operation because it keeps me from “riding my Inbox” all day. Nothing will kill true productivity faster than being on email all day, waiting for that next “important” message to arrive. Instead, I try to check email only a few times each day: (1) all mail in the morning after completing any other critical work; (2) important folders only, right before lunch; (3) important folders only, as an afternoon break and limited to no more than a specific time like 10 minutes; and (4) (optional) remaining mail before I leave for the day. This type of routine has helped me restrict my email processing time per day to a very reasonable amount of time — leaving me more time for work requiring heavy focus, deep thought, in-person contact, and/or dedicated writing or composition.
The second consideration for email was filtering. Server side filters are the most effective in my opinion, because among other benefits they don’t rely on me having the same local configuration wherever I’m retrieving or processing email. I have a lot of filters in place both with my GMail account and my internal Red Hat email, to sort each list separately, and prioritize specific kinds of mail such as those that come directly to me from specific people (like my manager).
These filters help me prioritize my email work, so that I don’t end up spending an amount of time on each list proportional to its list traffic. Certain lists, and of course my direct Inbox, are more important to me, and I handle those first. If I’m pressed for time I can defer all the rest. In addition my Mutt email client helps with this process because I have a routine to elevate specific mailboxes to the top of the list through which I cycle. I also practice judicious use of functions such as “mark thread as read,” avoiding spending too much time on discussions that don’t require my attention or input. Every six months or so I also make a practice of archiving old mail so that my mailbox searches, which usually are focused on the recent past, stay fast.
More generally, using this combination of tools and routine helped me conquer an internal focus problem as well as an external latency problem. Again, while my choices may not be right for everyone, they really helped me.
As a remote worker, I travel frequently, and often work away from my home office. My workhorse computer is my laptop, which has most of my important data. Unfortunately, I’m not always near a working open wifi, and that happens frequently in my hometown, which is not a super-wired community.
I do a lot of work via email — in fact, much of my job revolves around email, so I frequently need access to the email stored on the IMAP mail servers I use. I have a Gmail (Google Mail) account, and an account on the corporate email servers as well.
To make sure this mail is available when I don’t have a network connection, I use the offlineimap utility, which synchronizes the email stored on the servers periodically with the email stored on my laptop. This means that I have over 3 GB of email on my laptop, but hey, storage is cheap. It also means that when I read email with Mutt — an extremely functional and flexible text-mode email reader — I’m reading it off my hard disk, which is several orders of magnitude faster than reading over a network. Access to messages and sorting is practically instantaneous, as I’ve mentioned before.
The missing piece of the puzzle, though, is sending email. In the past, I’ve had Mutt set up to send through to the proper SMTP servers for each account. This isn’t hard, since Mutt lets you add hooks as you access an account, a folder, or even compose or reply to email on a per-message basis. But the problem is that network access is always slow, and the user interface blocks for many seconds while the email is sent — and I don’t really want to wait, I want to move on with my life and let the computer deal with that stuff, the way it’s supposed to.
There’s also another problem I need to solve — I want my email sending to also work when I’m on a plane. I don’t mean that I expect that email to actually go anywhere, but I want it to look that way to me and my Mutt reader. I don’t want to worry about whether I’m online or offline, I just want to write my email, and feel confident it will be sent when I’m next online.
Enter Postfix — the secure and highly configurable sendmail alternative, originally written by the inestimable Wietse Venema (of tcp_wrappers and other fame). By configuring a local Postfix mail server to handle the actual transfer of email, when I send email, Mutt pops right back instantaneously so I can continue doing my work. And if I’m not online, Postfix will hold that email and try again later when the network is up. And of course, I need this to work for multiple email accounts I use.
Once I started puttering around with Postfix’s many configuration options, I started to understand enough to be dangerous. I looked at the Postfix documentation online, and also a couple of HOWTOs to see how various subsystems work. Here’s how I did it:
Install Postfix, using PackageKit:
pkcon install postfix
pkcon install postfix
Set the system to use Postfix as the new MTA:
su -c 'alternatives --config mta'
su -c 'alternatives --config mta'
Configure Postfix with the following options in /etc/postfix/main.cf:
This did take a little trial and error, I confess, but the results are clear. This set of options allows Postfix to relay email based on the sender address, based on a map file I’ll show you below; enables SASL authentication for Postfix as an SMTP client, using TLS based on a policy map; and allows it to check certificates for validity using the certificates already installed on the system as part of the ca-certificates package.
Make a new file /etc/postfix/sender_relay, with the following contents, making changes as needed to reflect your Gmail and corporate account:
# per-sender provider; see also /etc/postfix/sasl_passwd
Note that my corporate email is being tunneled through SSH via a local forward, so my email server actually attaches to port 25025 on the localhost address to send email. If your corporate email server is available on the internet, you would probably just use [mail.corp.example], and make sure to include the appropriate TLS stuff later where needed.
Make a new file /etc/postfix/sasl_passwd, which will contain the authentication information for each of your accounts:
Add a line for each email address that needs to do SMTP authentication. (Update: Thanks, Bo, for the fix to this file.)
Make a new file /etc/postfix/tls_policy, which will include information on the way in which each account requires Postfix to use SMTP authentication as a client to the appropriate relay host:
GMail requires the use of TLS, so the encrypt rule here ensures that Postfix will do TLS-based authentication when it connects.
Rebuild all of the affected map files:
Now Postfix is configured. At this point, I needed to reconfigure Mutt, to remove all the smtp_url options I was setting before. By default, Mutt simply uses the system’s locally-installed sendmail to send email; and the Postfix package provides an equivalent sendmail.postfix binary, which the alternatives command above sets up with an appropriate link, so everything on the system continues running like a top without any further problems. Now Mutt will simply call sendmail blindly, and Postfix will handle all that mail as configured above.
Then I just had to turn off the running Sendmail process, and start up Postfix instead:
Turn off Sendmail:
service sendmail stop
service sendmail stop
Start up Postfix:
service postfix start
service postfix start
UPDATE: You’ll also probably want to have Postfix start by default at boot time:
chkconfig postfix on
chkconfig postfix on
Then I sent a couple of messages to make sure everything was working properly, using tail -f /var/log/maillog to watch the results. Now, to get really cool, I decided to add a hook for NetworkManager to take care of reconfiguring Postfix on the fly to automatically defer SMTP activity when offline, and to turn off that deferment and flush the waiting queue when returning online. I simply installed a script into /etc/NetworkManager/dispatcher.d that looks like this:
I don’t usually take a lot of time on my blog to write about apps, but as I was doing some administrative cleanup this morning, I realized I needed to give some props. I’ve been using the mail reader Mutt for about a year now, coupled with the offlineimap utility, and as a result my dealings with email are far more efficient than they were before. I used to be a heavy Evolution user, but a combination of oddities in the way Evo deals with folder refreshing, and its use of memory, prompted me to try other things. Now I use offlineimap to sync my remote IMAP stores to my hard disk, which makes reading email incredibly quick since Mutt now reads from the hard disk and not the remote IMAP server. I can run offlineimap as a one-time process, before I take my laptop on the road, or I can let it run regularly during the day in the background.
When I send email, I pass it back up to my IMAP servers, which tends to cause a delay while I wait for the email to go out. This year, my goal is to get Postfix installed and configured on my laptop to handle sent email, so that delay will be eliminated. Postfix will take care of the actual email delivery in the background while I get on with other things.
Mutt itself is a great email reader to use. There is a very brief delay for very large folders while it reads information off my hard disk, but it’s only a few seconds for a folder with tens of thousands of messages (I have many folders like this). Operations on folders after that are practically instantaneous. My latest favorite toy is the l (that’s the lowercase letter L, or “limit”) command, which lets me show just specific email messages from the folder listing. A very useful limit is ~d DD/MM/YY, where DD/MM/YY is a date input. (The format for the date is obviously a little different than some people use day to day, but it’s easy enough.) I can easily go back to my Sent folders and look through the mail I sent on a particular day.
There are many other limits you can use, like ~f or ~t for locating messages from or to a name or address, or ~s for limiting by subject line, or ~b for finding messages with certain text in the body of the message. Again, the searches are incredibly fast when running off the hard disk, and you can stack them as well, as in ~d 01/06/09 ~t John, which would find email to John sent on 1 June 2009.
Of course there are a ton of other commands available with Mutt. And because it has an extensive set of configuration options, including operation hooks that allow you to reset the environment as you work, there’s almost no end to the usability improvements you can make based on your own habits and mail setup. With or without a pedigree, this dog’s definitely a champion!