Integrating a VT220 (or similar DEC terminal) into my OS X workflow | Part 2
This post is the second in a series on my integration with a VT220-like (VT510) terminal, as my ‘daily driver’. The first post in this series is available here.
Applications
The tasks I am able to perform on my terminal are quite varied. I can
- Check my email and file it extremely quickly, using
mutt. This includes advanced features like S/MIME and GPG encryption, full-text search (extremely fast!), and dealing with mailing lists sensibly. - Convert my emails to task reminders in my task management tool of choice, OmniFocus.
- Read RSS feeds.
- Use the web (and other protocols!) in a text-based browser.
- Do almost anything else you might think to do on the command line (including editing some of this blog post).
In this post, I will be exploring mutt, for regular email reading, writing and management.
tmux Customisation
First things first, the DEC VT510 is a CRT monitor. It is succeptible to burn-in, leaving marks of any frequently-shown text visible on the monitor, even when the power is off. This can be very distracting!
The contents of my ~/.tmux.conf file alleviates this risk by changing from the default status bar with a light background, to one with a dark (black) background, and displaying only a modified clock in the right hand corner — there is no usual list of windows. To remove the window list in the tmux status bar and perform these other colour customisations, I have the following in my ~/.tmux.conf:
# Customise status bar
set -g status on
set -g status-right "%H%M %d %b"
set-option -g status-left ""
set -g status-bg black
set -g status-fg white
set -g window-status-current-format ''
set -g window-status-format ''
These commands will work in any any tmux environment, not just when used on a VT510 terminal.
Mutt
My full mutt setup is out of scope of this article, however there are a few features that are useful for me in using the DEC VT510 terminal.
Small screen, big world
The first issue to contend with was that I didn’t want to be changing all of my settings just to accomodate my VT510. To make things fit the screen, I didn’t want to be using my luxurious 15” screen with the same settings of an 80 column terminal.
Luckily, mutt allows you add some intelligence to its configuration by referencing shell scripts (however no arguments can be passed). Here’s my terminal-dependent setup in ~/.muttrc:
source '~/.mutt/vt220.sh|'
And vt220.sh itself is:
if [ $TERM = "vt220" ]
then
echo "set ascii_chars=yes"
echo "set wrap=80"
echo "set reflow_wrap=80"
echo "set pager_index_lines=5"
fi
if [ `tput cols` -lt "75" ]
then
echo "set sidebar_visible=no"
fi
Special Characters
There are a few settings required to support text-only email reading, particularly on a terminal that doesn’t know a character set such as UTF-8 (modern character sets support advanced characters like emoji). The DEC VT510 supports mainly ASCII.
To ensure that my emails are as readable as possible, I need to filter out undesireable characters - even a “smart” (curled) quote (e.g. “ — These come in single and double varieties, both openening and closing). I have a regex filter that in my ~/.muttrc is enabled by set display_filter="~/.mutt/filter.sh". filter.sh itself follows:
#!/bin/bash
output=$(tee)
# Removal of all smart quotes and weird spaces
output=`echo "$output" | sed "s/[’‘]/\'/g"`
output=`echo "$output" | sed 's/[”“]/"/g'`
output=`echo "$output" | sed -E "s/[[:space:]]+/ /g"`
if [ "$TERM" == "vt220" ];
then
# Start of a project to remove emoji and replace with 'emoticons'
output=`echo "$output" | perl -pCSD -e 's/\x{1F600-\x{1F608}/:)/g'`
output=`echo "$output" | perl -pCSD -e 's/\N{U+1F609}/;)/g'` # Can also be \x{1F609}
output=`echo "$output" | perl -pCSD -e 's/\N{U+20AC}\s?/euro /g'`
output=`echo "$output" | perl -pCSD -e 's/\x{1F610}-\x{1F612}/:|/g'`
fi
echo "$output"
This is a work in progress, however fixes a lot of characters that otherwise appear as underscores in my reading of mail.
HTML Alternatives
A lot of email today comes as HTML, but often with a text alternative that mutt and other plaintext programmes can utilise. Often this alternative is just as readible as the HTML (sometimes more so!). Sometimes however, it may be an annoying and terse message telling you only to upgrade your email client to read the HTML email (no chance!). There are ways around this.
In ~/.muttrc, I have:
message-hook . "unalternative_order *; alternative_order text/plain text/enriched text/html"
message-hook '~f bad_sender@domain.com "unalternative_order *; alternative_order text/html"
where bad_sender@domain.com is the name of an inconsiderate sender who does not send useful text email. Looking at you, ABC!
In ~/.mailcap I have:
text/html; w3m -I %{charset} -T text/html; copiousoutput;
This ensures that mutt knows how to deal with HTML, whenever it is selected (including if you manually select the HTML component of an email).
URLs in emails
Occasionally you may come across emails that contain links you’d like to visit. Mutt can parse these for you, using a helper programme called urlview. To extact URLs from emails, add the following to your ~/.muttrc:
macro pager \cb <pipe-entry>'urlview'<enter> 'Follow links with urlview'
This will allow you to press ‘control-b’ (twice in a row, if using tmux) and extract URLs in an email. By default these will be opened by your nearest web browser — if you’re on a VT510 that’s not preferable! To confiure urlview, create the file ~/.urlview:
COMMAND /usr/local/bin/lynx -cfg=~/.config/lynx/lynx.cfg %s
This will launch the text-based browser lynx with the url selected from urlview in mutt. You may remove the -cfg portion of the command — this specifies a configuration file (more on this later).
Task Management
Emails are often the first notification of needing to complete a task, often administrative and easily batched, and whose optimal execution time does not often coincide with their arrival in my inbox. I like to keep a clean inbox, but keep track of things I have to do. To do this, I have a series of scripts that allow me to place tasks (and references to their genesis, email) in OmniFocus. I further have a button in OmniFocus that allows me to select a task and view the email that it came from.
Without detailing them here, these features rely on my mutt setup using offlineimap for keeping a copy of my email locally, and notmuch for fulltext search.
To create a task from an email, I press control-l on my keyboard. ~/.muttrc is configured to execute a script to pipe this for text input
macro index,pager \cL "<enter-command>unset wait_key<enter><pipe-message>mutt-to-omnifocus.py<enter>" \
"Create OmniFocus task from message"
The script mutt-to-omnifocus.py is the following code. I did not write this, but I have somewhat modified it to support emails with encoded subjects. I have forgotten where I obtained it so I am unable to credit, sorry.
#!/usr/bin/env python
import sys
import os
import getopt
import email.parser
import subprocess
from email.header import decode_header
import re
def usage():
print """
Take an RFC-compliant e-mail message on STDIN and add a
corresponding task to the OmniFocus inbox for it.
Options:
-h, --help
Display this help text.
-q, --quick-entry
Use the quick entry panel instead of directly creating a task.
"""
def applescript_escape(string):
"""Applescript requires backslashes and double quotes to be escaped in
string. This returns the escaped string.
"""
if string is not None:
# Backslashes first (else you end up escaping your escapes)
string = string.replace('\\', '\\\\')
# Then double quotes
string = string.replace('"', '\\"')
return string
def parse_message(raw):
"""Parse a string containing an e-mail and produce a list containing the
significant headers. Each element is a tuple containing the name and
content of the header (list of tuples rather than dictionary to preserve
order).
"""
# Create a Message object
message = email.parser.Parser().parsestr(raw, headersonly=True)
# Escape special shell characters
#applescript2 = re.sub("(!|\$|#|&|\"|\'|\(|\)|\||<|>|`|\\\|;)", r"\\\1", applescript)
# Extract relevant headers
list = [("Date", message.get("Date")),
("From", message.get("From")),
# ("Subject", message.get("Subject")),
("Message-ID", message.get("Message-ID"))]
try:
sub, encoding = decode_header(message.get("Subject"))[0]
sub = sub.replace('\n', '');
pipe = subprocess.Popen(['/Users/joel/bin/item_name.sh', sub], stdout=subprocess.PIPE)
subject, error = pipe.communicate()
list.append(("Subject", subject.rstrip('\n')))
except KeyboardInterrupt:
print ""
sys.exit()
return list
def send_to_omnifocus(params, quickentry=False):
"""Take the list of significant headers and create an OmniFocus inbox item
from these.
"""
# name and note of the task (escaped as per applescript_escape())
name = "%s" % applescript_escape(dict(params)["Subject"])
note = "\n".join(["%s: %s" % (k, applescript_escape(v)) for (k, v) in params])
# Write the Applescript
if quickentry:
applescript = """
tell application "OmniFocus"
tell default document
tell quick entry
open
make new inbox task with properties {name: "%s", note:"%s"}
select tree 1
set note expanded of tree 1 to true
end tell
end tell
end tell
""" % (name, note)
else:
applescript = """
tell application "OmniFocus"
tell default document
make new inbox task with properties {name: "%s", note:"%s"}
end tell
end tell
""" % (name, note)
# Use osascript and a heredoc to run this Applescript
os.system("\n".join(["osascript >/dev/null << 'EOF'", applescript, "EOF"]))
def main():
# Check for options
try:
opts, args = getopt.getopt(sys.argv[1:], "hq", ["help", "quick-entry"])
except getopt.GetoptError:
usage()
sys.exit(-1)
# If an option was specified, do the right thing
for opt, arg in opts:
if opt in ("-h", "--help"):
usage()
sys.exit(0)
elif opt in ("-q", "--quick-entry"):
raw = sys.stdin.read()
send_to_omnifocus(parse_message(raw), quickentry=True)
sys.exit(0)
# Otherwise fall back to standard operation
raw = sys.stdin.read()
send_to_omnifocus(parse_message(raw), quickentry=False)
sys.exit(0)
if __name__ == "__main__":
main()
This script calls a shell script called item_name.sh, which takes input to determine the task’s name in OmniFocus. The default is Mutt: <email subject>
OS X has Bash 3 available at /bin/bash, which will not work for this script. item_name.sh relies on Bash v5 being installed (typically via homebrew):
#!/usr/local/bin/bash
RESET="\033[0m"
BOLD="\033[1m"
RED="\033[31m"
MUTT="Mutt: $1"
read -e -p "$(echo -e $BOLD$RED"Task Name: "$RESET)" -i "$MUTT" title </dev/tty
echo "$title"
The combination of these two scripts will allow you to enter text to change the name of the task being inserted into OmniFocus, and quickly insert the task into the OmniFocus inbox.
The following AppleScript, when compiled and placed into your OmniFocus script folder (~/Library/Application Scripts/com.omnigroup.OmniFocus3), will allow you to open a mutt instances within iTerm.app (my terminal programme of choice). I have to fiddle significantly with the timings (delay 2 statements, etc) — your mileage may vary on your own system.
tell application "OmniFocus"
tell front document
tell content of document window 1 -- (first document window whose index is 1)
set theSelectedItems to value of every selected tree
end tell
repeat with anItem in my theSelectedItems
if note of anItem contains "Message-ID" then
set noteBody to note of anItem
set MessageID to do shell script "echo '" & noteBody & "' | /usr/bin/python ~/bin/message_id.py"
tell application "iTerm"
activate
try
set newTab to true
set t to the first window
on error
set newTab to false
set t to (create window with default profile)
end try
tell t
if newTab then
set tt to (create tab with default profile)
else
set tt to (current tab of t)
end if
set s to current session of tt
-- Open Mutt
tell s
write text "/usr/local/bin/mutt"
end tell
-- Run "F8" keypress
tell application "System Events"
key code 100
end tell
-- Search for the MessageID
tell s
write text "id:" & MessageID
end tell
-- Wait one second, then hit 'enter' to open the email
delay 2
tell application "System Events"
key code 36
end tell
end tell
end tell
else
if note of anItem is "" then
set dialogText to "No text in note"
else
set dialogText to "Note does not appear correctly formatted: missing Message-ID"
end if
display dialog dialogText with icon stop ¬
with title ¬
"Error" buttons {"OK"} ¬
default button 1
end if
end repeat
end tell
end tell
return
This AppleScript relies on a script called message_id.py:
#!/usr/bin/python
#
# mutt_flagged_vfolder_jump.py
#
# Generates mutt command file to jump to the source of a symlinked mail
#
# Copyright (C) 2009 Georg Lutz <georg AT NOSPAM georglutz DOT de>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
import optparse
import os
import re
import sys
import types
from subprocess import call
from email.header import decode_header
VERSIONSTRING = "0.1"
def parseMessageId(file):
'''Returns the message id for a given file. It is assumed that file represents a valid RFC822 message'''
prog = re.compile("^Message-ID: (.+)", re.IGNORECASE)
prog2 = re.compile("^Subject: (.*)$", re.IGNORECASE)
msgId = ""
subject = ""
for line in file:
# Stop after Header
if len(line) < 2:
break
result = prog.search(line)
l = len(decode_header(line))
s = ""
for i in range(0,l):
s += decode_header(line)[i][0] + " "
s = s[:-1]
result2 = prog2.search(s)
if type(result) != types.NoneType and len(result.groups()) == 1:
msgId = result.groups()[0]
if type(result2) != types.NoneType and len(result2.groups()) == 1:
subject = result2.groups()[0]
if subject != "" and msgId != "":
prog3 = re.compile("([\[\(] *)?(RE|FWD?) *([-:;)\]][ :;\])-]*|$)|\]+ .*$", re.IGNORECASE)
result3 = prog3.sub('', subject)
subject = result3
break
return (subject.replace('\n',''), msgId.strip("<>"))
subject, msgId = parseMessageId(sys.stdin)
try:
print msgId
except KeyboardInterrupt:
print ""
sys.exit()
Result
After this, you should be able to see your email and integrate it across your other workflows. I’m not posting my inbox (the shame of never being able to reach inbox zero in inescapable!), so please enjoy this image of my RSS reader and a web browser running side by side in tmux, to whet your appetite for the next post in this series.
