As a PhD student, one of the first bibliographic annoyances I encountered was when I had to format a paper using the APA system, which requires titles to be in sentence-case. This means only the first word of a phrase and proper nouns are capitalized. Previously, I had kept the titles of my citations in title-case. The consequence of having to use the APA format was the need to then go in and manually lower case all words that were not proper-nouns in my bibliographic database. However, once this work was done, I realized keeping my data in sentence-case was preferable, as title-case essentially loses information. Yet, this still requires me to manually lowercase some words for automatically captured sources. I am not aware of any bibliographic software that handles this issue well, and the good folks at Zotero have an interesting bug ticket open on the issue.

On Friday, while I was doing my weekly fixes to the automatically captured sources in my field notes/mindmap/bibliography, I thought to myself that there are plenty of word lists around, such as those used by spellcheckers, and couldn't I finally automate this menial task? However, I knew that I use lots of proper nouns that probably do not appear in common dictionaries. Therefore, I applied Python's Natural Language Toolkit tokenizer and parts of speech tagger to the text of my dissertation to create a custom word list of proper-nouns that I use. These are used with the dictionary found on my system at /usr/share/dict/american-english to transform a title-cased sentence into a sentence-cased sentence. Basically, if the word is in my custom list, is in the word list only as a capitalized word, or not in the word list at all, it merits capitalization, else lower-case it. The code is available as a module to the Busy Sponge component of the Thunderdell bibliographic tools. It works fairly well and will certainly make that end of the week menial task all the more easier.

One of the last significant steps for an author is to compile an index, unless they opt to have someone else do it. Publishers often recommend authors create the index as they know the material best. However, while professionals have sophisticated -- but proprietary -- tools to help them, authors are offered only the techniques of using index cards or spreadsheets. Neither of these options is appealing to me.

I thought it would be nice to simply compile a list of entries in the form of topic (page#|see) (sub)topic and let a script do the rest. It's a bit of a hack but it does the job and can collapse all subentries below a particular threshold. I like having specified subentries, even if there is only one or two of them for a particular entry:

Apology
    and leadership, 124
    "Sorry but...," 54

But if the publisher says they want those collapsed, it is easy enough:

Apology, 54, 124

For the most part, I wrote my dissertation and book manuscript using a simplified version of markdown complemented with biblatex citations. Because it was a simple text file, it made managing the edits to the manuscript very easy. I could do global textual replacements trivially. Also, obviously, it was trivial to generate PDFs, HTML, etc. Using Mercurial, I could take advantage of some nice features like the "attic" extension which allows me to keep change sets on the side to be applied only when appropriate. So, for example, the changes necessary generate HTML were kept in the attic and would only be applied when I wanted that.

Unfortunately, once the manuscript went into the MIT Press system, I had to use Microsoft Word. As much as much as the Word document format annoys me, I understand it is widely used, and I can't think of an easy alternative that also provides the capability for editorial annotations. Nonetheless, I had a difficult time seeing changes in Microsoft Word, and want to backport the changes into my source files. And, there does not appear to be a nice textual difference tool for Word documents.

I have posted a small Python script that makes use of antiword and dwdiff but also gives me context on either side of the change. It, of course, doesn't work well with formatting, but is useful and will generate output like the following:

   reflects {-the-} [+a+] stabilization
   a {-number of pragmatic questions: it-} 
     [+project was conceived. It+] would
   there {-will-} [+would+] be
   article {-will-} [+would+] be
   linked {-to from-} [+via+] a

I have tagged a new release of Thunderdell, the Freemind mindmap to biblatex utilities. Improvements include:

• The ability to emit biblatex or BibTex.
• More consistent/improved Unicode support, particularly in bibliographic keys.
• New feature for requesting either the long or short URL for Wikipedia sources.
• New feature for requesting that only URLs for exclusively online sources be emitted.
• New test for online sources.
• More biblatex fields, such as original publisher and year.
• Faster processing by using lxml, but can still fall back to etree.
• Use of optparse.
• The "essential node" function was removed.

I use VirtualBox to run a Windows guest, and unfortunately the shared clipboard between the two is sometimes buggy. I recently posted a script for a very robust clipboard using a network shared file. Also, I'm doing the final checks on the book manuscript and the Chicago Manual of Style has odd and confusing rules for specifying ranges of page numbers. The CMS page range validator looks through my sources files and prints out any likely to be counter to Chicago style.

Because I use speech recognition software (SR) I'm forced to tangle with proprietary software and formats; this provides a continuous reminder of the benefits and joys of Free Software. However, I have learned a few things about maintaining a Windows system for SR over the past five years.

In 2004 I began using continuous SR with ViaVoice on a headless Shuttle box accessed over VNC. (This was a big improvement over the discrete speech system I used 10 years before.) Despite the ameliorative provided by imaging the OS partition (PING is great for this), Windows was still a dreadful thing to maintain; the advent of virtualization has been a blessing. And up until the beginning of this year, I relied upon Win2K so as to keep a lean and portable OS. However, security and software support for Win2K is ending and the excellent VirtualBox 2.* software permits one to emulate a consistent hardware profile (including the bios); this allows me to placate XP's annoying validation system.

I presently use NaturallySpeaking 10.1. While the underlying recognition is often remarkable, the user experience and Nuance's support are dreadful. To have useful macro support one must pay hundreds of dollars more for a "professional" version to a company that charges its users for tech support because of its own breakage, which, if reported as bugs, are ignored. Fortunately, there is a friendly FOSS community and DragonFly is an amazing (Python-based) macro application that helps me get around the worst annoyances in NaturallySpeaking.

Then there is the matter of application support. While coders might be content with Emacs or UltraEdit, I dictate prose and want a visually meaningful processor: paragraph/heading styles, a spelling and grammar checker, word counter, etc. Lyx, Amaya, OpenOffice, and Abiword are not "Select-and-Say" capable applications (i.e., not useful with NaturallySpeaking). That leaves Microsoft Word and its loathsome ".doc" binary format. These binary files are impervious to the more useful features of versioning systems, or simple scripting. If I need to fix the capitalization of a term in my manuscript, I have to manually open each chapter and do "find/replace" rather than fix it with a simple one-line command (or with KFileReplace). While I had some hope the new ".docx" format would be useful (it is easy enough to unzip and parse) making sense of it is an outrageously difficult task (particularly lists). So, for years now I've been writing pseudo-LaTeX in doc files, converting them to text via antiword and processing it from there.

However, I recently accumulated enough Microsoft Word hacks to turn it into a decent text editor.

1. Set the default save format as plain text and its default font to something nice like Andale Mono.
2. Bind {control-v} to this PasteUnformattedText() macro.
3. Bind {control-s} to this FileSave() macro to get rid of the annoying "you will lose your formatting saving to text" dialog.
4. Office XP doesn't use UTF-8 encoding by default and nags you with a dialog every time you open such a file. UTF-8 is the encoding used by every other sensible application of late. Make it the default with this registry edit, but realize it uses the byte-order-mark (BOM) which even otherwise sensible applications get confused by. When processing the text, you can remove it in Python with: line = line.lstrip(unicode(codecs.BOM_UTF8, "utf8")).
5. You can even "syntax highlight" your text with VBA: this AutoOpen() macro shows how editing markdown and LaTeX visually looks much like what I was seen before, but it is now an open format UTF-8 encoded text file!

In 2002 I began thinking about how to best capture and share the many web-pages and small tasks of the day. I thought of it as a "busy sponge": logging bookmarks and tasks to my team page with a minimum of typing. Furthermore, I wanted to tag each entry with a keyword which could then be used in queries. I posted an implementation in 2003 which was complemented by the fact that the tasks on my team page were syndicated (via RSS) -- and used to generate my "two minute reports" at the weekly staff meeting at MIT. This was a number of years before the notion of micro-blogging became popular.

Two interesting features have further matured: I wanted it to fetch the title of a URL -- typing HTML was a hassle -- and I wanted to tell it which of my pages to log the entry to: my personal weblog or work team page. For example:

urd:/home/reagle > b http://pesto.redgecko.org/dispatch.html j python Noted ^

is a sponge of a URL to my "j" (work) page where "^" becomes the hypertextual page title, resulting in:

<li class="event" id="e090313-f7fd">090313: python] <a href="http://pesto.redgecko.org/dispatch.html">Noted URL dispatch &mdash; Pesto: a library for WSGI applications</a></li>

With BusySponge 0.5 (now distributed as part of Thunderdell), this has matured into a set of classes for webpage screen scraping and a set of logger functions. So, for example, I might sponge a comment about a URL and indicate it should log it to my bibliographic mindmap (Thunderdell) and it will do its best to fetch the page author, title, date, publisher, permanent link, excerpt of first substantive paragraph, etc. The default heuristics do a surprisingly decent job -- certainly better than typing it from scratch -- and the specific scrapers (e.g., Wikipedia, MARC email archives) are quite good.

I'm releasing the latest set of Freemind Bibliographic Extraction scripts. I'm calling it a "1.0" release because:

1. I decided to give it a funny name.
2. I now address an unlikely but long-time screw case.
3. This and other cases are now tested by doc_tests.
4. I updated the generation of bibliographic keys to remove 'and' from the author portion and always include a title suffix -- instead of just when there is a collision. Keys are now a bit more terse and more stable.
5. I now emit biblatex, a much more complete and powerful bibliographic format.

It also now has its own webpage, from which you can download it.

So much time in teaching is spent on trivial but time-consuming tasks. Having taught for a couple of years now, some things have gotten easier, but many chores remain. I've tried various free tools, and there are many proprietary solutions, but none satisfy. The following are personal -- and poorly documented -- hacks that are useful to me.

Grade sheet

I used to use a sophisticated Excel spreadsheet I found on the Web, but the author has since taken it proprietary/commercial, plus it didn't support a simple point system (e.g., 100 points allocated throughout the semester). Also, my sheet now accommodates some of my idiosyncratic policies for class responses and attendance (e.g., dropping the lowest X grades, Y freebie absences).

Grade Reports

This Python script makes use of the XLRD library to read an instance of the above, and generate a report for the whole class or a single student, to the console or as a messages in my drafts mailbox.

Class Calendar

Given the duration of the semester, the days on which a class meets, university holidays, and the class on which an assignment is due, this generates a calendar for the syllabus. (A surprisingly time-consuming manual task.) I hope to incorporate Jewish holidays into this if I can find such a library.

Mailbox Prettyprint

Instead of using the inaccessible and proprietary BlackBoard product, I ask students to e-mail their responses which are automatically sorted into a mailbox. Before class, this script will prettyprint all responses sent since the last class to an HTML page, including (now) MS Word attachments.

I just discovered Python's awesome dateutil package which implements much of the iCalendar standard, including recurrences! Consequently, it's trivial to generate a calendar for the days classes meet. I assume with a little work one could even handle the holidays. In any case, here's an example:

#!/usr/bin/python2.5

from dateutil.rrule import *
from dateutil.parser import *

sem_start = '20070903T140000'
sem_end = '20071212T140000'
days = MO,WE

meetings = list(rrule(WEEKLY, wkst=SU, byweekday=(days),
    dtstart=parse(sem_start), until=parse(sem_end)))
for meeting in meetings:
    print meeting.strftime("%b %d %a")