Force Directed Emails Visualisation Using arbor.js

I have recently been playing around with ways of visualising my sent emails for an assignment at university. I have over 6 years worth of sent mail sitting in Gmail, collecting electronic dust (and occasionally being dusted off and looked at).

I downloaded all of my sent mail from Gmail (over 3000 conversations), making  use of my university’s internet connection. I made use of thunderbird for this, as it stores emails in flat files on disk. I made a simple python script that takes this raw file and converts it into JSON.

import mailbox
import json

# path to you sent mail mbox
mb = mailbox.mbox('Sent Mail')

fout = file('sent.json', 'w')
items = []
fields = ['date', 'subject', 'to']

for i in range(len(mb)):
    obj = {}
    for item in mb.get_message(i).items():
        if item[0].lower() in fields:
            obj[item[0]] = item[1]
        mb.remove(i)
    items.append(obj)

json.dump(items, fout)
fout.close()

A nice javascript vis library called arbor makes it easy to take a graph and apply a force directed algorithm to it. It allows you to only worry about the visualisation side by separating the layout computation from the graphical display. Using this library and some basic javascript processing I was able to produce the following display.

I have obscured the recipient names for the sake of their privacy only (my own privacy has already been destroyed by the likes of such sites as Facebook – see: Facebook is an appalling spying machine).

The line widths between time periods and recipients relates to the number of emails sent to that person. Nodes can be dragged about and added in by dragging the sliders to alter the time range.

The most interesting part of this process was probably making a wordle based on all of the recipient names for all of the sent emails. The wordles changing from year to year showed the people that I was in the most communication with and perhaps most important in a given time in my life.

To leave you with an idea – someone needs to make a web app to anonymize wordles (ie replace words with random, but unique common words of the same length). A google search did not find anything meaningful.

** EDIT **
Lots of people have been asking for the code. I will provide the link to my github project but it will not include the data file. https://github.com/oughton/email-wordle/