Refresh

About a week ago some friends launched Twittertinget.dk, a sleek app that aggregates tweets from politicians in the Danish parliament. In the name of democracy, the web application has a vital flaw though: It only lists tweets from politician, not to them. That makes for one-sided communication.

As a friendly gesture, Socialsquare asked me to create a complementary service, which does the exact opposite thing: Let’s you know what people are tweeting to their elected representatives. This ended up in Folketwinget.dk, a simple site hosted on Google AppEngine.

Now, aggregating tweets to and from the Danish Folketing seems like a faily specific, but it turns out that such an aggregator of can easily be used by any other group — so we’ve open sourced the code over on Github. The code runs directly on AppEngine so it’s easy to host and have running in a few minutes.

For example, if you have a company called 23 with a few employees on Twitter it might be cool to show all the incoming and outgoing tweets from the company.

Get started by installing the Google AppEngine SDK, start up a terminal and browse to the AppEngine folder on your local machine. Then…

1. Get the code
Run this command to download the code. (Obviously, you’ll want to change the folder name from 23tweets to something else.)
git clone git://github.com/steffentchr/twitterengine.git 23tweets

2. Update app.yaml and create config.yaml
Open up the file called app.yaml in the newly created folder and change the application name in the very first line of the file. In this example, we’ll change it from twitterengine to 23tweets

Also, rename the file called config.yaml.sample to config.yaml

3. Create an AppEngine project
Open up GoogleAppEngineLauncher (it’s called that on Mac anyway). Right-click in the window and select “Add existing..”. Now select the folder we’ve just created and click OK.

This is all you need to actually get the application up-and-running. Click the Run icon to start your development server and open up http://localhost:8000 in a browser.

4. Configure the application for your dirty purposes
Before you can import some tweets into your application you will need to update the configuration. Open up config.yaml in a text editor and change what you think is necessary. For my example, I’ll use this configuration:

site_name: 23 tweets
site_tag_line: What is up with 23?
site_domain: 23tweets.appsport.com

twitter_username: 23pull
twitter_password: secretpassword

users:
- steffentchr
- mygdal
- mmmmmariaaaaa
- abemad
- guan
- burtblancher
- 23

hashes:
- 23
- 23hq
- twentythree

Basically: Use the user account of 23pull to get all tweets to and from @steffentchr, @mygdal, etc and all tweets with the hash #23. Then call the site “23 tweets”.

(This could just as easily have been used to create a Folketwinget-sibbling for the Norwegian parliament or for aggregating conversations of everything involved in a specific open source community.)

4b. Import the tweets
Now, go to http://localhost:8080/import to trigger an import of everything specified in the config file. When you return to http://localhost:8080/ you should see all a batch of the latest twitterings.

When you’ve finished step 5., the application will automatically import tweets every five minutes. The frequency can be changed by editing cron.yaml.

5. Deploy your app
Before you can deploy your new twitter aggregator to the wide web, you will need to create the application with Google. Go to the AppEngine dashboard, click the “Create an application” button and fill in the form. Remember to use the correct application identifier, i.e. 23tweets.

When your application has been succesfully created, go back to GoogleAppEngineLauncher and right-click your project. Then select the deploy option and sign in to your Google account. That’s it. It’s online.

Again, you’ll need to visit http://projectname.appspot.com/import to import tweets into the online database and then go to http://projectname.appspot.com/ to see the result.

6. Customizing the design
You’ll find that your site is in Danish and generally doesn’t look the way you want it to. Changing the design though is fairly easy: Modify index.html and resources/style.css for your purposes. If you need anything fancy, the html file is structured using the Django templating system.

Update April 24th 2009: Originally, this post contained information for running external cron jobs to import tweets. Today, the code has been updated to utilize AppEngine’s new native cron feature which makes it all happend automagically..

ReadWriteWeb just posted an intriguing article about the next generation of e-government. The post basically lists the arguments from an upcoming scholarly article about the subject. The main argument is simple: Government shouldn’t spend it’s time building web sites, instead it should “focus on creating a simple, reliable and publicly accessible infrastructure that exposes the underlying data”. Rather than building and maintaining a website of upcoming bills at ft.dk the Danish parliament should make the data available and have others mash it up in usable ways.

Now, I’m not a deregulating liberalist, and I’m absolutely certain that a strategy of only making data available to private entrepreneurs simply wouldn’t work in a Danish contexts. But I’m also convinced that a mandated policy of making data easily available to developers has explosive potential.

Let me list a few examples. The best Danish mapping service, Findvej.dk is a mashup of both publicly and privately owned data. It was built by one guy in with a good idea and access to the necessary data. The data was probably hard to find, and the service would be much better with even more accessible data. But you get my point.

There are hundreds, if not thousands, of possible counter-examples and I’ll just pick one. The weather. Denmark has a tax-funded, public weather service, which generates plenty of data every day, and this data is available to everyone through a web browser. Now, given all the data from the weather service, would I be able to build a better website serving all the same user groups as DMI? Probably not. But given access to that data I would be able to build a service which suits my needs much, much better that DMI.dk. And most likely, my service would serve others as well.

Why isn’t there an Danish version of EveryBlock? The reason isn’t lack of interest. The reason isn’t lack of data, because statistics and incident reports are being kept by the police. The reason is that the data isn’t accessible as an open building block (that metaphor didn’t work).

This presents a major challenge to government, and even if I’ve used Danish examples, those challenges are probably both very global and very local. The past few years have shown convincingly that web developers are hungry for interesting data to reinvent by merging across data sources, and this potential could be easily harvested by some sly government initiatives.