Announcing TwitterEngine: Your own twitter aggregation site on AppEngine
About a week ago some friends launched Twittertinget.dk, a sleek app that aggregates tweets from politicians in the Danish parliament. In the name of democracy, the web application has a vital flaw though: It only lists tweets from politician, not to them. That makes for one-sided communication.
As a friendly gesture, Socialsquare asked me to create a complementary service, which does the exact opposite thing: Let’s you know what people are tweeting to their elected representatives. This ended up in Folketwinget.dk, a simple site hosted on Google AppEngine.
Now, aggregating tweets to and from the Danish Folketing seems like a faily specific, but it turns out that such an aggregator of can easily be used by any other group — so we’ve open sourced the code over on Github. The code runs directly on AppEngine so it’s easy to host and have running in a few minutes.
For example, if you have a company called 23 with a few employees on Twitter it might be cool to show all the incoming and outgoing tweets from the company.
Get started by installing the Google AppEngine SDK, start up a terminal and browse to the AppEngine folder on your local machine. Then…
1. Get the code
Run this command to download the code. (Obviously, you’ll want to change the folder name from 23tweets to something else.)
git clone git://github.com/steffentchr/twitterengine.git 23tweets
2. Update app.yaml and create config.yaml
Open up the file called app.yaml in the newly created folder and change the application name in the very first line of the file. In this example, we’ll change it from twitterengine to 23tweets
Also, rename the file called config.yaml.sample to config.yaml
3. Create an AppEngine project
Open up GoogleAppEngineLauncher (it’s called that on Mac anyway). Right-click in the window and select “Add existing..”. Now select the folder we’ve just created and click OK.
This is all you need to actually get the application up-and-running. Click the Run icon to start your development server and open up http://localhost:8000 in a browser.
4. Configure the application for your dirty purposes
Before you can import some tweets into your application you will need to update the configuration. Open up config.yaml in a text editor and change what you think is necessary. For my example, I’ll use this configuration:
site_name: 23 tweets
site_tag_line: What is up with 23?
site_domain: 23tweets.appsport.com
twitter_username: 23pull
twitter_password: secretpassword
users:
- steffentchr
- mygdal
- mmmmmariaaaaa
- abemad
- guan
- burtblancher
- 23
hashes:
- 23
- 23hq
- twentythree
Basically: Use the user account of 23pull to get all tweets to and from @steffentchr, @mygdal, etc and all tweets with the hash #23. Then call the site “23 tweets”.
(This could just as easily have been used to create a Folketwinget-sibbling for the Norwegian parliament or for aggregating conversations of everything involved in a specific open source community.)
4b. Import the tweets
Now, go to http://localhost:8080/import to trigger an import of everything specified in the config file. When you return to http://localhost:8080/ you should see all a batch of the latest twitterings.
When you’ve finished step 5., the application will automatically import tweets every five minutes. The frequency can be changed by editing cron.yaml.
5. Deploy your app
Before you can deploy your new twitter aggregator to the wide web, you will need to create the application with Google. Go to the AppEngine dashboard, click the “Create an application” button and fill in the form. Remember to use the correct application identifier, i.e. 23tweets.
When your application has been succesfully created, go back to GoogleAppEngineLauncher and right-click your project. Then select the deploy option and sign in to your Google account. That’s it. It’s online.
Again, you’ll need to visit http://projectname.appspot.com/import to import tweets into the online database and then go to http://projectname.appspot.com/ to see the result.
6. Customizing the design
You’ll find that your site is in Danish and generally doesn’t look the way you want it to. Changing the design though is fairly easy: Modify index.html and resources/style.css for your purposes. If you need anything fancy, the html file is structured using the Django templating system.
Update April 24th 2009: Originally, this post contained information for running external cron jobs to import tweets. Today, the code has been updated to utilize AppEngine’s new native cron feature which makes it all happend automagically..
April 2nd, 2009 at 10:11 am
[...] dag har Steffen Christensen fra 23 opensourcet koden bag folketwinget.dk så alle nemt kan lave en tilsvarende tweet aggregator på googles cloud platform [...]
April 2nd, 2009 at 11:48 am
[...] the code running Folketwinget.dk available under an open source Apache 2.0 license on Github. Check Steffen’s blog for further details on installation and [...]
April 2nd, 2009 at 6:12 pm
Nice. Very nice.
April 2nd, 2009 at 6:29 pm
Fantastic! I just created my first app using this guide – have a look at http://zybtwitter.appspot.com/.
Btw, under 3., you wrote port 8000, not 8080.
April 2nd, 2009 at 7:03 pm
I seem to have a problem getting OpenACS to ping more frequently than 1 hour – how do you set it to ping every 15 minutes (or 1 minute, for that matter)?
April 2nd, 2009 at 7:11 pm
I actually ended up cheating myself using cron every five minutes ;-) didn’t know that the frequency was down to an hour though. You could look fir other similar services, such as Pingdom? Or use something like google reader for the purpose ;-)
Let me know either way…
April 2nd, 2009 at 7:37 pm
Pingdom costs money after a month, I see – and I’m not technically intelligent enough to make a cron job.
Know of any other services that can ping automatically? 10/15 minutes is fine.
April 2nd, 2009 at 8:00 pm
I’m set up up in my own cron job.
The quick guide to setting that up is:
Type in “crontab -e” on a Linux server, then hit the “i” button and type in this in the bottom of the document:
*/5 * * * * /usr/bin/wget -O – -q http://zybtwitter.appspot.com/import
Then: hit esc and type “:x” and hit enter.
Otherwise, http://basicstate.com/ seems to be a good bet.
April 2nd, 2009 at 8:57 pm
Basicstate did the job – thanks again for the tutorial!
April 5th, 2009 at 8:30 am
[...] Refresh: Announcing TwitterEngine: Your own twitter aggregation site on AppEngine (tags: twitter api dansk mashup) [...]
April 24th, 2009 at 3:45 pm
With the latest SDK release, App Engine now provides cron support. You might want to add a cron.yaml to your app! :)
April 24th, 2009 at 6:05 pm
Thanks Nick,
I’ve made the changes and the update is available through github.com/steffentchr/twitterengine
April 27th, 2009 at 7:02 pm
[...] og tænkt på at gøre “noget” ifm. EP-valget, så da Steffen Tiedemann Christensen gjorde maskineriet bag Folketwinget tilgængeligt for alle under navnet Twitterengine, gik det meget stærkt. Domænenavnet twittervalg.dk blev indkøbt, og 2 [...]
May 11th, 2009 at 5:22 am
whenever I type this:
git clone git://github.com/steffentchr/twitterengine.git 23tweets
into the python console, i receive a syntax error. why?
May 11th, 2009 at 6:19 am
@James:
git is a stand-alone application — not a part of Python. So to get the code by following the instructions above, you’ll need to install git on your computer. See http://git-scm.com/.
It you want to take the easier way, Github lets your download the code as a zip file as well: http://github.com/steffentchr/twitterengine/zipball/master
December 23rd, 2009 at 10:43 pm
How can we make this to return only tweets “from” the users specified under config.py and not include the re-tweets “for/to” those users?
December 23rd, 2009 at 11:09 pm
@ian,
You cannot configure this through config.py alone, instead you’ll need to remove a line from main.py. The line reads:
t.append('@'+str(u))
Just delete this line and you should see the only tweets from the specified users; not replies back at them.
December 28th, 2009 at 2:37 am
@Steffen
Thanks! Figure it out after posting :(
The only issue I have right now is that it doesn’t seem to import tweets from accounts whose last tweet is like a week or so old.
I have more that 100 users that I’m monitoring. Could that be part of the reason?
December 28th, 2009 at 9:45 am
Hey Ian,
To save on requests to the twitter API the Twitterengine is bundling imports for multiple users into one request to Twitter. Since the import is done for new messages, you’ll find that active users will crowd out non-active users in the first imports. Over time, this is not an issue since you”ll be importing new stuff every five minutes and there won’t be time to for the crowding out to occur.
A simple fix is to change the “140″ in line 113 of main.py to a lower value. Possibly “20″ or “30″ — and then run the import again manually. This will give you more old tweets. Then change the value back to “140″ for the auto-imports when you have the service up and running. Otherwise you’ll run out of requests to the API.
May 30th, 2010 at 10:53 am
how can i get this on my website instead on on http://www.____.appspot.com?
May 30th, 2010 at 10:57 am
@jim: The guide in the blog post takes you through the steps of deploying the Twittengine code to your own *.appspot.com domain — basically, while following the steps you’ll be able to choose the subdomain (and you can even point you own domain to it if you want to). The key to all of this is The Google AppEngine SDK.
May 30th, 2010 at 11:14 am
no worrys got iti think. have to register your site with google app engine
June 25th, 2010 at 4:33 am
I get this message:
http://pastebin.com/V2ARV5Wz
Python error or configuration error?
I have another question. In your example have change twitterengine to 23tweets from first line of app.yaml, however the first seven lines of this file does not have any code or text. So I do not know to change. I suspect this may be causing the failure. I hope you can help me. Thank you.
June 25th, 2010 at 7:04 pm
@BiNaRiO: I think the two parts are related — at least from the error message you’re getting:
# Fatal error when loading application configuration:
# mapping values are not allowed here
# in “mediosdecomunicacion/app.yaml”, line 33, column 13
Can you paste your app.xml here and I’ll see if I can spot the issue?