Scraping Social Media

This is another open-source software tool. Created by Jason Dixon, the application is designed to work “real-time” during live events and would work best for people looking to capture data for a specific period of time, as it will not work to mine previously created posts. It filters the Twitter main feed for specific search terms while running and exports the captured data to an offline file. This software works on Windows, Mac and Linux and requires Java to run.

Downloading the program and running it is a fairly easy process (make sure to be running Java 1.8). The interface is fairly straight forward, where you enter the search term(s) you want to filter for and hit the button. It does require the application to be up and running the entire time your want to collect data, so depending on your capture time this could be problematic.

The downside is that the application requires you to apply for a Twitter API OAuth token, which can take some time to get, and is purposefully an opaque process. This is a not a last minute or “just-in-time” archiving solution (at least the first time).

The meta-data that My Twitter Scraper collects includes, account username, the poster’s location based on their profile, and tweet content including all text (@ and # included), but no images or video. However, it will include URL reference when able.

Completely free open source software. However, the software coder is requesting donations if you enjoy the product. My Twitter Scraper is designed to not take up a lot of RAM so you can run the program in the background of your regular computer use, so you will not need a dedicated machine or additional memory to use it for prolonged periods of time. Additionally, Twitter does not charge for API OAuth token, this is a registration system so that Twitter can keep track of who is mining the feeds, and to ensure that you are only accessing public information.

The export file is the common CSV (comma separated values) file, that will open in Excel and is easily pulled into MySQL or other database management programs.