A Massachusetts Institute of Technology researcher claims to know what’s going to become a trending topic hours before Twitter does—using an algorithm-driven crystal ball.
Associate professor Devavrat Shah claims his system can predict trends with 95 percent accuracy roughly 1.5 hours, and up to five hours, before they appear on Twitter’s list of trends.
HIs algorithm combines old data with tweets that are being sent in real time to make predictions based on previous instances that line up with what’s currently being tweeted about. Samples of previous data which line up most closely with the current term being analyzed are given more weight in helping the system predict if something will hit the trends list or not.
Shah used 200 topics that trended and 200 that didn’t to compare current topics to, resulting in a 95 percent accuracy rate and a 4 percent false positive rate. Shah predicted that the accuracy of his system will improve as more historical data is collected, so the patterns can be spotted more easily.
He argued that his system can scale up to much larger systems, say the entirety of Twitter’s 500 million daily tweets, should the company take an interest in his methods. (As GigaOM points out, Shah’s research assistant, Stanislav Nikolov, is a Twitter employee, so there’s a relationship between both parties.)
Twitter’s trends are not driven by the number of tweets on a given topic. (Otherwise, it’d be all Bieber, all the time.) Rather, Twitter’s systems can spot when a the number of tweets on a topic increases sharply.
Trends are often predictable. On a Monday night, the trending topics are routinely dominated by wrestling-related terms and Monday Night Football. If a major news event is taking place, such as Hurricane Sandy this week, it’s not too difficult to guess the words people will be tweeting at great volume.
You can’t really account for golden, unplanned soundbytes, though. No one could have guessed that Barack Obama would talk about horses and bayonets or that Mitt Romney would mention binders full of women in a the presidential debate. Both of those terms shot up the trends chart within a few minutes.
Just last weekend, Twitter researchers published their own report on how they’re able to identify breaking news topics and display relevant search results within 10 minutes of something happening. Being able to predict hot topics before they burn up the top trends list would surely help them in that regard.
Photo by ChazWags/Flickr