Twitter isn’t ephemeral—each tweet is etched into the Internet ether as soon as you send it out. But sometimes it feels like tweets disappear, thanks to Twitter’s historically awful search function.
That’s all about to change. On Tuesday, Twitter announced it’s indexing every tweet ever sent since 2006, which means everyone’s tweet history is now searchable on Twitter proper. Previously, the only way to find old tweets was to use services like Apple-owned Topsy, Tweet Machine, or Snapbird.
The search feature was a massively technical undertaking, which began all the way back in 2012. Because of the huge amount of information and data that’s produced on Twitter each day, the company had to figure out how to scale the search infrastructure to aggregate Twitter’s history without using expensive RAM, or memory, and create an interface that thinks about all the separate data chunks (like tweets and engagements) as one endpoint.
According to a profile of Twitter’s search indexing team in Wired, the guys behind the undertaking initially asked themselves if it was even possible—the hard part would be not only scraping the entire history of tweets, but also including the millions of tweets sent each day.
Twitter’s new search feature can comb through half a trillion documents and display tweets and data in less than one second.
Photo via mkhmarketing/Flickr (CC BY 2.0)