How to Scale Code Deployment Like Twitter Does

Learning the way smart people develop is always important

 

How to Scale Code Deployment Like Twitter Does

via ReadWriteWeb by Chris Cameron on 7/15/10

There is a fundamental problem that Internet startups face when they begin to grow: scalability. It is imperative to be able to not only handle large flows of data from usage, but also to deploy changes and upgrades to your software across multiple servers as fast as possible. Twitter faces this challenge every day as its farm of servers grows, and today the San Francisco-based company is open sourcing its solution to the deployment problem for all to use.

Named "Murder" after the collective noun used to describe a group of crows (I guess "Gaggle," "Flock" or any other bird plural didn't sound as nice), this part-Python, part-Ruby tool allows lighting fast code deployments across large groups of servers. Twitter engineer Larry Gadea says it took what once was a 40 minute deploy and shortened it just 12 seconds - an impressive feat, no doubt.

So how did Twitter accomplish this?
Murder is built on top of BitTornado, a version of BitTorrent modified to include advanced functionality. Gadea says they have tweaked the app to better function in the unique environment that exists in a datacenter, including "low-latency access to servers, high bandwidth, no NAT/Firewall issues, no ISP traffic shaping, only trusted peers, etc."
By integrating this tool with their existing deployment app, Capistrano, Twitter was able to shorten code deployments by 99.5%. Anyone can checkout Murder on GitHub right now and start thinking of ways to use it with their servers. It is important for young companies - not just those with several servers - to think about scalability before it becomes an issue. Projects like Murder are making that process much easier for first time entrepreneurs and system administrators.
Photo by Flickr user rebeccakoconnor.