When I started my new company Maximonster Interactive Things, I was as excited as anyone about the cloud and nearly called my company the Silver lining (as in “every cloud has a silver lining’’). We were setting out to build some new technology (starting off with MaxClass) and everything would be in the cloud and for the cloud. Now two years on, we have achieved just that: we’re very distributed (I just got out of a video conference with developers in the US, Estonia and the Netherlands) and we’re using a totally new category of tooling. It really feels like a new era! Developers are giving and getting open source solutions freely across the borders of corporations, start-ups are building amazing applications without getting tangled in corporate licenses and development is funner than ever. May I call this a paradigm shift?
Michiel Klønhammer – Maximonster interactive Things
Thanks for comments (all mistakes remain mine :-): Arjan Scherpeniss, Marc Worrell, George Serbanut
TOOLS DESCRIBED FOR THE REST OF US
I’m not a developer (those would be Marc Worrell and Arjan Scherpenisse and others in in the Erlang Talent Community – thank you everyone) and have noticed how little easy-to-read information there is out there for ‘the rest of us’. So I thought I would describe some of the tools we use, using slightly-less-tech-jargon. Key is scalability: affordable, well managed growth with little redesign of web applications.
Erlang
Erlang is a programming language that is extremely good for building sites and systems that need to be extremely scalable, extremely stable, are distributed across many servers (so it doesn’t matter if a few break down) and can do many things at the same time. Erlang is used for a growing number of websites, many chat-like systems like Facebook chat, IMVU and Jabber and for building tools that need to be fast too (see some of the tools below). Erlang is (depending on who you ask) both a reference to the Danish mathematician Agner Erlang and an abriviation of Ericsson Language as it was originally developed at Ericsson. Thank you Ericsson!
Note: You might have heard of Erlang/OTP. OTP is simply a set of development libraries for Erlang.
More info: www.erlang.org
Riak
Riak (written in Erlang) is a datastore which is particularly useful for storing massive amounts of data distributed across many servers without any one of the servers being the ‘master’ (basically all servers are equal making it very scalable) . Riak is open source but is mainly developed by Basho, a company which sells support and offers products like Riak search and Riak CS. Riak and other NoSQL solutions like MongoDB and CouchDB have disadvantages too: no standard consistency like SQL databases like Postgres, MySQL and MS SQL can have, no standard indexing to make search easy and limited support for ad-hoc queries. So for typical business solutions it isn’t always the best solution but for situations where data is ‘big’ and less structured (no tables), NoSQL solutions can be useful.
Riak was partly based on research done at Amazon. Users include: GitHub, Ask, Mozilla and Conduit
More info: www.basho.com
Zotonic
A framework for building Erlang based web applications. Zotonic helps developers build super-fast websites or mobile applications by providing standard functionalities, an integrated webserver and a way of building the front-end of a site, which typical Erlang developer find difficult. Zotonic also provides a standard back-end for things like managing content and adding functions to a site. Though Zotonic has been dubbed a content management system, I prefer calling it a framework as it’s suitable for a wide range of (very fast) applications. Examples are: MaxClass.com, Channel.me, Women on Waves. Alternatives are: Chicago Boss and Nitrogen (which Zotonic was originally based on).
Warning: I am involved with the Zotonic open source team as a non-developer contributor so don’t believe anything I say – have a look yourself.
More info: www.zotonic.com
RabbitMQ
RabbitMQ is an application (written in Erlang) used for passing messages between parts of large systems. MQ stands for Message Queues. You might think of it as an API with an advanced way of dealing with queues. For example: big banking systems need to pass information from one part of their system to the next and do this very very quickly. They would use RabbitMQ or other message queuing systems like the hosted Amazon SQS. RabbitMQ is an built on top of the AMQP protocol, a standard for messaging.
RabbitMQ is part of SpringSource, recently acquired by VMWare.
More info: www.rabbitmq.com
Elasticsearch
Scalable applications mean scalable search. Elasticsearch is an application built on top of the existing Apache Lucene search. Elasticsearch makes it possible to build one or many indexes (searchable lists) based on content in a data store and then distribute the index across servers. Again: for stability and speed.
Another open source search solution is IndexTank, open sourced by LinkedIn. There is a wave of start-ups stepping up to provide hosted services based on both Elasticsearch and IndexTank.
Users include: Mozilla, StumbleUpon, Klout.
More info: www.elasticsearch.com
MochiWeb
Scalable applications need a ‘webserver’ (the actual software serving the content) that is easy to use across lots of webservers and doesn’t slow anything down. MochiWeb is a set of tools that form the webserver originally written (in Erlang) by developers from MochiMedia, a games company. Zotonic (above) has MochiWeb integrated into it. Other popular erlang-based webservers are Cowboy and Yaws.
More info: https://github.com/mochi/mochiweb
GitHub
The list of open source tools above could go on for quite a bit, and for an average scalable application it does. Nearly all open source ‘projects’ or tools (certainly the Erlang related ones) are hosted on Github, a website which hosts code, making it possible to work together on code, add comments and manage versions. “Git” is one of the popular systems for storing- and managing versions of code. Alternatives are Google Code and Bitbucket, but Github is by far the most popular one. Github is a company with a huge following. An active Github profile has become essential for the CV of open source developers.
More info: www.github.com
Rackspace
Not long ago I was struggling with complicated contracts for hosting web applications. It could be scary at times as tiny errors in judgment could turn out to be fatal (or at least shockingly expensive when a site got busy). Now developers (certainly at ambitious start-ups)are turning to services by Amazon or – in our case – Rackspace, two category-killer-companies that are offering (low) pricing by the hour and unlimited amounts of servers that are ready to use within minutes. A new contender in this area is Joyent, focusing on hosting Riak, MongoDB (another noSQL datastore) and other new technologies.
More info: www.rackspace.com
Michiel Klønhammer 