How to build an incredibly fast internet application

I just received two bits of good news about an internet-application that was built under my management. I have to be a bit mysterious about the project itself as I’ve signed a NDA but it’s an application that will be visited extremely often with a totally unpredictable planning. Luckily I love a challenge.

NOTE: This project has gone live since the writing of the blog . It was for the Dutch Monarchy and the Orange Foundation, the largest charity of the Netherlands, on occasion of the inauguration of the new Dutch King Willem-Alexander.

So the good news was:

  1. Our stress tests showed we are over-performing by 10-100 times.
  2. The company that audited us (KPMG) said: “The application was made with love and we don’t see that very often’. They actually said LOVE and  the L-word is not used lightly by this large consultancy firm (and they hardly had any critical points at all).

So I’m a happy camper and thought I would share our approach with you. I’m focusing on performance issues here even though we paid just as much attention to usability, security, browser compatibility, etc etc.

Let me be fair and tell you that I’m just the person that comes up with concepts, knows a bit about the technology and manages things, so who am I to take credit for such a monster of an application? I worked with my Maximonster team:  Marc Worrell, Arjan Scherpenisse and Joost Faber and together with the brilliant team at my former company ICATT interactive media.

First of all, what is  fast?

To test the application, we’re using a scenario that simulates a user going through the application. It’s a simulation of many different types of browsers in about the same percentages as the real world (x% IE, Y% Firefox etc).  We ran tests of up to 400 people doing a complete task in 0.8 seconds. We were serving 120 Megabit/second.
In our case this compares to about 30.000 concurrent users. Enough to run 150.000 users in an hour. And all this while the application runs as smoothly as when it runs with 10 users.  We ran this on two servers, while we will be using at least three and can quickly  deploy more if needed. Why not test further? Well… our test server just can’t do it right now and for now it’s enough – we underpromised anyway. We estimate that we can double or triple this easily.

So here’s how to do it from a birds eye view.

The obvious things

Build a light application and pay attention to performance issues all the time. Every query, every piece of script is going to count. Don’t use images if you don’t need to, don’t add design to things that don’t need it. If you are working for a client, keep stressing the performance issue even if you get on there nerves. They never will be grateful: if there are no problems, nobody will thank you. If there are problems, you will be remembered so you might as well point out the risks ahead of time.

Architecture

In the past year we have built “Elastic Zontonic”, an architecture developed in Erlang that makes it possible to add nodes (virtual servers or hardware) on the fly. Zotonic is a framework built in Erlang which makes it possible for us to build applications and offers us a backend which gives a lot of controls out of the box.

Each Zotonic nodes does the same as the next one and can fail. We don’t even care if it does. If nodes go down, the others will consolidate missing data. If the node comes back up, it’s great but if not, the others will be just as happy.

Have a stress tester with some good software

In our case we used Visual Studio running off a virtual server running at Rackspace. Rackspace sells servers time by the hour, so you can do this quite easily. Stress testing itself is complicated and hard to understand. You’ll need time from the developers to help build a stress-script and you’ll need to do it several times. Some user scenarios really can’t be simulated and you’ll have to make some guesses. Though we haven’t used it yet there are some interesting cloud based testing products being launched. If you haven’t invested in time and money into offline web-stress-software, you should definitely go that way.

Work with extremely flexible, cheap hosting

Now I’ll start this by saying: I don’t get commission from Rackspace. I compared several hosters at the start of this project and found their pricing to fit the project (our client kept part of the planning a secrete so charging by the hour makes things easier). Rackspace isn’t all perfect (some of the DNS software had some problems) but their service was great, the pricing easy to understand and they won my heart when I called them about stress testing. I had been warned that if we do extreme testing at a virtual server provider, you could get kicked out. So I called Rackspace to check this and they told me; “Oh… do what you like. We probably won’t even notice it.”  What a bold thing to say.

We actually developed a module to automatically configure servers with the Rackspace DNS API, so we might just come back.

Content delivery network

Now I’m old enough to remember the days that bandwidth was a problem for websites. Those days are over for most sites but not for high performance situations like this one. So that’s why Amazon and Rackspace offer storage for files, be it images, video’s or just documents. It takes some extra work to use a cdn (it’s a bit more complicated than just a remote ftp site) but once you’ve got it running, it takes all the stress out of storage.

Don’t have a server too far away (Ping!)

One reason you might think that hosting at a hoster like Rackspace is a bad idea is that they are far away and the responstime (measured by the ping time) is much lower if the server is far away. Luckily we found out that Rackspace has a datacenter in the UK, making it much faster for us (in Amsterdam) that it would be in the USA.  Most of the visitors will be coming from the Netherlands so that’s important. The difference can be quite dramatic. Before (!) we started, we checked a few sites hosted at the site in the UK and found a ping of 7-8 ms which is fine compared to a server in the US.

Unlimited loadbalancing

Loadbalancing is a tricky subject. You have to configure things and it’s quite common that things go wrong if a loadbalancer looses track of the server. One thing we got from Rackspace is for all practical purposes and unlimited loadbalancer. It’s not free and all data is charged for but it’s cheap enough and it’s one less thing to worry about. What we did to make things a lot simpler is to have a sessionless application. This certainly not always possible but if you can, there are lots of reasons for it.

Underpromise

From a less technical perspective: really underpromise. What ever an individual developer will think, combining technology in general makes things go slower. And if you’re talking hundreds of thousands or millions of users, things will be really massively slower. So – be smart and under promise, add extra hosting budget to your plan, add redundancy and expect the worse.  We actually expected our application to run on one third of the hardware that we are using. And when one of the nodes did actually crash during our first major stress test, we were happy to see everything kept running smoothly. Your client will be totally impressed when things are even faster than they expected.

Michiel Klønhammer

Thanks to Marc Worrell and Arjan Scherpenisse!

So looking for an Erlang dream team? Check out www.maximonster.com and get in touch.

Advertisements

Zotonic the Movie – the making of

We had a lot of fun making “Zotonic – the movie”.  Here’s the “Making of ”

I’m working on the team as… um… what is my role? I guess in the Open Source world it might be “Community Manager” or manager of business affairs. In a ‘regular’ business it would be General Manager or  Marketing Manager. For the time being I’m eh… Michiel.

Want to know more about Zotonic – the Erlang Web Framework and Content Management system?
Check it out at http://www.zotonic.com

Zotonic: The Movie

This is a video I produced with Peet Sneeks to promote Zotonic, the Erlang web framework & content management system.

It features Charlotte, a successful “twitter girl” who wonders if she needs Zotonic. Charlotte is actually for real!  Marc Worrell and Arjan Scherpenisse explain what Zotonic is all about. The Video is available on Vimeo and Youtube or on the site.

I’m really excited about Zotonic: it’s an open source framework being used for many mission critical applications. I got involved in the project due to my sponsorship during the MaxClass project. The movie is our most recentl project, but a long list of activites are going to follow (like an article the next edition of “The architecture of Open Source Applications“, called the Performance of Open Source Applications) and an upcoming interview

Want to keep posted? follow @Zotonic or test. We’re developing professional services too soon.

For Erlang Geeks: in the course of the video we make a joke that has kept us laughing all through the production. You might notice that there is an old fashion telephone that resembles the phones from “Erlang: the movie” .  So when Arjan hears the phone ringing and picks up with “Hello Mike”, this isn’t me, but the Mike (Mike Williams, the co-inventorof Erlang). from the original movie. :-).

Totally Erlang – a site in one day

Now I don’t often believe start-up stories when people say the ‘site was nearly free’ or  “all it was, was a brilliant idea”. No, my experience is that everything – and I mean everything – is hard work. Fun, but hard work. The stories about cheap projects usually don’t cover the work that people put into it them selves.

In one case, I was involved in a small success that was actually born in one day: Totally Erlang.

In 2010, I started Maximonster Interactive Things with Marc Worrel and we set out to create new and exciting applications using Erlang (MaxClass, Zotonic and more). We needed an Erlang developer and I was researching how to find them. Recruiters I had worked with at my previous company ICATT, couldn’t really help and I didn’t want to waste money on the mainstream job sites.

So one morning, Marc and I decided we should make our own Erlang job site. We were really busy, so we gave ourselves exactly one day. I came up with a name, a logo, registered the domain, came up with some ideas and Marc set the site up with Zotonic – the Erlang web framework and content management system. We actually did do some spellchecking and bug fixing the next day, but it was minor. I am still impressed that Marc could build something in one day that has had virtually no downtime in two years.

The site did what it had to: Atilla Erdodi, a talented Hungarian developer who had done an internship at Erlang Solutions, joined the company and has since then joined the Zotonic core group. But we wanted the site to keep going. So we tweeted about it and told a few friends and the “rest is history”.

Totally Erlang

The site has now hit the nr.1 position on Google for Erlang jobs and is really starting to grow and we have decided to spend more time on it  – to start off with, we  finally have a Twitter account (Follow us on www.twitter.com/totallyerlang and a new logo!

We have moved on to organize a Erlang Talent Community  for a more personal network of developers but Totally Erlang will continue to grow and we hope to be adding more services to support the Erlang community.

 

 

 

 

 

Support us by following us on Twitter

Zotonic!

I proudly joined the Zotonic core team.

Zotonic is an open source framework for developing web and mobiles systems. My start-up Maximonster built MaxClass and other projects with it. Besides funding via the MaxClass project, I will be developing the business perspective and have started contributing by organizing a survey. If you are an open source thinker, developer or just plain smart. Feel free: http://zotonic.com/survey-future

zotonic

A new era in (scalable) technology (a blog for the ‘rest of us’ about Erlang and other things)

When I started my new company Maximonster Interactive Things, I was as excited as anyone about the cloud and nearly called my company the Silver lining (as in “every cloud has a silver lining’’). We were setting out to build some new technology (starting off with MaxClass) and everything would be in the cloud and for the cloud. Now two years on, we have achieved just that: we’re very distributed (I just got out of a video conference with developers in the US, Estonia and the Netherlands) and we’re using a totally new category of tooling. It really feels like a new era! Developers are giving and getting open source solutions freely across the borders of corporations, start-ups are building amazing applications without getting tangled in corporate licenses and development is funner than ever. May I call this a paradigm shift?

Michiel Klønhammer – Maximonster interactive Things

Thanks for comments (all mistakes remain mine :-): Arjan Scherpeniss, Marc Worrell, George Serbanut

TOOLS DESCRIBED FOR THE REST OF US

I’m not a developer (those would be Marc Worrell and Arjan Scherpenisse and others in in the Erlang Talent Community – thank you everyone) and have noticed how little easy-to-read information there is out there for ‘the rest of us’. So I thought I would describe some of the tools we use, using slightly-less-tech-jargon.  Key is scalability: affordable, well managed growth with little redesign of web applications.

Erlang

Erlang

Erlang is a programming language that is extremely good for building sites and systems that need to be extremely scalable, extremely stable, are distributed across many servers (so it doesn’t matter if a few break down) and can do many things at the same time. Erlang is used for a growing number of websites, many chat-like systems like Facebook chat, IMVU and Jabber and for building tools that need to be fast too (see some of the tools below). Erlang is (depending on who you ask) both a reference to the Danish mathematician Agner Erlang and an abriviation of Ericsson Language as it was originally developed at Ericsson. Thank you Ericsson!

Note: You might have heard of Erlang/OTP. OTP is simply a set of development libraries for Erlang.

More info: www.erlang.org

Riak

RiakRiak  (written in Erlang) is a datastore which is particularly useful for storing massive amounts of data distributed across many servers without any one of the servers being the ‘master’ (basically all servers are equal making it very scalable) . Riak is open source but is mainly developed by Basho, a company which sells support and offers products like Riak search and Riak CS.  Riak and other NoSQL solutions like MongoDB and CouchDB have disadvantages too: no standard consistency like SQL databases like Postgres, MySQL and MS SQL can have, no standard indexing to make search easy and limited support for ad-hoc queries. So for typical business solutions it isn’t always the best solution but for situations where data is ‘big’ and less structured (no tables), NoSQL solutions can be useful.

Riak was partly based on research done at Amazon. Users include: GitHub, Ask, Mozilla and Conduit

More info: www.basho.com

Zotonic

ZotonicA framework for building Erlang based web applications. Zotonic helps developers build super-fast websites or mobile applications by providing standard functionalities, an integrated webserver and a way of building the front-end of a site, which typical Erlang developer find difficult. Zotonic also provides a standard back-end for things like managing content and adding functions to a site. Though Zotonic has been dubbed a content management system, I prefer calling it a framework as it’s suitable for a wide range of (very fast) applications. Examples are: MaxClass.com,  Channel.me, Women on Waves. Alternatives are: Chicago Boss and Nitrogen (which Zotonic was originally based on).

Warning: I am involved with the Zotonic open source team as a non-developer contributor so don’t believe anything I say – have a look yourself.

More info: www.zotonic.com

RabbitMQ

RabbitmqRabbitMQ is an application (written in Erlang) used for passing messages between parts of large systems. MQ stands for Message Queues. You might think of it as an API with an advanced way of dealing with queues. For example: big banking systems need to pass information from one part of their system to the next and do this very very quickly. They would use RabbitMQ or other message queuing systems like the hosted Amazon SQS. RabbitMQ is an built on top of the AMQP protocol, a standard for messaging.
RabbitMQ is part of SpringSource, recently acquired by VMWare.

More info: www.rabbitmq.com

Elasticsearch

Scalable applications mean scalable search. Elasticsearch is an application built on top of the existing Apache Lucene search. Elasticsearch makes it possible to build one or many indexes (searchable lists) based on content in a data store and then distribute the index across servers. Again: for stability and speed.

Another open source search solution is IndexTank, open sourced by LinkedIn. There is a wave of start-ups stepping up to provide hosted services based on both Elasticsearch and IndexTank.
Users include: Mozilla, StumbleUpon, Klout.

More info: www.elasticsearch.com

MochiWeb

Scalable applications need a ‘webserver’ (the actual software serving the content) that is easy to use across lots of webservers and doesn’t  slow anything down. MochiWeb is a set of tools that form the webserver originally written (in Erlang) by developers from MochiMedia, a games company. Zotonic (above) has MochiWeb integrated into it. Other popular erlang-based webservers are Cowboy and Yaws.

More info: https://github.com/mochi/mochiweb

GitHub

The list of open source tools above could go on for quite a bit, and for an average scalable application it does. Nearly all open source ‘projects’ or tools (certainly the Erlang related ones) are hosted on Github, a website which hosts code, making it possible to work together on code, add comments and manage versions. “Git” is one of the popular systems for storing- and managing versions of code. Alternatives are Google Code and Bitbucket, but Github is by far the most popular one. Github is a company with a huge following. An active Github profile has become essential for the CV of open source developers.

More info: www.github.com

Rackspace

RackspaceNot long ago I was struggling with complicated contracts for hosting web applications. It could be scary at times as tiny errors in judgment could turn out to be fatal (or at least shockingly expensive when a site got busy). Now developers (certainly at ambitious start-ups)are turning to services by Amazon or – in our case – Rackspace, two category-killer-companies that are offering (low) pricing by the hour and unlimited amounts of servers that are ready to use within minutes. A new contender in this area is Joyent, focusing on hosting Riak, MongoDB (another noSQL datastore) and other new technologies.
More info: www.rackspace.com

Interested in the new era in technology too?  Need a speaker? Need a development team? Need a job? Please check us out at www.maximonster.com

E-learning & Flash: the end is nigh and what now? (8 non-flash solutions)

Since the web replaced cd-roms, there hasn’t been a bigger shock to the e-learning market than the killing of Flash by Apple. Ever since Apple so successfully introduced the iPad and announced that Flash would not be supported, the e-learning world has been in all states. And now that Microsoft will be following suit (no support for Flash or even Silverlight on tablets), the end is really nigh.

Let’s face it, Flash was the first technology that was suitable for (reliable) multimedia content on the internet. When it was first introduced, it was one of many plugin based technologies and there was some resistance as plugins were frowned upon. But Macromedia and later Adobe went on to develop an incredibly stable and useful technology and just about everybody happily installed the plugin. Many – actually nearly all – e-learning companies have been using tools which are based on Flash. Especially the rise of ‘rapid e-learning’ (relatively simple e-learning roughly based on PowerPoint presentations) which often used voice overs, video and relatively simple interactions and testing.

I was recently asked to offer my opinion on what strategy would be wise when developing a new e-learning course in the near future (in this case for a healthcare project). The big dilemma (there are lots) was of course: does one use flash for another few years and ignore iPad or find a tool that outputs HTML5 (or maybe even basic HTML) .

I didn’t realize what I was getting into when I doing some research. My wish list: an efficient production approach that can scale (at least a promise of workflow) and a tool that could offer rich media (video, voice over, learning interactions etc.) and all this based on HTML5. And – I thought this would be obvious: something that really works…

There is an incredible amount of software developers struggling with the HTML5 issue. Actually, it really isn’t possible to publish HTML5 without problems at all (right now) and I don’t think it will  become as easy to publish HTML5 as it was Flash for the next couple of years. As we speak, there is,for example, not one single HTML5 video format that will work in all browsers.

Anyway… I thought I would share some things I discovered. If you think I missed an important tool that produces HTML5 based e-learning, I’ll add it.
There’s one thing you need to know: not one of these tools can guarantee anything. HTML5 is just not finished and it won’t be easy to build a tool that makes e-learning production for all platforms as easy as it was during the Flash-era.

Articulate: one of the main players in the rapid e-learning market. A pretty good product which works with a Powerpoint plugin so that you can use Powerpoint to design about 90% of the product. It has been rumored Articulate is working on a HTML5 tool but nothing has been announced. www.articulate.com

Moodle: the famous open source LCMS isn’t really good at rich media but as it’s all HTML you can use it in combination with e.g. youtube video (which is HTML5). www.moodle.com

EXE: an open source editor which produces scorm complaint HTML-only courses (demonstrated on the Moodle demo site). Not supported anymore and outdated. There have been people willing to take over the project but I haven’t been able to find any progress being made. exelearning.org

Lectora:  they have announced publishing-templates for iPad but it’s a steep learning curve and you have to know what works and what doesn’t as Lectory does lots of things that won’t work on iPads. www.lectora.com

Captivate: Adobe announced HTML5 support (by export) for Captivate, one of the leading tools for rapid learning and tutorial development. I’m not the only one that has tested it and I’m not the only one that found that  it just doesn’t work. Rumor has it that Adobe is starting from scratch and will be offering something new. www.adobe.com/products/captivate

Udemy: an interesting startup offering basic course development tools (mostly video and text) and publishing tools (you can earn money!). As they have avoided really interactive learning, it will work on an iPad. www.udemy.com

Luminosity: this start-up offers an interesting tool for developing HTML5 e-learning. The interactions are all a bit simple but it’s new so worth looking at. Includes some workflow. www.cm-luminosity.com

Rapid Intake MLearning:  the only tool I have found that offers working examples for IOS. A bit rough and relatively expensive but worth taking another look at.  Rapid Intake also offers a LMS and they also offer cloud collaboration tools. rapidintake.com/mlearning-studio