I’m preparing a presentation for a large consulting firm on architectural options for a massively scalable, bi-directional real time application.
I’ll post about the technology and the architectural choices in the near future but I’ve been chuckling at most of the content that I’ve been reviewing and though I’d spill a few thoughts about it.
Specifically I’ve been looking at “alternative” choices that might be uniquely ideal for this particular use case.
As a basis for understanding the challenge…. Up to half a billion users (yes, billion with a B) sending an update to the server once every 3 minutes and retrieving a stat structure every one or two minutes.
About six months ago I started mapping this use case scenario to Node.JS and NoSQL (Redit or MongoDB).
There are a few pretty big / high profile on-line properties already using Node.JS in production.
- Walmart (Why Walmart is using Node.js )
- E-bay / PayPal ( Node.js at PayPal)
- Microsoft (Node.js Dev Center | Windows Azure) They have extensive support, in fact their Azure CLI tools are actually written in Node.js.
- LinkedIn (Exclusive: How LinkedIn used Node.js and HTML5 to build a better, faster app)
- Yahoo
- Google (Node at scale: What Google, Mozilla, & Yahoo are doing with Node.js)
- Yammer (now part of Microsoft) (Managing Node.js Dependencies and Deployments at Yammer – Yammer Engineering)
But, mining through articles and on line conversations has reminded me to the religious zeal with which many developers defend their choice of developer stack.
Node.JS is not SINGULARLY unique in it’s non-blocking architecture but it is somewhat unique in that it it’s FUNDAMENTAL PREMISE is non blocking, that it’s syntax is JavaScript (the Google V8 engine) and that it is INNATELY web (not requiring a separate http server.
What’s I’ve been chuckling at is how threatened certain camps of mainstream developers seem so threatened by the emergence of any new technology in their space.
In response to article on Node.JS .NET developers jump in to try to convince you that you can do the same thing in .NET using Async, Threads and customizing the IIS pipeline (just follow these 40 steps.)
Java developers start talking about NIO as though it’s the same thing as Node.JS.
PHP devs point you at ReactPHP
Python developers start talking about Async I/O and threads, etc, etc, etc.
But what’s wrong with just accepting that there is a new technology that’s very good at a specific category of use-case?
Ok, I’ll admit I never loved programming in Java, but I spent a decade at Microsoft digging C#. I also did PHP there and ended up Zend (“The PHP Company”). I spent a year and a half at Mozilla with is mostly a Python shop and I still like all three.
But I don’t need any of those to be the right answer for every software development challenge.
Too often I see dialog from a .NET developer who needs every solution to be only .NET or from a Java developer who needs every answer to be Java.
For the last 13 years I’ve worked for organizations that manufacture a specific set of developer technologies. I frequently donned my “application/solutions architect” hat to help companies answer the question “what is the right approach for solving THIS business problem”. Even when I was working for a company who’s success was based on a technically bias answer to those sorts of questions, sometimes I answered in the alternative. (How fun it was to explain to my Microsoft management that I recommended PHP for a project 🙂 )
It’s my personal opinion that many folks close their minds to this “best mapped use case” approach because technology grows and evolves so rapidly many of us just give up on the idea of staying on the “front edge” of of the software development industry. In some ways we’ve gone from looking for that silver bullet to trying to convince folks that the stuff we are already good at IS THE SILVER BULLET.
All of this I suppose by way of encouraging you to open your mind and spend a little time looking at how emerging technologies might provide better solutions to some of the problems you solve in your day job.
I’ll post the technical blueprint for the above referenced architecture using Node.JS and NoSQL in the near future.
One reason obviously is that investing in multiple technologies (in this case languages) can be very expensive.
That is one of the reasons nodejs has such a following among (web)developers it helps reduce the (mental) load.
Joe, given the problem you are trying to solve. I thought you might be interested in this video http://ayende.com/blog/166564/ravendb-conf-videos-tailored-database-solutions-with-voron Ayende demos a time series database he put together in a matter of about 20 minutes that handles an enormous amount of non-blocking writes per second. I know it won’t likely solve your problem, but its recent, relevant, and just simply cool. If you have some time you should check it out.
On the subject of best tool for the job. That is hard to answer if you don’t understand what all the tools are that are available. It usually ends up being the least common denominator problem. If you already have a team of people that know language x and the associated tools you probably won’t take much time to decide. In certain scenarios like the one you mentioned though it is probably best to do some prototyping and research and hire the crew to build it. Good luck with the gig.
Nice article Mr.Joe Stagner. I am also a Software Engineer from Karachi, Pakistan. I am an admirer of your blog too :). I have worked on .NET for the most but I had to write few (complete) Android native app (using ADT Eclipse and Java) and I personally am also fond of c#.
You see I also had to use my knowledge of .NET to complete the Middleware (JSon asmx) and the backend (MS SQL) and a front end in Android. I think If I know .Net than I will expand my skills only when it gets imminent. As I had invested my time in it. it will surely hurt to work on something else ….In short our brain restrains from relearning 😛
Achieving the same goals that you had for the website that is responsive (in php) in my view was better to be achieved with the asp/ IIS stack when you were a Microsoft Employee. As this will seems some sort of shame for a software giant to use external / 3rd party frameworks to complete some project in presence of it’s own framework that is capable too to deliver the same results. Where is the honor in it? …but for fun ;P.
Well I had learned that you shall only learn and dwell with new frameworks when it is imminent. Stick to your own stack as much you can get the mastery ..it is better than becoming a jack of all trades 😉
And yes there are sometimes exceptions too…I am pointing to Node.js.
Hope you don’t mind my bad English 😉
cheers