Recently, while evaluating storage options as part of a consulting engagement, I came across the Isilion offering from EMC and some of the articles in the associated literature talked about the use of Isilion for cluster computing. Given that the emphasis is still on storage, specifically HDFS, it was intriguing that the possibility of compute functions being delegated to nodes a la Map Reduce was discussed quite a bit. Further reading in to what is considered to be cluster computing got me to the Wikipedia article on Computer Clusters.
So it is quite clear as to what the difference is between cloud computing and cluster computing to the extent that we can even safely say the cluster computing is a subset of cloud computing especially given the offerings from Amazon Web Services such as Elastic Map Reduce and the newly launched Lamda. Hence in this blog article I will focus instead on how a solution needs to be architected to leverage cluster computing effectively to get the best bang for the buck out of cloud computing.
Lets start by addressing the biggest challenge with implementing cluster computing: co-location of data on the compute node. While this is an easy problem to solve while utilizing the Map Reduce paradigm it represents a real challenge to use cluster computing for achieving scalability in the typical usage scenarios. Although the use of technologies such as InfiniBand may be an option in some cases the cost benefit analysis would render it useless for most of the typical business applications.
One immediate option is to utilize microservices based architecture. But it is clear from the description in the seminal article by Martin Fowler that it does not address co-location of transaction data although he does talk about decentralized data management and polygot persistence. Clearly is not really meant to allow for easy adoption in a cluster computing scenario. Interestingly though there is a reference to Enterprise Service Bus as an example of smart end points and that is what got me thinking about extending the concept to cluster computing.
The trick then is to apply the event based programming model to the microservices architecture concept leveraging in turn the smart end points aspect. All the transaction data needs to be embedded in the event combined with any contextual state data. Through the use of interceptors or other adapters the data can be deserialized in to the appropriate service specific representation. This is key since the service need not and actually should not be built to consume the event data structure.
While the approach described above would require you to invest significantly in setting up the requisite infrastructure components to provision compute nodes on the fly to handle events, given the recent release of the AWS Lambda service provides us with an opportunity to apply this concept more easily albeit with some new terminology: microservices are implemented as AWS Lamda functions! It would be very interesting to figure out if argument reduction is supported! Check this blog again in a few weeks to find out…
Adventures in Cognitive Computing 