The Synergy Research Group’s last report of 2015 attributed 31% of the cloud computing market to Amazon’s Amazon Web Services (AWS), nearly four times as much as its nearest competitor, Microsoft. This would come as no surprise to any programmer, Data Engineer, or Data Scientist, AWS is a mainstay when it comes to working at scale. It is integral to the internet’s infrastructure. In light of this, Amazon’s recent analytics offering is surprisingly banausic. Competitors like Microsoft and IBM released dedicated analytics platforms to support Data Scientists and have open-sourced deep learning frameworks such as CNTK and TensorFlow. Comparatively, Amazon has been resting on its laurels. Yes, there is AWS IOT, AWS EMR, AWS Kinesis, and AWS Machine Learning, but these seem to cater more to engineering rather than Data Science. Outside of pre-built instances like the Data Science Toolbox – Python, Apache Jupyter, and Apache Spark – Data Scientists using AWS tools have to go through the process of setting up every component they need from scratch. This may be about to change with the recent open-sourcing of Amazon’s Deep Scalable Sparse Tensor Network Engine. It’s quite a mouthful and the abbreviation isn’t much better, DSSTNE. Amazon says you can just call it… Destiny.
Like other frameworks of its ilk, Amazon Destiny facilitates the training and deploying of deep learning models on GPUs, and can be run locally, on a Docker container, or on AWS on a GPU instance. Unlike similar frameworks, Amazon Destiny is orientated towards search and recommendations – no surprise given its parent company. The release notes claim that Destiny is the fastest deep learning library available, and that it’s the best at using multiple GPUs on a single server. At the other hand, Destiny is a bit bare-bones. The documentation is sparse, the library only works with data in the NetCDF format, and it has no support for Convolutional or Recurrent Neural Networks. (The addition of CNN’s and RNN’s is apparently next in the pipeline.)
Time and I will tell if this release is a token open-source offering or thoughtful gift to the wider analytics community. You can find the project here.