All about the API
Chewing the Eucalyptus with Mårten Mickos
Marten Mickos is the CEO of Eucalyptus Systems. A veteran of open source, infrastructure software and global businesses, he was previously CEO of MySQL AB where he grew the company from garage start-up to the second largest open source company in the world. In this article originally published in the August issue of JAX Magazine, we speak to him about AWS API, embracing Netflix OSS tools, and of course, the great OpenStack compatibility debate.
JAX: Could you outline the background to Eucalyptus? Why it was set up and the problems set out to tackle from the start?
Mickos: Eucalyptus started out of curiosity about cloud computing. The central question was "Would it be possible to create an on-premise system that would behave like AWS?" The group then spun out of UCSB and became a commercial company bringing the power of cloud inside the firewall, and building the best on-prem complement to the leading public clouds.
There seems to be a Java and C mix in Eucalyptus. What were the reasons for choosing those languages?
Java was considered the best language for building an operational and mission-critical distributed system. C was used in some hardware integration modules.
The relationship to AWS is well publicised and obviously very important to Eucalyptus. How does this work exactly?
Eucalyptus is the only cloud platform vendor that AWS has this sort of a partnership with. The partnership is twofold. On the one hand, the engineers and architects of AWS help us as we design new features and AWS services for Eucalyptus. On the other hand, when we identify customers who need a hybrid cloud solution, we approach them together. The goal is to enable effortless application workload mobility between the public cloud and on-premise infrastructure.
How important is the formal agreement with AWS to move in and out of AWS clouds to your strategy? What benefits does each side get out of it?
We believe that the future is hybrid – that most customers will have a need to run application workloads both on the public cloud and inside their firewall, and that they will be moving workloads back and forth.
For that reason we are making sure that Eucalyptus is API compatible with the leading public clouds. So far, AWS is the only public IaaS vendor with a large market share. As time passes, and other vendors build significant market share, we expect to support other APIs as well.
Is there not an over-reliance on Amazon - putting all your eggs in one basket so to speak?
We just follow market shares. Today, AWS is probably 10 times larger than its next closest competitor. As those market share proportions change, we will support the API of the next player as well. Jokingly we have stated that we have Coke covered, and we are looking for Pepsi. It's just not clear who will be Pepsi. Perhaps it's Google, perhaps Microsoft.
You are the first cloud company, to my knowledge, that support Netflix OSS tools. Which are they and what enticed you to those tools?
It all came out of developer enthusiasm to make it work. We realized it would be a very concrete demonstration of our AWS API compatibility. Netflix is the biggest and most advanced customer of AWS. They built their tools for managing their AWS deployments.
It's actually amazing that they could point the tools to Eucalyptus, and it just worked. We started by supporting ChaosMonkey, Asgard and Edda. By now I think we have added a few more. The Netflix tools have raised interest among other companies in the industry, and the fact that Eucalyptus is the only private cloud that they run on obviously brings new customers to us.
What do you make of Netflix's open source pledge in general?
It's excellent! It's good for them as it builds their reputation as a leading cloud expert. It's good for the world as it brings pioneering technology out in the hands of everyone. Yahoo open sourced Hadoop, Facebook open sourced Cassandra, and Netflix open sourced ChaosMonkey and other cloud management tools. It's a common behaviour in open source ecosystems that end-users publish tools and technologies they built for internal use.
On the recent OpenStack debate over AWS-compatibility – what are your thoughts on that?
That's a long and important topic! Randy Bias did the industry a huge favour by bringing up this topic. It's vitally important. Unfortunately, but not unexpectedly, he faced stiff dismissal from his fellow OpenStackers. OpenStack was founded in opposition of AWS, and they will probably never take API compatibility seriously. Instead, they are trying to establish their own standard. But for workloads to be able to move between clouds, APIs must be compatible. For hybrid clouds to happen, APIs must be compatible. OpenStack is now heading for hybrid cloud only within the narrow set of OpenStack clouds. And a number of them are not even internally and mutually compatible.
Do you agree with Randy Bias  or Robert Scoble ?
I agree with Randy Bias. Robert Scoble had some good points, especially the need to stay focused no matter how vast your resources, and the unquestionable need for more quantum leap innovation to make our new cloud world a reality. But I disagree with his statement that following the AWS API would consume the innovative power of OpenStack.
Quite on the contrary – I believe settling for a de facto leader in cloud APIs would free up Openstackers to innovate elsewhere in the technology stack.
A historical example of this is MySQL. The product was initially made to exactly mimic, in the interface, the then popular mSQL database engine. Thereafter MySQL set its eyes on the formal SQL standard. The discipline of not trying to re-invent the database interfacing layer allowed MySQL to innovate elsewhere: the storage engine architecture, the replication technology, and in general the scale-out design. Today, 18 years old and passed from Sun to Oracle, MySQL continues to power Google and Facebook, just to mention two examples. There is lasting power in the scale-out innovations.
Of course every rule has an exception – MySQL introduced the non-standard LIMIT clause in violation of the SQL standard – because web apps needed it. It was an instant success. But other than that, the adherence to the prevailing common practice and de facto standard was a way to free up resources to do other cool stuff.
OpenStack in its current constitution is undertaking a whole set of enormous tasks: define and create a cloud API standard, define and build a cloud platform, compete against the leading public cloud, compete against the leading private clouds.
At the same time, the various components of OpenStack do not necessarily stand out as supreme examples of innovation in comparison to other similar technologies. Swift is not the obvious winner in comparison to Ceph and RiakCS. Quantum hasn’t yet outperformed what VMware has. And so on.
Do you think this issue should have been discussed more in depth at the start?
The start of OpenStack? I think it was. But OpenStack was started by Rackspace as a competitive attack on AWS. For that reason it was probably inconceivable for them to adhere to the API of a company they wanted to compete against. As a side comment, the API issue was of course discussed at the start of Eucalyptus. It even became a core pillar of the strategy of Eucalyptus. This shows how different OpenStack and Eucalyptus are, philosophically.
Is there any merit in ignoring AWS entirely in this industry?
Some people argue that the API is not so important. Some people argue that customers will start to abandon AWS. Some people argue that hybrid cloud is not so important – that on-premise environments will be established with little regard for what's happening in the public cloud.
We happen to disagree. We believe that the AWS cloud APIs are far more important than AWS themselves. The APIs have a role not dissimilar to what x86 meant in the 80s, web in the 90s, and the LAMP stack in the 00s. AWS has become the dominant design – the place where the newest and most innovative developers congregate and share their work products. That's why AWS must not be ignored. It's the bearer of the cloud disruption.
Eucalyptus did this 5 years ago – what would the company look like today without that decision. How has the project changed over time?
Eucalyptus has changed a ton, but the core principles remain the same. We are a global distributed team that builds a mission-critical distributed system. We adhere to the dominant designs of the public cloud as we bring that innovative and disruptive power to on-premise environments. It's difficult for me to picture Eucalyptus without its vision to be API compatible with the leading public clouds.
Do you see this AWS issue running on and on?
Yep. For comparison, OS/2 kept trying to compete with Windows for a long long time, but people voted with their apps.
What is the future for OpenStack and also for Eucalyptus?
I believe that OpenStack will ultimately be successful, just not in the way the project is publicly claiming.
OpenStack is likely to become a Red Hat affair. Red Hat is betting their future on OpenStack, and OpenStack's future lies with Red Hat. As a commercial product, it will be sold as a stack with RHEL and OpenShift to CIO-led IT organizations with large and slow-moving budgetary cycles, in competition against VMware and Microsoft. The other OpenStack product vendors will be marginalized.
The public clouds built on OpenStack will not be able to catch up with the other vendors (AWS, Google, Microsoft, VMware).
In the OpenStack project, hardware certification will be an important part. Other vendors, such as Eucalyptus, will reach into the project and make use of various projects and components as they mature. For instance, Eucalyptus will allow you to use third-party object store technologies such as RiakCS, Ceph and Swift.
The future for Eucalyptus is quite different. We are not trying to sell a massive enterprise stack. We are offering an innovative and disruptive cloud platform that effortlessly links with the leading public clouds, thereby enabling the most powerful hybrid solutions. The role of Eucalyptus is to empower innovators – to empower the people and organizations who are building and deploying modern cloud applications and services.
Is the future public, private or hybrid?
Hybrid is a combo of public and private. We believe the world will go hybrid. Innovation used to be an enterprise affair. But today innovation happens in the consumer space and in the public cloud space. From there it expands into enterprise and on-premise environments. Building those on-premise environments without compatibility to the public cloud would be unwise.
As to how much of computing will be on public clouds and how much on private is something we don't know. At Eucalyptus we are assuming that the majority of all compute cycles will run on public clouds. But the minority that happens on-prem will be enormous in its own right. That's where we are focused.