Cloud Foundry Tutorial : Taking a PaaS on the hard stuff
Unless youve been living under a rock for the last five years, youve probably heard a lot of buzz surrounding clouds, and unless somebodys given you a clear, specific introduction, youre no doubt leery of the term. Andy Piper and Josh Long dig deeper into Cloud Foundry
Unless you’ve been living under a rock for the last five years, you’ve probably heard a lot of buzz surrounding “clouds,” and – unless somebody’s given you a clear, specific introduction, you’re no doubt leery of the term. Broadly, “clouds” are being used to describe applications and services that are delivered over the internet through a self-service consumption model: need e-mail but don’t want to learn how to set up Microsoft Exchange or Sendmail? Just sign up for an e-mail account with GMail. Need presentation software or backup software? Sign up for SlideRocket or Mozy. And, if you need a place to run your next application, but don’t want to invest in setting up a datacenter and purchasing or requisitioning hardware, then you can use a compute cloud. This last segment is what we’re going to look at in this article.
IaaS, or Infrastructure-as-a-Service, describes the ability to dynamically set up operating systems (and operating-system resources like RAM, hard disk space, etc.). Common examples of IaaS solutions include Amazon Web Services, and VMware’s vCloud Director. These solutions let you treat a large array of computing resources – hard disks, memory – as a pool from which individual instances may be drawn and used. If your problem is, “I want to set up a new Linux (or Windows or Solaris) machine quickly,” then IaaS is probably what you’re looking for. For a long time, this was what most people thought they wanted.
After all, once you can set up new servers with the literal push of a button, then how “hard could it be to get a working application up and running?” we collectively balked. The trouble is that for most application of any significance, it can be quite daunting. After all, suppose your application is /.’d (or “Oprah”’d or Digg’d), will standing-up Linux machines be enough to meet the demand? Will you be able to get from a Linux machine to a load-absorbing, redundant application instance in time to make a difference?
If you’re on-ramping new developers, will those developers be able to deploy your code base and iterate with it quickly? Or, will they spend the day (week?!) figuring out the unique permutations of drivers, operating system revisions, directory structure combinations and networking settings to get the application even running? Platform-as-a-Service is the answer to these questions.
A PaaS provides push-button deployment of everything below your code: the computing infrastructure (hardware, memory), middleware services (like databases and message queues) and runtimes (like Ruby, Java, or .NET). The ideal is that you can kiss six-month application server requisition tickets and database administration woes goodbye, and focus on your code. Code becomes the currency.
There is one other dimension to this discussion: choice. It’s important to point out what “cloud” does not imply. “Cloud” does not imply publicly-hosted, and it does not imply lock-in. A good PaaS technology will run both on-premise or in somebody else’s datacenter. A good PaaS will be open source, and applications written against it portable and easily moved from one environment to another as the requirements dictate.
Enter Cloud Foundry
CloudFoundry.com is a publicly hosted cloud environment (sign up for an account at www.cloudfoundry.com that is free while the hosted account is in beta), as well as a solution that you can install entirely behind your firewall. You can use BOSH, the Cloud Foundry orchestration engine, to install Cloud Foundry entirely behind your firewall on vSphere, or on Amazon Web Services. Naturally, because it’s open source, there are still other ways to install it in your environment, including Chef recipes and an Ubuntu package.
Cloud Foundry is licensed under the Apache 2 license and has enjoyed very successful ecosystem growth. People have taken the code, forked it, and created their own, more specialized versions of Cloud Foundry. There are, let’s say, more hobby-ist centric ports of Cloud Foundry – supporting Haskell and Erlang – as well as more robust, productized implementations. ActiveState provides a private PaaS called Stackato that supports Python and Perl, in addition to the supported runtimes and languages included in the base release. ActiveState, as many will know, is the leading maker of IDEs and tools for languages like Python, Perl, and TcL. AppFog (formerly PHPFog) provides a PaaS based on Cloud Foundry that supports PHP.
This is all important because it gives you freedom: You can run the application on a publicly hosted environment if you like, to simplify getting started, and – if scale, QoS demands, supported services, or regulatory restrictions require – feel no hesitation in moving the cloud (and your application) in-house, confident that the same command line tools, APIs, and services are supported across environments.
Deploying an application
There are a few ways to get started. The easiest, perhaps, is to sign up on the public site and create an account there. As development becomes more involved, you may decide you’d like to iterate quicker and work with a cloud that’s running locally. For this very specific use case, consider the Micro Cloud Foundry (MCF). MCF is a virtual machine image that runs in VMware Player on Linux and Windows, and VMware Fusion on OSX (Player is free, but Fusion has a nominal price attached to it.) You can sign up for an account and download the image for free from the MCF website then simply follow the setup guide. For our purposes, let’s simply work with the online site.
There are a few tools we need to effectively work with any Cloud Foundry instance: the vmc command line tool, and – if you’re a developer working with the JVM (Spring, Grails, Scala, regular applications, etc.) – the Eclipse support for Cloud Foundry.
There are many ways to interact with Cloud Foundry. Cloud Foundry’s public facing endpoint, called the cloud controller, is a RESTful API through which all commands pass and eventually effect some sort of response from one of the components in the cloud’s architecture itself. While you manipulate Cloud Foundry using this RESTful API itself, it’s easier to use language and client-specific integrations. The vmc command line tool provides a great way to interact with Cloud Foundry for those more comfortable banging around in the shell; the Eclipse support provides a WTP-connector view of Cloud Foundry; both the Grails and Roo RAD frameworks provide add-ons, and there is support for deploying applications with a Maven plug-in.
Signing up for a CloudFoundry.com account
Signing up is easy. While the project’s in beta, accounts on CloudFoundry.com are free. Go to the Cloud Foundry registration page and follow the steps.
Working with vmc
The instructions are presented here, but they may change in time, so be sure to consult the official documentation section for the latest and greatest, if anything is in doubt. The vmc command line tool is written in Ruby. So, first, you need to install Ruby on your operating system. At the moment, in the second quarter of 2012, Ruby versions 1.9.2 (or later) and 1.8.7 (or later) are supported. On Linux, you can use your operating system’s package manager to install Ruby (if it’s not already installed). If you’re on Windows, you can use the Ruby installer from RubyInstaller.org. On OSX, things are less rosy. The built-in Ruby is fragile. It’s easier to install something like the Ruby Version Manager (rvm) which lets you maintain multiple concurrent installations of Ruby on a given machine. The setup for Ruby, with rvm, looks like this:
$ bash -s stable <<(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
$ rvm install 1.9.2
$ rvm use 1.9.2
Once you have Ruby installed, getting the vmc command line tool’s a snap! Ruby comes with a package management tool called gem, which it uses to manage dependencies. It’s akin to a Ruby-specific version of the yum, deb, or apt package management systems on various Linux flavors. To install vmc, you will use the gem command line tool, like this:
$ sudo gem install vmc
If you’re running on Windows, then remove the word sudo, which is used on Unix variants like OSX and Linux to obtain elevated privileges when invoking a command, and is unnecessary on Windows. Once vmc is installed, open a new shell to test some of the basics out. When working with Cloud Foundry instances, you need to follow some basic workflow steps.
First, you need to target the correct instance, since Cloud Foundry is not one cloud by one vendor, but a stack that anybody can run. This is similar to the way you work with Git and Subversion: You wouldn’t install the Git command and expect it to automatically know which host to clone the source code from, would you? As a good rule of thumb, targeting involves specifying the api URL for the Cloud Foundry instance. This is pretty consistent of the pattern api.foo.com, where foo.com is the host of your Cloud Foundry instance. Here’s my console session targeting CloudFoundry.com:
$ vmc target api.cloudfoundry.com
Successfully targeted to [http://api.cloudfoundry.com]
Then, you need to login. You only need to login once for a given session.
$ vmc login
Attempting login to [http://api.cloudfoundry.com]
Successfully logged into [http://api.cloudfoundry.com]
Working with Cloud Foundry
If everything succeeds, you can now start to ask questions and interact with Cloud Foundry. Here’s my output of the vmc info command.
$ vmc info
VMware’s Cloud Application Platform
For support visit http://support.cloudfoundry.com
Target: http://api.cloudfoundry.com (v0.999)
Usage: Memory (1.6G of 2.0G total)
Services (6 of 16 total)
Apps (3 of 20 total)
Another handy command is the vmc apps command, which tells you which applications you’ve got running. I get the following output when I run the vmc apps command:
$ vmc apps
Is there anything more depressing? We need to deploy an application!
I’ve created a simple Spring MVC application that runs on both your local machine (working with an in-memory database, called H2, and working with Cloud Foundry-managed instances of PostgreSQL, a powerful relational database and Redis, the lightning fast data-structure service. To get the code, simply clone it from the cloudfoundry-samples GitHub project.
$ git clone firstname.lastname@example.org:cloudfoundry-samples/springmvc-hibernate-template.git
The application is a Java project, and uses Maven for its build. To compile the code, you simply enter the root of the project and then run the mvn install commands, like this:
$ mvn install
Now that we’ve got the platform, and an application, let’s deploy it using the vmc tool. I can imagine your thoughts: “C’mon, man! We can’t deploy this thing yet! It’ll blow chunks – you just finished telling us it needs PostgreSQL and Redis!” And, that’s a good point! But, don’t worry! We’ll take care of deploying the application and configuring those services in one stroke. You see, the vmc tool has to rank among the friendliest of command line tools out there. Let’s walk through the push interaction:
$ vmc –path target/springmvc31-1.0.0 push
This tells vmc to push the compiled code in the directory specified in the –path argument. If you don’t specify –path, it looks for the code to deploy in the current directory. If you’re deploying a project in a language that doesn’t have a compiled form, then specify the root directory of that project instead.
Application Name: springoncloudfoundryforjax
Specify a name for the application. The name you specify must be different than the one I’ve specified here (otherwise, there would be a collision!)
Application Deployed URL [springoncloudfoundryforjax.cloudfoundry.com]:
Specify the URL (or hit enter to accept the default).
>Detected a Java SpringSource Spring Application, is this correct? [Yn]: n
9: Erlang/OTP Rebar
Select Application Type: 5
Selected Java Web Application
vmc has detected the type of the application (“a Java SpringSource Spring Application”). We have instead selected option 5, “JavaWeb,” because our application doesn’t need to be auto reconfigured. Auto reconfiguration is a service that Cloud Foundry provides when applications are uploaded: It redefines the definition of services in certain types of applications and automatically binds them to the appropriate definition for the data source managed by Cloud Foundry. This is useful if you have only one data source, or one Redis declaration, and would simply like to have that automatically rewired to work on Cloud Foundry. The alternative is to dynamically look up the connection information from a well-known context, like the VCAP_SERVICES environment variable that is set in the Cloud Foundry environment. There is also a Java API, called the cloudfoundry-runtime which provides a type-safe Java API to do the same work. The application already looks up the definition using the cloudfoundry-runtime, and does not need auto-reconfiguration.
Memory reservation (128M, 256M, 512M, 1G, 2G) [512M]:
How many instances? :
Next, vmc asks us to specify some of the runtime characteristics of the application: How much memory does it need? How many instances should be started? For now, we’ll just accept the defaults.
Bind existing services to ‘springoncloudfoundryforjax’? [yN]: n
Create services to bind to ‘springoncloudfoundryforjax’? [yN]: y
What kind of service?: 3
Specify the name of the service [postgresql-63c39]:
Create another? [yN]: y
What kind of service?: 5
Specify the name of the service [redis-48adb]:
Create another? [yN]: n
Finally, the services: the Redis and PostgreSQL instances, are set up. Cloud Foundry is nice enough to ask us if we need services for our application and prompts us to specify as many services as we need. Services in Cloud Foundry are not coupled from the applications that consume them: You can specify as many services as you like. They must be bound to the applications that use them. You may share services across applications, and may use as many services as you like for any application. Our application is written to lookup references by type, not by name, so it does not matter how we name the services.
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
That was tedious! Sure, it did deploy our application and completely configure the services we need, but I’d sure hate to answer all those questions every time I deploy or update an application. No, no. Leave that to somebody else, I say. This is what the Cloud Foundry manifests provide: a manifest of everything Cloud Foundry needs to deploy the application without prompting you. This makes it easy to share your configuration with others. Cloud Foundry’s even willing to write the manifest.yml file for us!
Creating Service [postgresql-63c39]: OK
Binding Service [postgresql-63c39]: OK
Creating Service [redis-48adb]: OK
Binding Service [redis-48adb]: OK
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (12K): OK
Push Status: OK
Staging Application ‘springoncloudfoundryforjax': OK
Starting Application ‘springoncloudfoundryforjax': OK
At this point, the application’s available and running! You can confirm this by checking the output of vmc apps (Figure 1).
Figure 1: Output of vmc apps
That’s much better! Bring up the application by opening the URL specified in the output of vmc apps in your browser. This application has a simple UI to let manipulate records of customer data. There is also a RESTful endpoint that you can hit at /crm/customers. The result is a JSON structure with individual records. When you first deploy the application, there are no records! Let’s fix that.
The light (of data) at the end of the tunnel
We’ve had an easy time of deploying an application thus far, but the operations and DBAs reading this are probably nervously tapping their fingers at this point: “Where is my database console? My backup scripts?” Indeed, the operations people in your organization are the ones who have to wear the beeper when they go to bed, and they want to know that they can get access to the data with ease, or run their routine diagnostic and backup jobs, just like before. Cloud Foundry provides caldecott (named for the caldecott tunnel in San Francisco, in case you’re wondering…), which is a tunnel. Caldecott lets you poke a hole in the cloud, connecting you to your services running inside the cloud from the outside. Let’s connect to our PostgreSQL instance and install some data using the vmc tunnel command.
$ vmc tunnel
Which service to tunnel to?: 2
The command will list the services that we have created, and to which we can establish a tunnel. Specify the number for the PostgreSQL instance in your account, in my case, #2.
Deploying tunnel application ‘caldecott’.
Checking for available resources: OK
Packing application: OK
Uploading (1K): OK
Push Status: OK
Binding Service [postgresql-63c39]: OK
Staging Application ‘caldecott': OK
Starting Application ‘caldecott': OK
Getting tunnel connection info: OK
Service connection info:
username : u63a3924f90c6446f9737390675f1a9e0
password : p9ccfd8a6b4d04e308396a872a9711155
name : d040a2dbbc977418f9eb599a2245fa96d
By default, the command outputs the information you’d need to establish your own connection from any client, command-line or not: the host, port, username, password and name, which basically just means schema, for most databases.
Starting tunnel to postgresql-63c39 on port 10000.
Which client would you like to start?: 2
Launching ‘psql -h localhost -p 10000 -d d040a2dbbc977418f9eb599a2245fa96d -U u63a3924f90c6446f9737390675f1a9e0 -w’
psql (9.0.5, server 9.0.4)
Type “help” for help.
If that weren’t enough, vmc will even start up many well-known, local command line tools like psql for you! I let it, and it dumps me into the standard psql shell, where I can execute my SQL statements. The project provides some sample data in the file src/main/resources/import_psql.sql. Copy and paste that into the shell, and then revisit the /crm/customers JSON dump page. You should see your new data reflected in the results. The tunnel is a powerful tool, and provides everything you need for ad-hoc queries, as well as operational features like nightly dumps of the data.
We’ve looked at how to deploy a simple Spring application to Cloud Foundry entirely from the command line. We did not look at how to use the Eclipse WTP support, which is ably covered on the Cloud Foundry website and in the SpringSource blog. We also have not looked at the development model support for Spring, Ruby, Node.js, Scala, etc. Cloud Foundry is, after all, a polyglot PaaS!
If you’re a Spring developer, then you have a lot of clean integrations to help you along; it provides a rich set of APIs that maps nicely to the supported services on Cloud Foundry: Spring AMQP and Spring Integration make RabbitMQ-based projects very easy. Spring Core and Spring Batch make batch workloads simply with MySQL and PostgreSQL. Spring Data Redis and Spring Data MongoDB both provide rich support for Redis and MongoDB. If you’re a Java developer, and want to build applications on top of Cloud Foundry, you’re in luck!
At this point you’ve got all you needed to get started and plenty of information on where to go from here. If you have any questions, don’t hesitate to reach out to me at email@example.com, or reach out at one of the numerous public forums on the Cloud Foundry website. So, sit back, relax, and enjoy the flight to the clouds!
Josh Long (@starbuxman, and firstname.lastname@example.org) is the Spring developer advocate. Josh is the lead author on Apress’ Spring Recipes, 2nd Edition, O’Reilly’s “Getting Started with Spring Roo,” Manning’s “Cloud Foundry in Action.” and a SpringSource committer and contributor. When he’s not hacking on code (for Spring Integration, Spring Batch, Spring MVC, Activiti, and much more), he can be found at the local Java User Group or at the local coffee shop. He blogs at springsource.org, blog.springsource.org or joshlong.com.
Andy Piper is part of the VMware Developer Relations team, and works as Developer Advocate for Cloud Foundry, the Open Source Platform-as-a-Service. He is probably best known online as a “social bridgebuilder” spanning a number of areas of technology and interest. He was previously with IBM Software Group for more than 10 years, as a consultant, strategist, and WebSphere Messaging Community Lead.
This tutorial originally appeared in Java Tech Journal – PaaS Above The Cloud Hype. Check out more of that issue in the link.