At UDS in October we had a session about the things we wanted to do for Openstack in Precise, the list of work items that we want to get through is found in the blueprint. Right now, in the archive we have again a weekly snapshot of Nova and Swift like in Oneiric, but we have also added a weekly snapshot of keystone, horizon, and quantum that is available as an apt installable package. Soon we will be adding Melange to that rotation since it is now packaged in Ubuntu as well. Please try them out and report any bugs in Launchpad.

On the Juju side, Adam has updated the charms for precise and have added a charm for keystone, I also believe he is close to finishing a charm for horizon as well. This will make installing OpenStack really easy since all the configuration is done for you in the charms.

Since Precise is an LTS release we care about the stability of the Openstack packages we are the middle of figuring out a way of doing on commit tests from the Openstack git repository. The way that this will probably work is that we will create packages from a commit that will happen, and the packages will get rebuild with a new snapshot and will use the charms to deploy Openstack on in the QA Lab. This will give us confidence that the packages and the charms are well tested before they are uploaded to the Ubuntu archive.

Looking back to Oneiric and the Diablo release, there were a lot of bugs that affected the stability and quality for production deployment. With that in mind we had a session at the OpenStack Developers summit about creating a stable release team. The team is responsible for accepting back ported patches from Essex into  Diablo, kind of like what the stable kernel tree maintainers do for the Linux kernel. The team is made up of people from both Fedora and Ubuntu, so it is nice to see some collaboration taking place. I do not have any statistics on how many patches were back ported from Essex to Diablo but I did review *alot* of patches. What that means for Ubuntu is that we have a stable diablo release for Oneiric that is available for testing in the proposed repository. To learn how to enable the proposed repository you can read the instructions on the Ubuntu Wiki.

With that I will be going on Christmas holidays near the end of the week, so I want to wish you a Merry Christmas and a Happy Holidays. I hope to have another development update soon.

The design summit was for the first 3 days while I was in Boston. Since I work at Canonical and as a member of the Ubuntu Server Team, these days I am mostly interested in Distros/Integration (since our next release is an LTS release), Orchestration, and the EC2 API.

• Integration – Distros such as Ubuntu, Fedora, and RHEL (and ultimately providers such as HP and Dell) care more about having a stable release the an cutting edge release so that their users can use Openstack without having to pull their hair out. Dave Walker hosted a session about doing a Stable Release where interested parties can use a common base for distros and integrators role out releases to their users. The result of this session was that a group of interested parties will be created to take care of it. Later in the week we had a Debian packaging BOF and a Distro BOF and we got to talk about Packaging and what other people are doing with Openstack as well.
• Orchestration – providing a path for users to deploy Openstack easily is becoming a *BIG* thing in the Openstack ecosystem. Companies such as Dell, Nebula, and Canonical are providing tools such as Crowbar and Juju/Orchestra. They  are all competing to do the same thing and all have basically the same ideas. The real question is how can they innovate a deployment and set themselves apart from one another.
• EC2 API - I hosted a session about the EC2 API in Openstack. Quite frankly, I was surprised how full the room was and there was a lot of good discussion moving forward. In the Diablo release, there was more focus on the Rackspace API since Rackspace is going to be deploying the Openstack for their customers so a lot of bugs didnt get fixed in time. However, there is a lot of discussion on how people still want their tools such as ElasticFox and boto to still work on Openstack. I think the consensus at the end of the session was to form  a “working group” of people who are interested in fixing and maintaining the EC2 API in Nova.

The final two days were for the conference. The conference was definitely more business and marketing focus than the design summit. Some of the highlights for me at the conference were the following:

• Juju – On the final day of the conference Jane Silber, Canonical’s CEO, gave a presentation of what we are doing with Openstack and juju. It was a pretty cool demo because the deployment of a hadoop cluster was visualized using gource as it was deployed. I think a lot of people are visual people and it was a good way to get a lot of peoples attention. After the demo, I think there was a lot of buzz around juju since people wanted to know more about it. The juju session in the afternoon on Friday was packed and the canonistack use case session was well attended because people wanted to know more.
• HP – At the conference HP gave a keynote on how they are using Openstack to offer a public cloud. Right now it is in beta testing for people that have been signed up and have been accepted by HP. What a lot of people didn’t know was that their cloud was running on Ubuntu Server, which was also announced in Jane’s keynote.
• ARM – Ever since the Corel Netwinder was released I have been an ARM fan boy and Jane announced in her keynote that Oneiric is one of the first server based distros that can run on ARM. However since this is a Openstack conference,  yes in Oneiric you can run Openstack on ARM. There are ARM specific images that you can use and you can get them at the regular places.  It uses LXC for virtualization. I will probably have another blog post about this later.
• Openstack Foundation – In the Rackspace keynote, it was mentioned that there is over 150+ companies developing for Openstack, that is a pretty big ecosystem. So it was announced that a foundation will be created to manage the trademark and allow for a better community governance model. This is a good move so it will be interesting to see how it goes forward

It was a great week to be in Boston and seeing a lot of old friends in the community. However I just realized that UDS is not far down the road and I will be doing this all again soon.

At UDS-O in Budapest I hosted a session about bringing back Xen support for Ubuntu in Oneiric. It was a well attended and well received session and was kind of surprised about the attendance. At the time there was and still is having Ubuntu users run the Xen hypervisor, kernel, and user land tools on Ubuntu.

So how do you install In short this is how to do it:


sudo apt-get install linux-image-server
sudo apt-get install xen-utils-4.1 xen-utils-common
sudo apt-get install xen-hypervisor-4.1 (xen-hypervisor-4.1-amd64 if you are running amd64)
sudo apt-get install xenstore-utils


Its really easy to use libvirt and xen on Oneiric as well. There are a couple of bugs when trying to launch an HVM domain, but it will get fixed for the next LTS. Obligatory screen shot of running the Ubuntu Server installer on an HVM instance can be found below:

If you have any questions or have troubles running Xen on Oneiric. Please use launchpad to open bugs and so we can track the problems people are having with it.

There has been a lot of changes between the Cactus release and Diablo release of Nova. This blog will help you get started running a basic cloud on Ubuntu Oneiric. So lets get started!

First install Nova and Glance, Glance is now required for the API server:
sudo apt-get install nova-api nova-compute-kvm nova-scheduler nova-network nova-objectstore nova-volume
rabbitmq-server dnsmasq glance bridge-utils euca2ools python-novaclient

You will need to setup a bridge, information on how to do it can be found at the Ubuntu Server Guide.

Once Nova and Glance is installed you have to setup Nova. Setting up Nova  is relatively easy to do, all you need to do is to type the following:

sudo nova-manage db sync
sudo nova-manage user admin [username]
sudo nova-manage project create [name of cloud] [username]
sudo nova-manage network create public 10.0.0.0/8 3 16 --bridge_interface=br0
sudo nova-manage project zipfile [name of cloud] [username]

Once the cloud is up and running you will need to have an image to run on your cloud. You can download the latest release from here.

To upload an image to your cloud is easy as well, all you have to do is the following once you download the image:

unzip nova.zip
source novarc
uec-publish-tarball [name of tarball] bucket [arch]
euca-add-keypair -k [name of key] > [filename]
chown 600 [filname]


Finally to run your instance all you have to do is:

euca-run-instance -k [name of key] [instance-id]

Once the image has started, you should have a virtual machine running on your workstation. So there you have it..an EC2 compatible cloud running on your workstation. If you find any bugs please report them in launchpad. The more users we get testing Nova on Ubuntu the better. If you have any question please let me know.

The Openstack ecosphere has grown a lot of over the past couple of months. With major hardware vendors such as HP, Dell, and Cisco announcing support for Openstack, the marketplace is getting crowded with everyone announcing an Openstack IaaS solution.  With more and more companies offering a cloud solution, it is more than likely that they  are going to be using the same “Diablo” release of the Openstack.  So how are they going to set themselves apart from each other and hopefully stand out? Its all about the infrastructure baby! Right now, there is still a lot of fiddling a user has to do in order to use Nova. Providing an infrastructure for deployment for any number of machines have to be easy and straight forward as possible. Right now, there is a lot of deployment tools that users can use, which is great because it offers users choice.  Lets look at  some of the offerings that I am aware about.

Lets start off with Dell. Last week at OSCON, Dell had announced “crowbar”. Crowbar is Dell’s solution to the Openstack deployment infrastructre. The twitter universe has been all gaga for it (good marketing) and it looks nice. It is based on Chef and the source is available. Dell is trying to get a community around it which is a good thing. However it looks very Dell specific and right now its based off of Ubuntu 10.10.

Next lets have a look at StackOps. StackOps has their users download an ISO. Once the user installs a base system, it has a web base installer as second stage installer. The second stage installer seems to be hosted by StackOps which is probably not an ideal way to deploy Openstack.  This is eerily similar what we did with Eucalyptus minus the web based second stage installer, and have problems with in the past.   It is also based off of Ubuntu right now as well. However, this mode of delivery seems to me not for scale for large deployment.

Finally what have Canonical been working on, well its called Orchestra. The idea behind Orchestra is given a number of nodes in a deployment, deploy nova, swift, and glance easily. The tools that we use are the best of the open source technologies, so in Orchcestra we use  cobbler to deploy the base operating system and ensemble to deploy Openstack. Orchestra has other features such as monitoring your deployment using Nagios, and a central rsyslog to have all your log files in one place.

Right now, Orchestra is still undergoing a high pace of development and I am quite excited to see where it is going and being the premier tool for deploying Openstack on Ubuntu.

• Openstack  At UDS, Ubuntu Cloud was announced which is run by Openstack. Everyone has assumed that this means that UEC and Eucalyptus is going away, which is not the case in Oneiric. Both Eucalyptus and Openstack will be in “main” and both will be supported in Oneiric. One way to think of this is that we both have MySQL and Postgresql in “main” because people still want to have a choice in which database you want to use. Providing both allows users to have an easier time to deploy either MySQL or Postgresql.  Users are free to use either MySQL or Postgresql, which is really the essence of FOSS. Similarly,  Ubuntu users for Oneiric will have either Eucalytpus or Openstack as a choice to deploy their private cloud. The session that we had  Openstack mostly resolved about what integration points that we need to do for Oneiric and how we are going to do them.
• Orchestra was a big topic at UDS. Orchestra is an all in one provisioning server to deploy Ubuntu Server. Orchestra has several moving parts which includes cobbler, puppet, mcollective, nagios, cloud-init, and ensemble. The idea behind Orchestra is that given any N number of  servers running bare-metal, easily deploy Ubuntu Server on it. This is a big change since the only way you could get Ubuntu Server was through the ISO and it was designed for a single server mind. With Orchestra, we are taking the best of opensource and making it easier to deploy Ubuntu Server for large scale deployments. Most of the sessions that I attended were either directly or indirectly apart of Orchestra.
• Ubuntu Server on Arm A long time ago when I was first starting out in the computer industry. I had access to a small little server called the netwinder. It was a StrongARM desktop server that runs quietly on a desk or as a 1U. The Netwinder was basically my first exposure to the ARM architecture and it was pretty cool, however it was way ahead of its time and died when Corel got out of the Linux business and Rebel.com died.  Jump to, well, now, and there is a demand for servers with low power consumption and ARM is well suited in my opinion. Ubuntu Server is getting on the ground floor and I quite envious of the Ubuntu Arm team of the work they are doing in this area.
• Ubuntu as a host or what is old is new again Along with sessions about LXC and KVM we had a session about discussing Xen. Xen in the past has not been really supported by the Ubuntu Server Team or the Ubuntu Kernel Team  due to the Xen patchset not included in the mainline kernel.  Since the dom0 support has been included in the mainline kernel, distros such as Ubuntu can now support it properly. We had a good discussion about Xen and what needs to be done in order for Ubuntu to be a good host for Xen. I see us carrying the Xen 4.1 hypervisor and userland tools and associated bits that is needed to make Xen work really well on Ubuntu.  It was great to have Ian Jackson an upstream xen.org developer from Citrix to answer questions for us as well.
• Server SRU process review Surprisingly, this was a well attended session as well on the last day. Most of our users on Ubuntu Server use the LTS releases and they want to have a stable server. In order to support those users we usually try to cherrypick fixes from the development release and backport them to the LTS releases and believe me we do care about our LTS uses. However, this takes time and we have a long list of things that we want to fix but just have not  gotten around to it yet. A couple of ideas was risen from this session include using the backports, SRU process, and PPAs. However, the idea brought up of having a dedicated SRU person on the Ubuntu server who would rotate on a fixed time period was especially interesting because it would make sense for us since we would model it on how to the security team would do it.

It was nice to see everyone again and I’m looking forward to the next six months but there is a lot of work to be done. So okay go!

Openstack is still the new cloud on the block, it is providing lots of new cool features that users will enjoy. Also it has gotten a lot of publicity over the past couple of months in the twitter and blog sphere. For me, the features that I like the best in this release are the ability to use live migration and LXC. Live migration allows the system administrator to move VM from server to server. LXC support is also important because its a lightweight technology that is up and coming as well. The release schedule is also great because it is basically the same as Ubuntu, so the latest release of Ubuntu will also have the latest release of Openstack as well. However, like all open source projects it is going to have its growing pains which I have no doubt that they are going to work through them.

Ubuntu users can use Openstack using the upcoming Natty Narwhal release. The packages were uploaded this morning a couple of hours after the cactus release so it will be in the archive soon. It just so happens that the beta2 just happened to be released yesterday as well. So please help find bugs before we release the final version at the end of the month. If you need VMs to run openstack my suggestion is that you use the UEC images since they are compatible with Openstack as well.

But have you ever wonder if you can run the “cloud” within the “cloud”, especially on Amazon EC2. Well now you can using Ubuntu Server, Openstack, and LXC as a base. Here is how..

First you need an Amazon EC2 instance. For our little project I suggest you use the latest Ubuntu Server release called “Natty Narwhal” on EC2 running an amd64 image (m1.large) . Get the largest instance you can because its easier to use in the long run, less headaches, and less hassles.

Once you instance has booted the first thing you have to do is to make sure your apt repositories are up to date and to enable cgroups. Cgroups is a linux kernel feature to limit, account, and isolate resource groups and processes. Cgroups is needed for libvirt to run LXC. So in order to enable cgroups in your EC2 instance all you have to do is the following:

sudo mkdir /cgroups


And modify your /etc/fstab accordingly

none /cgroups cgroup cpuacct,memory,devices,cpu,freezer,blkio 0 0


Just to be on the safe side reboot your EC2 instance. After you have rebooted its time to start installing Openstack on your instance. Since we are running a cloud on a single instance you need to do the following first:

sudo apt-get install rabbitmq-server dnsmasq


We are installing rabbitmq-server because its the AMQP provider that interacts with the different components of Nova. We are also installing dnsmasq because its basically your DHCP server for you cloud setup.

So next you have to install Nova. To do this on Ubuntu you have to do the following:

sudo apt-get install python-nova nova-common nova-api nova-compute nova-network nova-objectstore nova-scheduler nova-volume python-nova.adminclient unzip euca2ools


After you have installed the packages the next thing you have to do is configure Nova for LXC. To do this you need to modify the /etc/nova/nova.conf configuration file and add the following “–libvirt_type=lxc”. This will tell the nova-compute component to run libvirt to run with lxc. Then just to make life easier, reboot the instance again.

Once the instance has come backup the next step is configure your cloud, to do that you have to do the following:

sudo nova-manage network create 10.0.0.0/8 1 64
sudo nova-manage user admin
sudo nova-manage project create
mkdir me
cd me
sudo nova-manage project zipfile
source novarc


Once you have your cloud ready you need to run images on your cloud. Serge Hallyn has made some upstart scripts that are part of a the lxcguest package in Natty which makes it easier to run Ubuntu as a LXC container. Luckily, the UEC images found in “Natty Narwhal” have this package installed by default so the user doesn’t have to do any fiddling with their images to get it to work. To download the images you can point your favorite web browser here. Just remermber to download your images to your images to EC2.

Finally you need to upload the images to Nova’s objectstore. Its pretty easy to do:

tar -zxvf natty-server-uec-amd64.tar.gz
euca-bundle-image -i natty-server-uec-amd64.img
euca-upload-bundle -b -m /tmp/natty-server-uec-amd64.img.manifest.xml
euca-register-image /natty-server-uec-amd64.img.manifest.xm


Finally…you will  want to run the instance that you just uploaded to the objectstore:

euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv
euca-run-instances -k mykey -t m1.tiny


Using the “euca-describe-instances” command you should be able to see the instance running in a couple of minutes. Then finally should be able to ssh into the instance as the root user.

On a final thought is that I would like to the thank the Nova developers for making deploying a public/private cloud pretty easy and painless to do. If you have any feedback for me then you can contact me via email. Bug reports should be opened in Launchpad under the appropriate project. Enjoy your cloud with a cloud!

A feature for Natty that the Ubuntu Server team has been adding and improving is LXC container support. Linux containers is a light weight virtualization technology described by some as “chroots on steroids”.

In the past running LXC containers need a lot of hacks in the image due to various issues with upstart. But that has changed to the awesomeness of the lxcguest package. The lxcguest package will make running Ubuntu on LXC containers a trivial task. So running scripts to modify the images is now a thing of the past. Thanks Serge.

Also a new version of libvirt was uploaded to Natty recently. This version contains numerous fixes for LXC, previously in maverick if you ran container, the container would disappear once you connect to the console. Thankfully this is also fixed in more recent versions of libvirt, thanks to the upstream maintainers for libvirt. Also to  Serge again for back porting the fix to Natty.

So you are probably where this is all leading to. Well now with the trunk version Open Stack you can run LXC enabled images as an EC2 or OpenStack API instance. This is a good thing for a couple of reasons, one reason is that it allows system administrators who doesn’t have the hardware to easily start deploying clouds in their data centers. However, for the high end system administrators LXC is a low overhead  and fast deployment lots of instances in their Private Cloud. For developers of the openstack community it gives Nova developers who did not want to deploy either KVM or Xen a way to test their changes to the Nova code more easily.

Next we will look at how to deploy Nova with LXC.

Install cobbler

sudo apt-get install cobbler cobbler-common python-cobbler cobbler-web

Grab an Ubuntu server Iso

wget http://cdimage.ubuntu.com/ubuntu-server/daily/current/natty-server-amd64.iso
sudo mount -o loop natty-server-amd64.iso /mnt

Configure cobbler

sudo cobbler check

And follow the instructions it gives you. This makes sure that cobbler is configured properly on your server.

Import the ISO

sudo cobbler import --name=profile_name --path=/mnt --breed=ubuntu

And essentially you are ready to deploy Ubuntu Server via netboot. Since it is brand new we users to use this software and would to hear about bugs that you may have encounter. You can file them here.