Friday night I returned from Boston where I was attending the Openstack Design Summit and Conference hosted by Rackspace. The Design Summit is a UDS like where developers come together and hammer out a feature in the form of a blueprint for the next release of Openstack. I think the concept is still new to a lot of people since I still saw more presentations than discussions at the sessions I attended. However people will get use to the format as time goes on.
The design summit was for the first 3 days while I was in Boston. Since I work at Canonical and as a member of the Ubuntu Server Team, these days I am mostly interested in Distros/Integration (since our next release is an LTS release), Orchestration, and the EC2 API.
- Integration – Distros such as Ubuntu, Fedora, and RHEL (and ultimately providers such as HP and Dell) care more about having a stable release the an cutting edge release so that their users can use Openstack without having to pull their hair out. Dave Walker hosted a session about doing a Stable Release where interested parties can use a common base for distros and integrators role out releases to their users. The result of this session was that a group of interested parties will be created to take care of it. Later in the week we had a Debian packaging BOF and a Distro BOF and we got to talk about Packaging and what other people are doing with Openstack as well.
- Orchestration – providing a path for users to deploy Openstack easily is becoming a *BIG* thing in the Openstack ecosystem. Companies such as Dell, Nebula, and Canonical are providing tools such as Crowbar and Juju/Orchestra. They are all competing to do the same thing and all have basically the same ideas. The real question is how can they innovate a deployment and set themselves apart from one another.
- EC2 API – I hosted a session about the EC2 API in Openstack. Quite frankly, I was surprised how full the room was and there was a lot of good discussion moving forward. In the Diablo release, there was more focus on the Rackspace API since Rackspace is going to be deploying the Openstack for their customers so a lot of bugs didnt get fixed in time. However, there is a lot of discussion on how people still want their tools such as ElasticFox and boto to still work on Openstack. I think the consensus at the end of the session was to form a “working group” of people who are interested in fixing and maintaining the EC2 API in Nova.
The final two days were for the conference. The conference was definitely more business and marketing focus than the design summit. Some of the highlights for me at the conference were the following:
- Juju – On the final day of the conference Jane Silber, Canonical’s CEO, gave a presentation of what we are doing with Openstack and juju. It was a pretty cool demo because the deployment of a hadoop cluster was visualized using gource as it was deployed. I think a lot of people are visual people and it was a good way to get a lot of peoples attention. After the demo, I think there was a lot of buzz around juju since people wanted to know more about it. The juju session in the afternoon on Friday was packed and the canonistack use case session was well attended because people wanted to know more.
- HP – At the conference HP gave a keynote on how they are using Openstack to offer a public cloud. Right now it is in beta testing for people that have been signed up and have been accepted by HP. What a lot of people didn’t know was that their cloud was running on Ubuntu Server, which was also announced in Jane’s keynote.
- ARM – Ever since the Corel Netwinder was released I have been an ARM fan boy and Jane announced in her keynote that Oneiric is one of the first server based distros that can run on ARM. However since this is a Openstack conference, yes in Oneiric you can run Openstack on ARM. There are ARM specific images that you can use and you can get them at the regular places. It uses LXC for virtualization. I will probably have another blog post about this later.
- Openstack Foundation – In the Rackspace keynote, it was mentioned that there is over 150+ companies developing for Openstack, that is a pretty big ecosystem. So it was announced that a foundation will be created to manage the trademark and allow for a better community governance model. This is a good move so it will be interesting to see how it goes forward
It was a great week to be in Boston and seeing a lot of old friends in the community. However I just realized that UDS is not far down the road and I will be doing this all again soon.
In the past before I started working at Canonical, I volunteered to maintain Xen on Ubuntu. This included supporting the hypervisor and the rather large kernel patch that needed to be ported to whatever version that Ubuntu is running on at the time. This process was not sustainable and KVM was eventually chosen as the hypervisor of choice for Ubuntu Server. Today the Xen dom0/domU kernel code is the upstream vanilla source so distros like Ubuntu can get from Xen.
At UDS-O in Budapest I hosted a session about bringing back Xen support for Ubuntu in Oneiric. It was a well attended and well received session and was kind of surprised about the attendance. At the time there was and still is having Ubuntu users run the Xen hypervisor, kernel, and user land tools on Ubuntu.
So how do you install In short this is how to do it:
sudo apt-get install linux-image-server
sudo apt-get install xen-utils-4.1 xen-utils-common
sudo apt-get install xen-hypervisor-4.1 (xen-hypervisor-4.1-amd64 if you are running amd64)
sudo apt-get install xenstore-utils
Its really easy to use libvirt and xen on Oneiric as well. There are a couple of bugs when trying to launch an HVM domain, but it will get fixed for the next LTS. Obligatory screen shot of running the Ubuntu Server installer on an HVM instance can be found below:
If you have any questions or have troubles running Xen on Oneiric. Please use launchpad to open bugs and so we can track the problems people are having with it.
On October 13, 2011 – the Oneiric Ocelot release will be available to Ubuntu Server users everywhere. With it, a new release of Nova, Swift and Glance called “Diablo” will be available for users to test drive for their EC2 compatible private cloud. Thierry Carrez, Openstack release manager and Ubuntu developer has a detailed blog post about the features of the last Nova milestone at his blog.
There has been a lot of changes between the Cactus release and Diablo release of Nova. This blog will help you get started running a basic cloud on Ubuntu Oneiric. So lets get started!
First install Nova and Glance, Glance is now required for the API server:
sudo apt-get install nova-api nova-compute-kvm nova-scheduler nova-network nova-objectstore nova-volume
rabbitmq-server dnsmasq glance bridge-utils euca2ools python-novaclient
You will need to setup a bridge, information on how to do it can be found at the Ubuntu Server Guide.
Once Nova and Glance is installed you have to setup Nova. Setting up Nova is relatively easy to do, all you need to do is to type the following:
sudo nova-manage db sync
sudo nova-manage user admin [username]
sudo nova-manage project create [name of cloud] [username]
sudo nova-manage network create public 10.0.0.0/8 3 16 --bridge_interface=br0
sudo nova-manage project zipfile [name of cloud] [username]
Once the cloud is up and running you will need to have an image to run on your cloud. You can download the latest release from here.
To upload an image to your cloud is easy as well, all you have to do is the following once you download the image:
uec-publish-tarball [name of tarball] bucket [arch]
euca-add-keypair -k [name of key] > [filename]
chown 600 [filname]
Finally to run your instance all you have to do is:
euca-run-instance -k [name of key] [instance-id]
Once the image has started, you should have a virtual machine running on your workstation. So there you have it..an EC2 compatible cloud running on your workstation. If you find any bugs please report them in launchpad. The more users we get testing Nova on Ubuntu the better. If you have any question please let me know.
Wow…long time no blog. It has been really busy the last couple of months since I have written a blog post, busy you know with Oneiric development work on Ubuntu Server. Oneiric seems to be coming along nicely and we are shaping up for an interesting release when its done. Especially with the the Openstack bits that are being worked on both by the Ubuntu Server Team and the Openstack community.
The Openstack ecosphere has grown a lot of over the past couple of months. With major hardware vendors such as HP, Dell, and Cisco announcing support for Openstack, the marketplace is getting crowded with everyone announcing an Openstack IaaS solution. With more and more companies offering a cloud solution, it is more than likely that they are going to be using the same “Diablo” release of the Openstack. So how are they going to set themselves apart from each other and hopefully stand out? Its all about the infrastructure baby! Right now, there is still a lot of fiddling a user has to do in order to use Nova. Providing an infrastructure for deployment for any number of machines have to be easy and straight forward as possible. Right now, there is a lot of deployment tools that users can use, which is great because it offers users choice. Lets look at some of the offerings that I am aware about.
Lets start off with Dell. Last week at OSCON, Dell had announced “crowbar”. Crowbar is Dell’s solution to the Openstack deployment infrastructre. The twitter universe has been all gaga for it (good marketing) and it looks nice. It is based on Chef and the source is available. Dell is trying to get a community around it which is a good thing. However it looks very Dell specific and right now its based off of Ubuntu 10.10.
Next lets have a look at StackOps. StackOps has their users download an ISO. Once the user installs a base system, it has a web base installer as second stage installer. The second stage installer seems to be hosted by StackOps which is probably not an ideal way to deploy Openstack. This is eerily similar what we did with Eucalyptus minus the web based second stage installer, and have problems with in the past. It is also based off of Ubuntu right now as well. However, this mode of delivery seems to me not for scale for large deployment.
Finally what have Canonical been working on, well its called Orchestra. The idea behind Orchestra is given a number of nodes in a deployment, deploy nova, swift, and glance easily. The tools that we use are the best of the open source technologies, so in Orchcestra we use cobbler to deploy the base operating system and ensemble to deploy Openstack. Orchestra has other features such as monitoring your deployment using Nagios, and a central rsyslog to have all your log files in one place.
Right now, Orchestra is still undergoing a high pace of development and I am quite excited to see where it is going and being the premier tool for deploying Openstack on Ubuntu.
Last week in Budapest, Hungary the Ubuntu community had our semi-annual Ubuntu Developer summit. I was there along with both sponsored Ubuntu developers, upstreams, and canonical employees participating mostly in the Server and Cloud Tracks. Here are some of the sessions that were highlights for me during the week.
- Openstack At UDS, Ubuntu Cloud was announced which is run by Openstack. Everyone has assumed that this means that UEC and Eucalyptus is going away, which is not the case in Oneiric. Both Eucalyptus and Openstack will be in “main” and both will be supported in Oneiric. One way to think of this is that we both have MySQL and Postgresql in “main” because people still want to have a choice in which database you want to use. Providing both allows users to have an easier time to deploy either MySQL or Postgresql. Users are free to use either MySQL or Postgresql, which is really the essence of FOSS. Similarly, Ubuntu users for Oneiric will have either Eucalytpus or Openstack as a choice to deploy their private cloud. The session that we had Openstack mostly resolved about what integration points that we need to do for Oneiric and how we are going to do them.
- Orchestra was a big topic at UDS. Orchestra is an all in one provisioning server to deploy Ubuntu Server. Orchestra has several moving parts which includes cobbler, puppet, mcollective, nagios, cloud-init, and ensemble. The idea behind Orchestra is that given any N number of servers running bare-metal, easily deploy Ubuntu Server on it. This is a big change since the only way you could get Ubuntu Server was through the ISO and it was designed for a single server mind. With Orchestra, we are taking the best of opensource and making it easier to deploy Ubuntu Server for large scale deployments. Most of the sessions that I attended were either directly or indirectly apart of Orchestra.
- Ubuntu Server on Arm A long time ago when I was first starting out in the computer industry. I had access to a small little server called the netwinder. It was a StrongARM desktop server that runs quietly on a desk or as a 1U. The Netwinder was basically my first exposure to the ARM architecture and it was pretty cool, however it was way ahead of its time and died when Corel got out of the Linux business and Rebel.com died. Jump to, well, now, and there is a demand for servers with low power consumption and ARM is well suited in my opinion. Ubuntu Server is getting on the ground floor and I quite envious of the Ubuntu Arm team of the work they are doing in this area.
- Ubuntu as a host or what is old is new again Along with sessions about LXC and KVM we had a session about discussing Xen. Xen in the past has not been really supported by the Ubuntu Server Team or the Ubuntu Kernel Team due to the Xen patchset not included in the mainline kernel. Since the dom0 support has been included in the mainline kernel, distros such as Ubuntu can now support it properly. We had a good discussion about Xen and what needs to be done in order for Ubuntu to be a good host for Xen. I see us carrying the Xen 4.1 hypervisor and userland tools and associated bits that is needed to make Xen work really well on Ubuntu. It was great to have Ian Jackson an upstream xen.org developer from Citrix to answer questions for us as well.
- Server SRU process review Surprisingly, this was a well attended session as well on the last day. Most of our users on Ubuntu Server use the LTS releases and they want to have a stable server. In order to support those users we usually try to cherrypick fixes from the development release and backport them to the LTS releases and believe me we do care about our LTS uses. However, this takes time and we have a long list of things that we want to fix but just have not gotten around to it yet. A couple of ideas was risen from this session include using the backports, SRU process, and PPAs. However, the idea brought up of having a dedicated SRU person on the Ubuntu server who would rotate on a fixed time period was especially interesting because it would make sense for us since we would model it on how to the security team would do it.
It was nice to see everyone again and I’m looking forward to the next six months but there is a lot of work to be done. So okay go!
Congratulations to the Openstack development team for releasing Nova 2011.2, Glance 2011.2 and Swift 1.3.0 this morning. The release notes contain a list of features that are found in the cactus release of nova, swift, and glance.
Openstack is still the new cloud on the block, it is providing lots of new cool features that users will enjoy. Also it has gotten a lot of publicity over the past couple of months in the twitter and blog sphere. For me, the features that I like the best in this release are the ability to use live migration and LXC. Live migration allows the system administrator to move VM from server to server. LXC support is also important because its a lightweight technology that is up and coming as well. The release schedule is also great because it is basically the same as Ubuntu, so the latest release of Ubuntu will also have the latest release of Openstack as well. However, like all open source projects it is going to have its growing pains which I have no doubt that they are going to work through them.
Ubuntu users can use Openstack using the upcoming Natty Narwhal release. The packages were uploaded this morning a couple of hours after the cactus release so it will be in the archive soon. It just so happens that the beta2 just happened to be released yesterday as well. So please help find bugs before we release the final version at the end of the month. If you need VMs to run openstack my suggestion is that you use the UEC images since they are compatible with Openstack as well.
But have you ever wonder if you can run the “cloud” within the “cloud”, especially on Amazon EC2. Well now you can using Ubuntu Server, Openstack, and LXC as a base. Here is how..
First you need an Amazon EC2 instance. For our little project I suggest you use the latest Ubuntu Server release called “Natty Narwhal” on EC2 running an amd64 image (m1.large) . Get the largest instance you can because its easier to use in the long run, less headaches, and less hassles.
Once you instance has booted the first thing you have to do is to make sure your apt repositories are up to date and to enable cgroups. Cgroups is a linux kernel feature to limit, account, and isolate resource groups and processes. Cgroups is needed for libvirt to run LXC. So in order to enable cgroups in your EC2 instance all you have to do is the following:
sudo mkdir /cgroups
And modify your /etc/fstab accordingly
none /cgroups cgroup cpuacct,memory,devices,cpu,freezer,blkio 0 0
Just to be on the safe side reboot your EC2 instance. After you have rebooted its time to start installing Openstack on your instance. Since we are running a cloud on a single instance you need to do the following first:
sudo apt-get install rabbitmq-server dnsmasq
We are installing rabbitmq-server because its the AMQP provider that interacts with the different components of Nova. We are also installing dnsmasq because its basically your DHCP server for you cloud setup.
So next you have to install Nova. To do this on Ubuntu you have to do the following:
sudo apt-get install python-nova nova-common nova-api nova-compute nova-network nova-objectstore nova-scheduler nova-volume python-nova.adminclient unzip euca2ools
After you have installed the packages the next thing you have to do is configure Nova for LXC. To do this you need to modify the /etc/nova/nova.conf configuration file and add the following “–libvirt_type=lxc”. This will tell the nova-compute component to run libvirt to run with lxc. Then just to make life easier, reboot the instance again.
Once the instance has come backup the next step is configure your cloud, to do that you have to do the following:
sudo nova-manage network create 10.0.0.0/8 1 64
sudo nova-manage user admin
sudo nova-manage project create
sudo nova-manage project zipfile
Once you have your cloud ready you need to run images on your cloud. Serge Hallyn has made some upstart scripts that are part of a the lxcguest package in Natty which makes it easier to run Ubuntu as a LXC container. Luckily, the UEC images found in “Natty Narwhal” have this package installed by default so the user doesn’t have to do any fiddling with their images to get it to work. To download the images you can point your favorite web browser here. Just remermber to download your images to your images to EC2.
Finally you need to upload the images to Nova’s objectstore. Its pretty easy to do:
tar -zxvf natty-server-uec-amd64.tar.gz
euca-bundle-image -i natty-server-uec-amd64.img
euca-upload-bundle -b -m /tmp/natty-server-uec-amd64.img.manifest.xml
Finally…you will want to run the instance that you just uploaded to the objectstore:
euca-add-keypair mykey > mykey.priv
chmod 600 mykey.priv
euca-run-instances -k mykey -t m1.tiny
Using the “euca-describe-instances” command you should be able to see the instance running in a couple of minutes. Then finally should be able to ssh into the instance as the root user.
On a final thought is that I would like to the thank the Nova developers for making deploying a public/private cloud pretty easy and painless to do. If you have any feedback for me then you can contact me via email. Bug reports should be opened in Launchpad under the appropriate project. Enjoy your cloud with a cloud!
A feature for Natty that the Ubuntu Server team has been adding and improving is LXC container support. Linux containers is a light weight virtualization technology described by some as “chroots on steroids”.
In the past running LXC containers need a lot of hacks in the image due to various issues with upstart. But that has changed to the awesomeness of the lxcguest package. The lxcguest package will make running Ubuntu on LXC containers a trivial task. So running scripts to modify the images is now a thing of the past. Thanks Serge.
Also a new version of libvirt was uploaded to Natty recently. This version contains numerous fixes for LXC, previously in maverick if you ran container, the container would disappear once you connect to the console. Thankfully this is also fixed in more recent versions of libvirt, thanks to the upstream maintainers for libvirt. Also to Serge again for back porting the fix to Natty.
So you are probably where this is all leading to. Well now with the trunk version Open Stack you can run LXC enabled images as an EC2 or OpenStack API instance. This is a good thing for a couple of reasons, one reason is that it allows system administrators who doesn’t have the hardware to easily start deploying clouds in their data centers. However, for the high end system administrators LXC is a low overhead and fast deployment lots of instances in their Private Cloud. For developers of the openstack community it gives Nova developers who did not want to deploy either KVM or Xen a way to test their changes to the Nova code more easily.
Next we will look at how to deploy Nova with LXC.
As I mentioned in a previous post, the server team has been working on getting cobbler into shape and usuable in the Ubuntu archive. Well that day has finally happened and cobbler has been accepted into the Ubuntu Archive. if you want to use cobbler in your data center its pretty easy to do:
sudo apt-get install cobbler cobbler-common python-cobbler cobbler-web
Grab an Ubuntu server Iso
sudo mount -o loop natty-server-amd64.iso /mnt
sudo cobbler check
And follow the instructions it gives you. This makes sure that cobbler is configured properly on your server.
Import the ISO
sudo cobbler import --name=profile_name --path=/mnt --breed=ubuntu
And essentially you are ready to deploy Ubuntu Server via netboot. Since it is brand new we users to use this software and would to hear about bugs that you may have encounter. You can file them here.
Last week the server team along with members of the Canonical OEM server team, members of the Landscape team, and the rest of the Ubuntu platform team sprinted/rallyed in Dallas. Some of you might know the hotel that we were staying at because it was the same hotel as the UDS in Dallas, the lovely Renaissance Hotel.
The rally is one of the few chances that the server team get to work in the same room and work together in the flesh. Usually we work from 9 in the morning to 6 in the night. After that we have dinner together and hang out as well, in our case a couple of us went to see a Dallas Stars hockey game and even had a Karaoke night. We usually have a team dinner, lucky for us it was at the Medieval Times. We were by far the loudest ones there.
Like a lot of the previous rallies/sprints we got a lot of stuff done, closed some of the items in the work tracker, and discussed future ideas. Some of the highlights were the following:
- Clusters and power saving Andres Rodriguez was focusing on the cluster stack in the community and recently started working at Canonical working on the cluster stack. He continued his great work beating Natty’s clustering story into a good shape. A lot of time and work has been put into making it rock and it will continue. I dont have any doubt about that.
- Hudson^WJenkins With the fiasco with the Hudson project and Oracle recently in the news, the server team has been using Jenkins in automating the ISO testing and eventually our UEC images. In order for our users to easily use Jenkins it needs to be packaged and be in the Ubuntu Archive. James Page has been doing an admirable job packaging the tonnes and tonnes of Java dependency it needs to be in the archive at all. Also he has taken care of other Java applications such as tomcat and Ant as well.
- LXC For a while now KVM has been the defacto virtualization choice for Ubuntu Server sinn ce before Hardy. We have been focusing on LXC a bit more because its a lightweight virtualization technology that low end servers can use. Serge Hallyn has been beating it into shape. We plan on introducing LXC into Openstack in the near future, and the ground work has been done to contribute the feature back to the Openstack upstream team.
- UEC/EC2 Images I read somewhere that Ubuntu Server is one of the most used Linux distros on Amazon EC2, which is really really cool. Now that amazon adopting is cloud-init for their own images it makes even more cooler. A lot of nice new features are coming down the pipe for Natty as well. I don’t want to steal any of Scott Moser’s thunder, since you can read all about it at his own blog.
- The futre… What the server team is really excited about (I think) is to provide an infrastructure to deploy Ubuntu easily for large scale deployments. We are keeping in mind both cloud deployments and bare-metal regular server deployments. In order to do this we have been looking at technologies like Puppet, cobbler, and collectd, a best of breed if you will. We are starting small and building our way up to the next LTS release of Ubuntu. With that in mind, we have been working on Cobbler. We have been re-adding support for both Ubuntu and Debian and have pushing patches back upstream to cobbler proper so others can benefit from our work. In the last week at the rally we were able to deploy a tftp PXE server using cobbler and a regular Ubuntu Server ISO. Cobbler should be available in the archive in the next couple of days and I will blog on how to do a basic import of the Ubuntu Server ISO. Going forward we would like to make things like Puppet really easy to use and convenient to the Ubuntu Server system administrator, but like I said before we are starting small and still finding our way.