nova-compute-flex: Introduction and getting started

What is nova-compute-flex?

For the past couple of months I have been working on the OpenStack PoC called nova-compute-flex. Nova-compute-flex allows you to run native LXC containers using the python-lxc calls to liblxc. It creates small, fast, and reliable LXC containers on OpenStack. The main features of nova-compute-flex are the following:

  • Secure by default (unprivileged containers, apparmor, etc)
  • LXC 1.0.x
  • python-lxc (python2 version)
  • Uses btrfs for instance creation.

Nova-compute-flex (n-c-flex) is a new way of running native LXC containers on OpenStack. It is currently designed with Juno in mind since Juno is the latest release of OpenStack.  This tutorial to get nova-compute-flex up and running assumes that you will be using Ubuntu 14.04 release and will be running devstack with it.

How does n-c-flex work?

N-c-flex works the same way as the other virt drivers in OpenStack. It will stop and start containers,  use neutron for networking, etc. However it does not use qcow2 or raw images, it uses an image that we call “root-tar”.

“Root-tar” images are simply a tarball of the container which is similar to the ubuntu-cloud templates in LXC. They are relatively small and contain just enough to get you running a LXC container. These images are published by Ubuntu as well, and they can be found here.. If you wish to use other distros you can simply tar up the directories found on a given qcow2 image. As well as you could use the templates found in LXC. Its just that simple.

The way that nova-compute-flex works is the following:

  1. Download the tar ball from the glance server.
  2. Create a btrfs snapshot.
  3. Use lxc-usernsexec to un-tar the tar ball into the snapshot.
  4. When the instance starts create a copy of the snapshot.
  5. Create the LXC configuration files.
  6. Create the network for the container.
  7. Start the container.

It just takes seconds to create a new instance since it is just doing a copy of the btrfs snapshot when the image was downloaded from the glance server.

When the instance is created, the container is an unprivileged LXC container. This means that nova-compute-flex uses user-name-spaces with apparmor built in (if you are using Ubuntu).  The instance behaves like a container, but it looks and feels like a normal OpenStack instance.

Getting Started with n-c-flex

Assuming that you already have btrfs-tools is installed and you don’t have a free partition. You will need to create the instances directory where your n-c-flex instances are going to live. To do that you simply have to do the following:


dd if=/dev/zero of=<name of your large file> bs=1024k count=2000
sudo mkfs.btrfs <name of your large file>
sudo mount -t brfs -o user_subvol_rm_allowed <name of your large file> <mount point>

To make the changes permanent, modify your /etc/fstab accordingly.

Installing devstack and n-c-flex

In your “/opt” directory,  run the following commands:


mkdir -p /opt/stack
git clone https://github.com/zulcss/nova-compute-flex
git clone https://github.com/openstack-dev/devstack
/opt/stack/nova-compute-flex/contrib/devstack/prepare_devstack.sh

This will prepare your devstack to install software like LXC that been back ported to the Ubuntu Cloud Archive. The reason for the back port is that  some of the features that is needed in nova-compute-flex is not
found in the trusty version of LXC

After running the above commands you will have the following in your localrc:

virt_type=flex

To make your devstack more useful you should have the following in your localrc as well:


disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-dhcp
enable_service q-l3
enable_service q-meta

GIT_BASE=https://github.com
DATA_DIR=<mount point>

ADMIN_PASSWORD=devstack
MYSQL_PASSWORD=devstack
RABBIT_PASSWORD=devstack
SERVICE_PASSWORD=devstack
SERVICE_TOKEN=token

NOVA_BRANCH=stable/juno
CINDER_BRANCH=stable/juno
GLANCE_BRANCH=stable/juno
HORIZON_BRANCH=stable/juno
KEYSTONE_BRANCH=stable/juno
NEUTRON_BRANCH=stable/juno

This will allow you to use the stable branches of juno with neutron support. After modifying your localrc, you can proceed to install by running the “./stack.sh” script.

Running your first instance

As said before nova-compute-flex uses a different kind of image compared to regular nova. To upload the image to the glance server you have to do the following:


source openrc
wget http://cloud-images.ubuntu.com/utopic/current/utopic-server-cloudimg-amd64-root.tar.gz
glance image-create --name='lxc' --container-format=root-tar --disk-format=root-tar < utopic-server-cloudimg-amd64-root.tar.gz

After uploading the image you can run the regular way of creating instances by either using the python-novaclient tools or the euca2ools.

Looking forward

At the OpenStack Developrs Summit last week, Mark Shutleworth announced lxd (lex-dee). LXD is a container “hypervisor” that is built on top of the LXC project. LXD is meant to be used as system container, rather than application containers like Docker.

I will be using the knowledge that we have gained from working on  nova-compute-flex and applying it nova-compute-lxd. LXD will have a Rest-API to interact with the LXD containers, so nova-compute-lxd will be the lxd api to stop/start containers and other functions one expects to find in Nova. More discussion will be going on at the lxc-devel mailing list over the next couple of months.

However, if you want to use nova-compute-flex now go for it! If you wish to submit patches, the github project can be found at https://github.com/zulcss/nova-compute-flex. The work will be fed back into the nova-compute-lxd project as well.  It also has an issue tracker where you can submit bugs as well.

If you run into road blocks please let me know, and I will be happy to help.

November 14, 2014. Uncategorized.

Leave a Comment

Be the first to comment!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Trackback URI

%d bloggers like this: