Building your own Crowbar ISO for OpenStack Havana

Crowbar, a great cloud unboxer, is currently evolving at a rapid pace, if you want to see the latest and greatest thing without waiting any longer, you can build your own Crowbar ISO. In this article we’ll show you how to do just that using the Roxy branch which is supposed to support OpenStack Havanna. We will suppose you aren’t planning to contribute to the code, so we won’t use our any personalized Git repository. If you don’t know what’s Crowbar, it’s platform for server provisioning and deployment from bare metal. But if you want to see how it could be used to deploy OpenStack, read our previous article.

This article is based on the offical documentation from the Crowbar team plus some tips & tricks.

Build Machine

Let’s first install all the prerequisites on our build machine.

You first need a fresh Ubuntu 12.04.02 OS installed.

SSH to that box and follow the steps below.

Make sure to enable password-less sudo:

sudo sed -ie "s/%sudo\tALL=(ALL:ALL) ALL/%sudo ALL=(ALL) NOPASSWD: ALL/g" /etc/sudoers

Note: This is required because the build process will mount ISO which require root access but you don’t want to stay close to the installation to enter it when required.
Install required package with

sudo apt-get update
sudo apt-get install \
    git rpm ruby rubygems1.9 curl build-essential debootstrap \
    mkisofs binutils markdown erlang debhelper python-pip \
    build-essential libssl-dev zlib1g-dev \
    libpq-dev byobu cabextract ruby1.9.1-dev

If you plan to install the lastest developement branch also install

sudo apt-get install libsqlite-dev libsqlite-dev libsqlite-dev

Now make sure you use the latest Ruby (1.9.1) version with

sudo update-alternatives --config ruby

PostgreSQL 9.3 installation

Crowbar is now using PostgreSQL 9.3 which isn’t part of Ubuntu 12.04 so you need to remove the older Ubuntu version

sudo apt-get remove postgresql

Add a new repository key

wget --quiet -O - | sudo apt-key add -

determine your OS codename

lsb_release -c

Create the following file /etc/apt/sources.list.d/pgdg.list with the following content and update apt database

deb <codename>-pgdg main

Now you can run

apt-get update

Note: The codename should be the one from the previous command.

Intall the newer version from this new repository

sudo apt-get install postgresql-9.3 pgadmin3

Configure PostgreSQL

sudo vi /etc/postgresql/9.3/main/pg_hba.conf

Add the following section at the beginning of the following file /etc/postgresql/9.3/main/pg_hba.conf

local  all   all    trust

And change Port like this in /etc/postgresql/9.3/main/postgresql.conf

port = 5439

Start the service

sudo service postgresql restart

Now create a Crowbar user

sudo createuser -s -d -U postgres -p 5439 crowbar

Test PostgreSQL installation

psql postgresql://crowbar@:5439/template1 -c 'select true;'

Gems installation

Install required Gems

sudo gem install builder bluecloth
sudo gem install json net-http-digest_auth kwalify bundler delayed_job delayed_job_active_record rake rspec pg --no-ri --no-rdoc
sudo gem install rcov -v 0.9.11 --no-ri --no-rdoc

Ubuntu 12.04 ISO

Create repository for Ubuntu ISO

mkdir -p ~/.crowbar-build-cache/iso
cd ~/.crowbar-build-cache/iso

You now have to download Ubuntu–12.04-server ISO to it


Crowbar repository

You currently have all the pre-requisites to start the building process. The first step is to clone the Crowbar Git Repository

cd ~
git clone
cd ~/crowbar

Now do the following, make sure you append --no-github if you don’t want to submit any pull-requests

./dev setup --no-github
./dev fetch --no-github

Build Sledgehammer ISO

Sledgehammer is a bare OS by default based on CentOS 6.2 used to bootstrap Nodes during the discovery process, build it once with

cd ~/crowbar
sudo ./

It takes some time but you won’t have to repeat this process for each Crowbar ISO Build fortunately. It should terminate with the following message:

Your pxeboot image is complete.

Copy tftpboot/ subdirectory to /tftpboot or a subdirectory of /tftpboot.
Set up your DHCP, TFTP and PXE server to serve /tftpboot/.../pxeboot.0

Note: The initrd image contains the whole CD ISO and is consequently
very large.  You will notice when pxebooting that initrd can take a
long time to download.  This is normal behaviour.

Configure Crowbar builder

Almost there, now list the different release available to you

./dev releases

Show all the builds

./dev builds

Select what you want to build for example

./dev switch development/master

Display your current release and build

./dev release
./dev build

As of now mesa-1.6/openstack-os-build is the most stable OpenStack release based on 1.6 (mesa). The current Crowbar 2.0 branch (master) doesn’t support OpenStack yet.


If you’re using a Cloud hosted VM to build Crowbar, I would recommend to start Byobu like that :


In case you are disconnected from it, building will still continue, and you’ll be able to reconnect to it once being reconnected to your instance.

Now for fun let’s build roxy/openstack-os-build which was introduced to support Havana, Pebbles is the previous one which support Grizzly. So we do:

./dev switch roxy/openstack-os-build
./dev build --os ubuntu-12.04 --update-cache

If you can strange README.empty-branch files, you can run the following command to solve that problem

cd ~/crowbar/barclamps
for bc in *; do (cd "$bc"; git clean -f -x -d 1>/dev/null 2>&1; git reset --hard 1>/dev/null 2>&1); done

You now have to be really patient, building Crowbar is a long process, lots of things to download, it took me around 80 minutes, so you can detach from the Byobu session using F6 and do other stuff. When you’ll come back in your builder node, you’ll see what’s going on and if you’re lucky it will be done with the following message.

2013-11-02 17:04:54 -0700: Copying over Sledgehammer bits
2013-11-02 17:04:55 -0700: Creating new ISO
2013-11-02 17:05:51 -0700: Image at /home/user/crowbar/
2013-11-02 17:05:51 -0700: Finished.

You can now transfer around the ISO to boot a Crowbar admin node

scp user@crowbar-builder:/home/user/crowbar/ .

It’s a 2Gb transfer so if it fails you can resume it with

rsync --partial --progress --rsh=user@crowbar-builder:/home/user/crowbar/ .

Or you can also publish the crowbar directory over HTTP using Ruby like this

ruby -run   -e httpd -- -p 3333 ~/crowbar

Now connect to your client machine and download it with


If the transfert fails you can resume it using the -c wget option

wget -c

Knife Configuration

Current Crowbar Roxy version embbed Chef Server v10.18.2. You can interact with the Chef API using the Knife command line interface, but your first have to create an admin user from Chef Web UI accesible at It will give you a private key that you have to save to ~/.chef/admin.pem. You’ll also need to copy the /etc/chef/validation.pem from the crowbar admin to ~/.chef/validation.pem on your client node.

Now, to create a ~/.chef/knife.rb configuration file, use the following command:

knife configure -i \
  -u <user> \
  -s \
  -r /home/<user>/chef-repo \
  --admin-client-name admin \
  --admin-client-key ~/.chef/admin.pem \
  --validation-key ~/.chef/validation.pem \
  --validation-client-name chef-validator

-u                           new API user to create
-s                           API endpoint
-r                           The path to your chef-repo
--admin-client-name          existing API user
--admin-client-key           existing user private key
--validation-client-name     The existing validation clientname

Building a Crowbar OpenStack Image

If you plan to deploy a Crowbar Admin node within an OpenStack cloud, you first need to generate a QCOW image. Install the some required tools

sudo apt-get install kvm-qemu
pip install python-glanceclient

Now generate a Raw disk

kvm-img create -f raw crowbar.img G9

Now you can install Crowbar within this image

sudo kvm -m 1024 -cdrom \
    -drive file=crowbar.img,if=virtio,index=0\
    -boot d -net nic -net user -nographic -vnc :0 \
    -usbdevice tablet

To look at the installation of Crowbar connect to your node IP using VNC and wait until it terminate. I still have to debug the last message saying no root partition.


Compared to the official docs, here is what what necessary on Ubuntu 12.04

  • libopenssl-ruby1.9 wasn’t found so I bypassed the installation of this package.
  • ruby1.9.3-dev no gems with this name but documentation says it’s ok on Ubuntu12.04, we installed instead ruby1.9.1-dev which is required by json gem.
  • gem install pg require installation of libpq-dev Ubuntu package, or postgresql-devel on Redhat
  • gem install rcov -v 0.9.11 Necessary to force version here because 1.0 isn’t supported on current Ruby 1.9
  • sudo apt-get install cabextract was necessary or build would fail.
  • ~/.crowbar-build-cache/barclamps/dell_raid/files/dell_raid/tools d/l file below inside that directory
    • wget
    • wget
  • sometimes the provisionning steps enters an infinite loop with message saying waiting for pxe file to contain: .*_install ..... to exit the loop just run chef-client on the admin node or restart the tftpd process.
  • When I applied first the Nova barclamp, it failed I had to manually start OpenvSwitch like this start openvswitch-switch and re-run chef-client on the node.
  • Roxy already support Docker, see docs for details.


It’s great to be able to build your own custom Crowbar ISO, you can then follow the development as it happens. Once you boot a Node using your shiny new ISO that you just built, login/password is crowbar/crowbar. You can then run the following command to install your crowbar admin node, but you should review our previous article before going any further.

sudo /tftpboot/ubuntu_dvd/extra/install <NODE_FQDN>

It’s then detached from the terminal, so to track the progress run

sudo screen -r -S crowbar-install

You can also look at

tail -f /var/log/install.log 

Once the Admin is ready, boot other node and start applying proposals from Crowbar WebUI

or like this from the command line

/opt/dell/bin/crowbar machines allocate
/opt/dell/bin/crowbar mysql proposal create proposal 
/opt/dell/bin/crowbar mysql proposal show proposal > mysql.json

edit the json file (put the host in the appropriate elements element)

"elements": {
  "mysql-server": [

Save it

/opt/dell/bin/crowbar mysql proposal edit proposal  --file mysql.json
/opt/dell/bin/crowbar mysql proposal commit proposal 

Do the same for other proposals: keystone, glance, cinder, neutron, nova, nova_dashboard.

Watch proposal deployment progress

tail -f /var/log/syslog 

You can start over with


For more details consult our previous article

Being near trunk, there is a a non null probability that the installation can fails, so consult /var/log/crowbar/install to troubleshoot. If the first failed to avoid a current bug in the Roxy release, re-provision your Admin node from your ISO, re-running the install-chef command will bring strange errors with deb-deb-deb and rack.rb not found !!!

Crowbar Admin node IP address is static and will be set to So If it worked you should then be abble to access crowbar web interface at