VMware NSX 4.x on RHEL OpenStack Platform 5.0

Redhat announced a month ago OpenStack Platform 5.0 which officially support VMware NSX 6.0 and vSphere 5.5. In our lab today, we’ll deploy this OpenStack distribution and connect it to NSX and vSphere to see how easy it is and if everything works as expected. By the way it’s not, so beware, this article is just a preview of what’s coming. If you want to reproduce the same setup, you’ll have to wait until VMware officially support RedHat 7.0. This article will be updated accordingly when it will be publicly available.

Hardware/Software requirement

You’ll need at least 3 physical servers totalling (16Gb). You can choose to install everything on Bare Metal but you’ll then need more servers, over here I’ll deploy everything on a vSphere 5.5 infrastructure to reduce the footprint. Please note that NSX Controllers is only supported on bare metal for production environments.

If you don’t have a NSX Environment up and running yet with NSX Controllers, Manager, Service node and Gateway, you can consult my NSX 4.0 component installation article.

To get RHEL OpenStack Platform 5.0, you can request a 60 days evaluation. You’ll get access to RHEL 7.0 ISO (3.7GB) with an entitlement for their OpenStack Platform.

For the OpenStack cloud controller the requirements are

64-bit x86 with Intel 64 or AMD64 CPU extensions with Intel VT or AMD-V HW Virtualization extensions enabled
2 GB RAM
50 GB HD (but 1 TB would be more realistic)
2 x 1 Gbps Nics.

If you want to install a nested RHEL 7.0, you can check if your vSphere platform will support it by accessing the following URL

https://[your-esxi-host-ip-address]/mob/?moid=ha-host&doPath=capability

Authenticate using your root credentials and look for nestedHVSupported, the value should be True. Don’t forget to enable this feature for your VM by checking the following checkbox in the VM setting.

Expose hardware assisted virtualization to the guest OS

RHEL 7.0 installation, entitlement and channels

The first step of our lab is to install Redhat Enterprise Linux 7.0. Just boot the ISO and follow the installation wizard. Make sure you have at least 2 Nics Cards available. One will be connected to the NSX Transport Network, the other one to the Uplink.

When you get a root shell access, first check that your system has the required CPU extensions for virtualization.

# grep -E 'svm|vmx' /proc/cpuinfo | grep nx

# subscription-manager register

Enter your Red Hat Customer Portal user name and password you used when you’ve requested an OpenStack Platform Evaluation. You should get :

The system has been registered with id: IDENTIFIER

Attach Pool IDs to your subscription

# subscription-manager list --available

Take note of the Pool ID Identifier for the 60 Day Self-Supported Red Hat Enterprise Linux OpenStack Platform Preview and replace it in the commands below

# subscription-manager attach --pool=<POOLID>

Now subscribe to the following channels

# subscription-manager repos --enable=rhel-7-server-rh-common-rpms
# subscription-manager repos --enable=rhel-7-server-rpms
# subscription-manager repos --enable=rhel-7-server-openstack-5.0-rpms

To update the repository configuration type

# yum repolist

Update your system and reboot

# yum update -y

OpenStack doesn’t like Red Hat Network Manager, check if it is enabled and disable it if necessary.

# systemctl status NetworkManager.service | grep Active:
# systemctl stop NetworkManager.service
# systemctl disable NetworkManager.service

Edit your interface configuration file in /etc/sysconfig/network-scripts/ to add the following configuration key

NM_CONTROLLED=no

Ensure that the standard network service is started and enabled

systemctl enable network.service

Reboot

# reboot

Deploying OpenStack using PackStack

PackStack uses Puppet modules to easily deploy OpenStack but it’s not supported for production environment which is fine here. So we’ll use it here to avoid doing all the install & configuration manually, install it.

yum install -y openstack-packstack

For the VMware integration part, we need to customize the default configuration. Lets generate an answer file

packstack --gen-answer-file lab.txt

This file is used when bypassing the interactive mode to give all the parameters to PackStack for an unattended installation. You can now deactivate the services you don’t need like

CONFIG_SWIFT_INSTALL=n
CONFIG_CEILOMETER_INSTALL=n

But the most important config is the one related to Neutron. Make sure you comment out all reference to L3 or ML2 stuff.

Optionally to consume vSphere resources from OpenStack update the following section with relevant information:

CONFIG_VMWARE_BACKEND=y

# The IP address of the VMware vCenter server
CONFIG_VCENTER_HOST=<vCENTER_IP>

# The username to authenticate to VMware vCenter server
CONFIG_VCENTER_USER=<vCENTER_USER>

# The password to authenticate to VMware vCenter server
CONFIG_VCENTER_PASSWORD=<vCENTER_PWD>

# The name of the vCenter cluster
CONFIG_VCENTER_CLUSTER_NAME=<vCENTER_CLUSTER>

That’s all we need, networking features will be provided by NSX. We can now launch the OpenStack installation

packstack --answer-file lab.txt

If you’re luck you should then see

**** Installation completed successfully ******


Additional information:
 * Did not create a cinder volume group, one already existed
 * File /root/keystonerc_admin has been created on OpenStack client host 192.168.1.15. To use the command line tools you    need to source the file.
 * To access the OpenStack Dashboard browse to http://192.168.1.15/dashboard .
Please, find your login credentials stored in the keystonerc_admin in your home directory.
 * Because of the kernel update the host 192.168.1.15 requires reboot.
 * The installation log file is available at: /var/tmp/packstack/20140816-184004-yhiqR4/openstack-setup.log
 * The generated manifests are available at: /var/tmp/packstack/20140816-184004-yhiqR4/manifests

Your host now need a reboot

reboot.

Open vSwitch SSL Configuration

We need to configure SSL for Open vSwitch :/ If we were using VMware Open vSwitch binaries it would have been configured at install time, but as we’ve said in the introduction, it’s not yet available. So, because we’ll be using the Red Hat ones we have to do it ourselves.

First we have to determine the directory PKI Infrastructure, look for --dir option in the output of.

ovs-pki --help

On Red Hat 7, it’s located in /var/lib/openvswitch/pki. There are two PKI strategies for Open vSwitch.

Self Signed Certificates
Switch Certificate Authority

To Create and populate a new PKI directory

ovs-pki init --force

Make sure you secure the following files which contains sensitive informations: pki/controllerca/private/cakey.pem and pki/switchca/private/cakey.pem

Now you can create self signed certificate like this

ovs-pki req sc
ovs-pki self-sign sc

You are now ready to configure Open vSwitch to use the certificate /var/lib/openvswitch/pki/sc-cert.pem produced by the previous command. Before you can type the command below you just need to create /var/lib/openvswitch/pki/cacert.pem with the Certificate you’ll find on the NSX Manager > Cluster > Switch > PEM menu.

To finish the Open vSwitch configuration type:

ovs-vsctl set-ssl /var/lib/openvswitch/pki/sc-privkey.pem /var/lib/openvswitch/pki/sc-cert.pem /var/lib/openvswitch/pki/cacert.pem

NSX Plugin Installation and Configuration

We need to setup the foundation for network virtualisation which will leverage the Open vSwitch. Start by creating a Bridge for each of your Physical Nic like this:

ovs-vsctl add-br br0
ovs-vsctl br-set-external-id br0 bridge-id br0
ovs-vsctl set Bridge br0 fail-mode=standalone
ovs-vsctl add-port br0 eno

Replace en0 by the physical Nic connected to the transport network.

Now create a /etc/sysconfig/network-scripts/ifcfg-br0 config file and move over the IP address to it.

root@pc42 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
ONBOOT=yes
BOOTPROTO=none
IPADDR=192.168.1.15
NETMASK=255.255.255.0
GATEWAY=192.168.1.1

[root@pc42 network-scripts]# cat /etc/sysconfig/network-scripts/ifcfg-eno16780032
DEVICE=eno16780032
ONBOOT=yes
BOOTPROTO=none
PROMISC=yes

Restart the network stack and check everything looks good

service network restart
ip addr show

Now Instead of installing the NSX Plugin RPM from VMware, lets install the one from Red Hat to avoid any dependency hells.

yum install openstack-neutron-vmware.noarch

ovs-vsctl set-manager ssl:192.168.1.11

Now check the ovsdb-server log file to see if the connection to the Controller is successfull

[root@localhost openvswitch]# tail -f ovsdb-server.log
2014-08-17T15:59:22.367Z|00014|stream_ssl|ERR|ssl:192.168.1.11:6632: connect: Permission denied
2014-08-17T15:59:22.367Z|00015|reconnect|INFO|ssl:192.168.1.11: connecting...
2014-08-17T15:59:22.367Z|00016|reconnect|WARN|ssl:192.168.1.11: connection attempt failed (Permission denied)
2014-08-17T15:59:22.367Z|00017|reconnect|INFO|ssl:192.168.1.11: waiting 4 seconds before reconnect

The permission denied is caused by SELinux, so lets deactivate it for now. Edit SELINUX=disabled with

SELINUX=disabled

Check again after a reboot

[root@localhost brauns]# tail -f /var/log/openvswitch/ovsdb-server.log
2014-08-17T16:07:40.972Z|00162|reconnect|WARN|ssl:192.168.1.11: connection attempt failed (Protocol not available)
2014-08-17T16:07:40.972Z|00163|reconnect|INFO|ssl:192.168.1.11: waiting 8 seconds before reconnect
2014-08-17T16:07:48.972Z|00164|stream_ssl|ERR|Certificate must be configured to use SSL

If you get like above a message asking to configure SSL, go back to the SSL section above and check you’ve done all the steps. If the connection works you should get

2014-08-17T17:01:08.836Z|00167|reconnect|INFO|ssl:192.168.1.11:6632: connecting...
2014-08-17T17:01:08.855Z|00168|reconnect|INFO|ssl:192.168.1.11:6632: connected

Now if you check the NSX-Manager logs, here is what you’ll find.

Unsupported transport node neutron-server at 192.168.1.15:43702: OVS version 2.0.0 does not meet minimum version 2.1

That explain why even if your Open vSwitch is connected, Management and OpenFlow status is reported as down because of Open vSwitch version mismatch. Again for this to work we need a newer version of Open vSwitch then the one provided by the current Redhat 7.0.

OpenStack IceHouse integration

We now need to tell our Open Stack Neutron node to use NSX Plugin.

Edit neutron configuration /etc/neutron/neutron.conf to edit the following line

core_plugin = neutron.plugins.vmware.plugin.NsxPlugin
quota_driver = neutron.db.quota_db.DbQuotaDriver

The last line allows Neutron to have per tenant quota.

Now you need to configure the plugin itself in /etc/neutron/plugins/vmware/nsx.ini

nsx_user = admin
nsx_password = admin
nsx_controllers = 192.168.1.11
default_tz_uuid = <ID>
default_l3_gw_service_uuid = <ID>
metadata_mode = access_network
[database]
connection = mysql://neutron:6a42866dce274070@192.168.1.15/neutron

Note:

You can get UUID for transport zone and gateway service from NSX Manager.
The last database needs to be created, you’ll find the DB URL within /etc/neutron/neutron.conf

Before restarting neutron server, it’s a good idea to check the configuration file

neutron-check-nsx-config /etc/neutron/plugins/vmware/nsx.ini

Each OpenStack distribution have some specific stuff to care about, Red Hat expect the plugin configuration in /etc/neutron/plugin.ini, so create the following symbolic link

ln -s /etc/neutron/plugins/vmware/nsx.ini /etc/neutron/plugin.ini

If everything looks good restart neutron server

service neutron-server restart

If neutron refuse to start, use journalctl -xn to get all the details and check again your nsx.ini file before retrying.

Configure Neutron DHCP Agent

VMware NSX rely on Neutron DHCP Agent to provide dynamic addresses to VMs. Configure /etc/neutron/dhcp_agent.ini with

[DEFAULT]
enable_metadata_network = True
enable_isolated_metadata = True
ovs_use_veth = True
use_namespaces = True
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq

And start it

# service neutron-dhcp-agent start

Configure Neutron Metadata Agent

This agent is responsible for providing connectivity between instances and the Nova metadata service used to customize instances appropriately. Lets configure it.

Edit /etc/neutron/metadata_agent.ini and ensure the following settings:

[DEFAULT]
# Show debugging output in log (sets DEBUG log level output) # debug = False
# The Neutron user information for accessing the Neutron API. auth_url = http://<keystone-host>:35357/v2.0
auth_region = RegionOne
admin_tenant_name = services
admin_user = neutron
admin_password = <NEUTRON_PASSWORD>
# IP address used by Nova metadata server nova_metadata_ip = <ip of nova-metadata-server>
# TCP Port used by Nova metadata server
# nova_metadata_port = 8775
# When proxying metadata requests, neutron
# signs the Instance-ID header with a
# shared secret to prevent spoofing.
# You may select any string for a secret,
# but it must match here and in the
# configuration used by the Nova Metadata # Server. NOTE: Nova uses a different key: # neutron_metadata_proxy_shared_secret
# metadata_proxy_shared_secret =

Configure Nova

You’ll need to check the /etc/nova/nova.conf on all nodes running nova service (nova-api, nova-scheduler, nova-compute) for the following lines (should be fine)

neutron_admin_username=<username>
neutron_admin_password=<password>
neutron_admin_auth_url=http://<keystone host>:35357/v2.0
neutron_auth_strategy=keystone
neutron_admin_tenant_name=<tenant-name>
neutron_url=http://<neutron-server>:9696
network_api_class=nova.network.neutronv2.api.API
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
firewall_driver=nova.virt.firewall.NoopFirewallDriver
security_group_api=neutron
service_neutron_metadata_proxy = True

For KVM nodes also add

libvirt_ovs_bridge=<integration bridge used by NVP>
libvirt_type=<libvirt type, e.g. xen>
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

For vSphere nodes, apart from requiring NSX vSwitch with nsxcli --allow-custom-vifs to give OpenStack-generated UUID not the vSphere ones to the NSX Controller, nova-compute nodes that leverage ESXi 5.5 hypervisors only require the following lines in /etc/nova/nova.conf (as we’ve seen earlier). The logic used to bind VMs interfaces to the NSX vSwitch is part of the native VC Driver.

compute_driver=vmwareapi.VMwareVCDriver

Restart nova services

# cd /etc/init.d/; for i in $( ls nova-* ); do service $i restart; done

Check services are running

# /etc/init.d/; for i in $( ls nova-* ); do service $i status; done

All nodes running nova-compute need to have the following lines in /etc/libvirt/qemu.conf

cgroup_device_acl = [
"/dev/null", "/dev/full", "/dev/zero", "/dev/random", "/dev/urandom", "/dev/ptmx", "/dev/kvm", "/dev/kqemu", "/dev/rtc",    "/dev/hpet","/dev/net/tun",
]

Add them if necessary and run

service qemu-kvm restart
service libvirt-bin restart

Testing

Here is a simple process to check that everything works

source /root/keystonerc_demo
neutron net-create net1
neutron subnet-create net1 10.0.0.0/24
nova boot --image <img> --flavor <flavor> --nic net-id=<net-id> <vm-name>

Now assign a floating IP to your VM and try to connect to it, this is your big reward ;)

ssh user@floating-ip

Add Neutron as a Transport Node

Still a work in Progress.

Appendix

Open vSwitch from source tarball

Here are the steps involved to build Open vSwitch RPMs for Red Hat.

First install dependencies

yum install gcc make python-devel openssl-devel kernel-devel graphviz \
    kernel-debug-devel autoconf automake rpm-build redhat-rpm-config libtool

Download and untar the source tarball put a copy of the tarball in /root/rpmbuild/SOURCES as openvswitch-2.1.0.34867.tar.gz. Update the version number accordingly.

To build Open vSwitch userspace run:

rpmbuild -bb rhel/openvswitch.spec

You’ll get the following two artefacts: /root/rpmbuild/RPMS/x86_64/openvswitch-2.1.0.34867-1.x86_64.rpm /root/rpmbuild/RPMS/x86_64/openvswitch-debuginfo-2.1.0.34867-1.x86_64.rpm

To buil Open vSwitch kernel module run:

cp rhel/openvswitch-kmod.files /root/rpmbuild/SOURCES
rpmbuild -bb rhel/openvswitch-kmod-rhel6.spec

You should get a kmod-openvswitch RPM or an Error :( which was my case.

Conclusion

Wow, what a journey. It’s not trivial isn’t it. There is room for improvement. For example, Packstack should be able to configure the NSX Plugin and do all the tweaking without requiring the admin to do all by himself. On the positive side, this version already integrate nicely with the VMware vSphere Driver.

Don’t forget to do your homework, you’ll need to prepare your ESXi hosts with the NSX vSwitch and connect them to the NSX Controller. If you try to deploy instances using this driver without NSX, you’ll get an error message from the Nova Scheduler saying he can’t find any compatible hosts, he’s looking for hosts that have the NSX bridge configured. One of my VMware coworker based in Spain published a nice article which details how to setup an ESXi to integrate with NSX.

Apart that I find Red Hat OpenStack platform 5.0 practical for unattended installation. Most of the issues happen when the installation fails in the middle, it’s leaving the environment kind of dirty, so if you re-run it without cleaning what Packstack left behind it can add another layer of failure ! And for the cleaning trick see my troubleshooting note below. A good advice is to keep a clean RedHat 7.0 snapshot where you can revert to in case of failure.

I hope the Red Hat diclaimer which says Packstack isn’t supported for production environment will disapear soon. After all it’s Puppet which is used by thousands of customers around the globe. Configuring everything by hand is a lot more risky than automating everything.

Troubleshooting

If in the previous run, you got the following error

Error: mysqladmin -u root  password '4705ed178f714e28' returned 1 instead of one of [0]

Just remove /root/.my.cnf (an artefact of a previous failed run) and re-run packstack again.

To clean everything run (beware it will delete all)

for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do virsh destroy $x ; virsh undefine $x ; done ; yum remove -y nrpe "*nagios*" puppet "*ntp*" "*openstack*" "*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" mysql mysql-server httpd "*memcache*" scsi-target-utils iscsi-initiator-utils perl-DBI perl-DBD-MySQL ; ps -ef | grep -i repli | grep swift | awk '{print $2}' | xargs kill ; rm -rf /etc/nagios /etc/yum.repos.d/packstack_* /root/.my.cnf /var/lib/mysql/ /var/lib/nova /etc/nova /etc/swift /srv/node/device*/* /var/lib/cinder/ /etc/rsync.d/frag* /var/cache/swift /var/log/keystone ; umount /srv/node/device* ; killall -9 dnsmasq tgtd httpd ; setenforce 1 ; vgremove -f cinder-volumes ; losetup -a | sed -e 's/:.*//g' | xargs losetup -d ; find /etc/pki/tls -name "ssl_ps*" | xargs rm -rf

# Warning! Dangerous step! Destroys VMs
for x in $(virsh list --all | grep instance- | awk '{print $2}') ; do
    virsh destroy $x ;
    virsh undefine $x ;
done ;

# Warning! Dangerous step! Removes lots of packages, including many
# which may be unrelated to RDO.
yum remove -y nrpe "*nagios*" puppet ntp ntp-perl ntpdate "*openstack*" \
"*nova*" "*keystone*" "*glance*" "*cinder*" "*swift*" \
mysql mysql-server httpd "*memcache*" scsi-target-utils \
iscsi-initiator-utils perl-DBI perl-DBD-MySQL ;

ps -ef | grep -i repli | grep swift | awk '{print $2}' | xargs kill ;

# Warning! Dangerous step! Deletes local application data
rm -rf /etc/nagios /etc/yum.repos.d/packstack_* /root/.my.cnf \
/var/lib/mysql/ /var/lib/glance /var/lib/nova /etc/nova /etc/swift \
/srv/node/device*/* /var/lib/cinder/ /etc/rsync.d/frag* \
/var/cache/swift /var/log/keystone ;

umount /srv/node/device* ;
killall -9 dnsmasq tgtd httpd ;
setenforce 1 ;
vgremove -f cinder-volumes ;
losetup -a | sed -e 's/:.*//g' | xargs losetup -d ;
find /etc/pki/tls -name "ssl_ps*" | xargs rm -rf ;
for x in $(df | grep "/lib/" | sed -e 's/.* //g') ; do
    umount $x ;
done

yet.org