yet.org

MCP Cookbook - Ceph

In the last few articles, I’ve detailed the workflow to deploy Mirantis Cloud Platform (aka MK now MCP) based on Mirantis OpenStack and a Reclass Model Driven Architecture (MDA). But you may want to use a different backend for storage then our standard MCP one, the reference Cinder LVM ISCSI Driver. In this MCP cookbook article, I’ll guide you step by step, to use Ceph as your storage backend for Glance (Images), Cinder (Volumes) and Nova (Guest Disks). You can use any of these options alone or combined.

Introduction

OpenStack Salt doesn’t deploy Ceph as Mirantis Fuel does, at Mirantis we use Decapod for this task instead. A detailed installation guide on how to use Decapod isn’t the objective of this post. I suppose you’ve already found your way and have a running Ceph cluster. I’ll focus instead on how to adapt our Reclass model to consume your Ceph cluster. So before we start, make sure your Ceph cluster is healthy

# ceph health
HEALTH_OK

glance-api, cinder-volume, nova-compute and cinder-backup will be acting as Ceph clients, so we’ll need a corresponding /etc/ceph/ceph.conf file on each of them.

But first of all, even if our model doesn’t deploy Ceph, it can at least prepare the required infrastructure. For example, by default our model propose only three Infrastructure compute nodes (kvm01, kvm02, kvm03), we could add more if necessary.

Adding Infra compute, updating control and compute Nodes

If you don’t want to host Decapod and Ceph monitoring vms on your existing infra nodes, you can add three more (kvm04,kvm05,kvm06). It can also help to better spread all the required services. Up to nine nodes are pre-configured in Mirantis system repo you just need to add their IPs and hostname as shown below.

# vi classes/cluster/xxx/init.yml
parameters: 
  _param:
    infra_compute_node04_address: <CTL_NET.NODE04_IP>
    infra_compute_node05_address: <CTL_NET.NODE05_IP>
    infra_compute_node06_address: <CTL_NET.NODE06_IP>
    ...
    infra_compute_node04_deploy_address: <PXE_NET.NODE04_IP>
    infra_compute_node05_deploy_address: <PXE_NET.NODE05_IP>
    infra_compute_node06_deploy_address: <PXE_NET.NODE06_IP>
    ...
    infra_compute_node01_storage_address: <STORAGE_FRONTEND_NET.NODE01_IP>  <.>
    infra_compute_node02_storage_address: <STORAGE_FRONTEND_NET.NODE02_IP>
    infra_compute_node03_storage_address: <STORAGE_FRONTEND_NET.NODE03_IP>
    infra_compute_node04_storage_address: <STORAGE_FRONTEND_NET.NODE04_IP>
    infra_compute_node05_storage_address: <STORAGE_FRONTEND_NET.NODE05_IP>
    infra_compute_node06_storage_address: <STORAGE_FRONTEND_NET.NODE06_IP>
    ...
    infra_compute_node04_hostname: kvm04
    infra_compute_node05_hostname: kvm05
    infra_compute_node06_hostname: kvm06

Replace xxx with your cluster name in the remaining of this article.

We are also adding a storage address on all our infra compute nodes.

Now we need to declare them in our linux.network.host pillar within infra/init.yml to add them to each /etc/hosts files.

# vi classes/cluster/xxx/infra/init.yml
linux:
  network:
    host:
      kvm04:
        address: ${_param:infra_compute_node04_address}
        names:
        - kvm04
        - kvm04.${_param:cluster_domain}
      kvm05:
        address: ${_param:infra_compute_node05_address}
        names:
        - kvm05
        - kvm05.${_param:cluster_domain}
      kvm06:
        address: ${_param:infra_compute_node06_address}
        names:
        - kvm06
        - kvm06.${_param:cluster_domain}

Decapod and Ceph monitoring vms > sizing

We need to find a server to install Decapod, and at least three monitoring nodes, we can virtualize all of them and create the required vms with Salt Virt. So we first need to add sizing information for Decapod and Ceph monitoring nodes

# vi classes/cluster/xxx/infra/kvm.yml
parameters
  salt:
    control:
      size: 
        infra.decapod:
          cpu: 4
          ram: 8192
          disk_profile: small
          net_profile: storage
        infra.storage_monitor:
          cpu: 2
          ram: 8192
          disk_profile: small
          net_profile: storage

Decapod and Ceph monitoring VMs > placement | os release | vms specs

In the same file we can also tell Salt on which Infra compute node to deploy each of the required vms, the target operating system and vms specs.

# vi classes/cluster/xxx/infra/kvm.yml
cluster:
  internal:
    node:
      ceph-mon01:
        provider: kvm04.${_param:cluster_domain}
        image: ${_param:salt_control_xenial_image}
        size: infra.storage_monitor
      ceph-mon02:
        provider: kvm05.${_param:cluster_domain}
        image: ${_param:salt_control_xenial_image}
        size: infra.storage_monitor
      ceph-mon03:
        provider: kvm06.${_param:cluster_domain}
        image: ${_param:salt_control_xenial_image}
        size: infra.storage_monitor
      decapod:
        provider: kvm06.${_param:cluster_domain}
        image: ${_param:salt_control_xenial_image}
        size: infra.decapod

Ceph Storage and Ceph monitoring > ip’s

Not to repeat ourself, lets parameterized Ceph storage and monitoring nodes IP Addresses.

# vi classes/cluster/xxx/init.yml
parameters: 
  _param:
    #Ceph storage nodes
    infra_ceph_storage_node01_backend_address: <STORAGE_BACKEND_NET.CEPH01_IP> <.>
    infra_ceph_storage_node02_backend_address: <STORAGE_BACKEND_NET.CEPH02_IP>
    ...
    infra_ceph_storage_node01_frontend_address: <STORAGE_FRONTEND_NET.CEPH01_IP>
    infra_ceph_storage_node02_frontend_address: <STORAGE_FRONTEND_NET.CEPH02_IP>
    ...

    #Ceph monitor nodes
    infra_ceph_storage_monitor_node01_address: <CTL_NET.CEPH_MON01_IP>
    infra_ceph_storage_monitor_node02_address: <CTL_NET.CEPH_MON02_IP>
    infra_ceph_storage_monitor_node03_address: <CTL_NET.CEPH_MON03_IP>
    infra_ceph_storage_monitor_node01_backend_address: <STORAGE_BACKEND_NET.CEPH_MON01_IP>
    infra_ceph_storage_monitor_node02_backend_address: <STORAGE_BACKEND_NET.CEPH_MON02_IP>
    infra_ceph_storage_monitor_node03_backend_address: <STORAGE_BACKEND_NET.CEPH_MON03_IP>
    infra_ceph_storage_monitor_node01_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>
    infra_ceph_storage_monitor_node02_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>
    infra_ceph_storage_monitor_node03_frontend_address: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>

    #CTL nodes
    openstack_control_node01_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE01_IP>  <.>
    openstack_control_node02_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE02_IP>
    openstack_control_node03_storage_address: <STORAGE_FRONTEND_NET.CTL_NODE03_IP>

    #Compute nodes
    openstack_compute_node01_storage_address: <STORAGE_FRONTEND_NET.COMPUTE_NODE01_IP>
    openstack_compute_node02_storage_address: <STORAGE_FRONTEND_NET.COMPUTE_NODE02_IP>
    ...

We are also adding storage frontend IPs for openstack control, and compute nodes, it’s required for them to be able to consume Ceph Storage.

Ceph monitoring > net_profile

Our Ceph monitoring VMs needs to be connected to storage backend and frontend networks, so we have to define a new network profile for these vms

# vi classes/cluster/int/infra/kvm.yml
virt:
  nic:
    ...
    storage:
      - name: eth0
        bridge: br-pxe
        model: virtio
      - name: eth1
        bridge: br-mgmt
        model: virtio
      - name: eth2
        bridge: br-sto-front
        model: virtio
      - name: eth3
        bridge: br-sto-back
        model: virtio

Note: don’t think much about nic names, Salt Virt has weird sorting, it work from bottom up <.>

Ctl > Nics

Our OpenStack Controller will require connectivity to our storage frontend network, lets create another net_profile for these vms

# vi classes/cluster/int/infra/kvm.yml
virt:
  nic:
    ...
    storage-front:
      - name: eth0
        bridge: br-pxe
        model: virtio
      - name: eth1
        bridge: br-mgmt
        model: virtio
      - name: eth2
        bridge: br-sto-front
        model: virtio

Update the openstack.control vms to use this profile instead

# vi classes/cluster/int/infra/kvm.yml
salt:
  control:
    size:
    # Default production sizing
      openstack.control:
        cpu: 8
        ram: 32768
        disk_profile: small
        net_profile: storage-front

You may already have your controller vms running, in such a case running salt.control state won’t touch the existing ones. So you have to manually add the required additional interface connected to our br-sto-front bridge.

Connect on each infra compute node to list vms

kvm01# virsh list
 Id    Name                           State
----------------------------------------------------
 12    ctl01.domainname.com         running

List existing interfaces of the ctl vm

kvm01# virsh domiflist 12
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet8      bridge     br-pxe     virtio      ac:de:48:ec:47:6d
vnet9      bridge     br-mgmt    virtio      ac:de:48:c8:4b:2f

Attach the storage one, mac address has been generated randomly

kvm01# virsh attach-interface --domain 12 --type network \
    --source br-sto-front --model virtio \
    --mac 52:54:00:4b:73:5f --config --live

Check if it has been correclty added

kvm01# virsh domiflist 12
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet8      bridge     br-pxe     virtio      ac:de:48:ec:47:6d
vnet9      bridge     br-mgmt    virtio      ac:de:48:c8:4b:2f
vmnet10    bridge     br-sto-front    virtio   52:54:00:4b:73:5f <.>

You can also check that it will persist

kvm01# virsh edit 12

Interface IP will be configured later on, when we’ll run the linux state on the control plane.

Infra compute > VLAN interfaces | Bonds

For our openstack.control VMs to be able to communicate on the storage frontend network, we need to create the corresponding bridges on our infrastructure compute nodes. They may not all be using the same VLANs, so we can parameterized the tag for each node by using our reclass.storage.node pillars like this for each of your infra compute nodes.

# vi classes/cluster/int/infra/config.yml
...
  reclass:
    storage:
      node:
        infra_compute_node01:
          params:
            keepalived_vip_priority: 100
            linux_system_codename: xenial
            sto_front_interface: <BOND>.<STO_FRONT_VLAN>
            sto_back_interface: <BOND>.<STO_BACK_VLAN>
        ...

Now you can use this _params that will be generated when you’ll run reclass.storage state at the end of this article. Use it to create the required storage VLAN interfaces and bonds

# vi classes/cluster/int/infra/kvm.yml
parameters:
  linux:
    network:
      interface:
        ...
        sto_front_interface: 
          name: ${_param:sto_front_interface}
          enabled: true
          type: vlan
          proto: manual
          use_interfaces:
          - bond0
        br-sto-front:
          enabled: true
          type: bridge
          proto: manual
          use_interfaces:
          - ${_param:sto_front_interface}

        sto_back_interface: 
          name: ${_param:sto_back_interface}
          enabled: true
          type: vlan
          proto: manual
          use_interfaces:
          - bond0
        br-sto-back:
          enabled: true
          type: bridge
          proto: manual
          use_interfaces:
          - ${_param:sto_back_interface}

Compute > VLAN interface | Bond

Compute node only need a connection to the storage frontend, so lets create such a bridge on them, this time without parameters

# vi classes/cluster/int/openstack/compute.yml
parameters:
  linux:
    network:
      interface:
      ...
      <BOND>.<VLAN>: 
        enabled: true
        type: vlan
        proto: static
        address: ${_param:storage_address}
        netmask: <NETMASK>
        mtu: 9000
        use_interfaces:
        - <BOND>
      br-sto-front:
        enabled: true
        type: bridge
        proto: manual
        use_interfaces:
        - <BOND>.<VLAN>

As you see above, this time we need to bind an address to them.

Ceph Storage and Ceph monitoring > nodes declaration

For our Ceph monitoring and storage nodes to be managed by Salt, we need to declare them in our reclass.storage.node pillar. When we’ll run the corresponding reclass.storage state, it will magically create the nodes definitions in nodes/_generated, great isn’t it.

# vi classes/cluster/xxx/infra/config.yml
parameters:
  reclass:
    storage:
      node:
        openstack_compute_node01:
          ...
          storage_address: ${_param:openstack_compute_node01_storage_address} <.>
          ...
          ...
          ...
        infra_storage_node01: <.>
          name: ceph01
          domain: ${_param:cluster_domain}
          classes:
          - cluster.${_param:cluster_name}.infra.storage
          params:
            salt_master_host: ${_param:reclass_config_master}
            linux_system_codename: xenial
            single_address: ${_param:infra_ceph_storage_node01_frontend_address}
            backend_address: ${_param:infra_ceph_storage_node01_backend_address}
        infra_storage_node02:
          name: ceph02
          domain: ${_param:cluster_domain}
          classes:
          - cluster.${_param:cluster_name}.infra.storage
          params:
            salt_master_host: ${_param:reclass_config_master}
            linux_system_codename: xenial
            single_address: ${_param:infra_ceph_storage_node02_frontend_address}
            backend_address: ${_param:infra_ceph_storage_node02_backend_address}
        ...

        infra_storage_monitor_node01:
          name: ceph-mon01
          domain: ${_param:cluster_domain}
          classes:
          - cluster.${_param:cluster_name}.infra.storage_monitor
          params:
            salt_master_host: ${_param:reclass_config_master}
            linux_system_codename: xenial
            single_address: ${_param:infra_ceph_storage_monitor_node01_address}
            frontend_address: ${_param:infra_ceph_storage_monitor_node01_frontend_address}
            backend_address: ${_param:infra_ceph_storage_monitor_node01_backend_address}
        infra_storage_monitor_node02:
          name: ceph-mon02
          domain: ${_param:cluster_domain}
          classes:
          - cluster.${_param:cluster_name}.infra.storage_monitor
          params:
            salt_master_host: ${_param:reclass_config_master}
            linux_system_codename: xenial
            single_address: ${_param:infra_ceph_storage_monitor_node02_address}
            frontend_address: ${_param:infra_ceph_storage_monitor_node02_frontend_address}
            backend_address: ${_param:infra_ceph_storage_monitor_node02_backend_address}
        infra_storage_monitor_node03:
          name: ceph-mon03
          domain: ${_param:cluster_domain}
          classes:
          - cluster.${_param:cluster_name}.infra.storage_monitor
          params:
            salt_master_host: ${_param:reclass_config_master}
            linux_system_codename: xenial
            single_address: ${_param:infra_ceph_storage_monitor_node03_address}
            frontend_address: ${_param:infra_ceph_storage_monitor_node03_frontend_address}
            backend_address: ${_param:infra_ceph_storage_monitor_node03_backend_address}

We’ve also added a storage_address to each of our compute node.

Note: Keep in mind that infra_ceph_storage_monitor_node03_address is set in classes/cluster/int/init.yml and abstracted away by classes/cluster/int/infra/config.yml into single_address and consumed by classes/cluster/int/init.yml to create interface definitions.

Ceph monitoring > networking

As we’ve said earlier, Decapod will be used to install all of Ceph but Salt is creating the VMs. So lets decribe required network connectivity on each of the storage monitoring VM

# vi classes/cluster/int/infra/storage_monitor.yml
classes:
  - system.linux.system.repo.saltstack_2016_3_xenial
  - cluster.int
  parameters:
    linux:
      network:
        interface:
          # CTL NET
          eth1: ${_param:linux_single_interface}
          # STO_FRONT NET
          eth2:
            enabled: true
            type: eth
            proto: static
            address: ${_param:frontend_address}
            netmask: 255.255.252.0
          # STO_BACK NET
          eth3:
            enabled: true
            type: eth
            proto: static
            address: ${_param:backend_address}
            netmask: 255.255.252.0

We do not touch eth0 with our model, it will be configured by dhcp.

Note: This class is associated to each Ceph monitoring by classes/cluster/xxx/infra/config.yml when we declare them.

Ceph client > authentication

For our client to authenticate to Ceph using cephx we need to create new user for Nova/Cinder and Glance

ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=object'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder, allow rwx pool=nova, allow rwx pool=images'
ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=nova'

Verify the users exist

ceph auth list

Ceph client > _params

system.ceph.client.single will configure our nodes as Ceph clients by running the salt formula ceph with the required Pillar data which are.

ceph_fsid: uuid of the Ceph Cluster
ceph_mon_initial_members: initial Ceph monitoring nodes FQDN
ceph_mon_host: IP:Port List of Ceph monitoring nodes
ceph_auth_client_required: choose cephx here
ceph_public_network: Ceph frontend Network CIDR
ceph_cluster_network: Ceph backend Network CIDR

So you now have to provide all the required parameters in classes/cluster/xxx/openstack/init.yml

# vi classes/cluster/xxx/openstack/init.yml
parameters:
  _param:
    ...
    cinder_storage_user: cinder
    glance_storage_user: object
    nova_storage_user: nova

    nova_storage_secret_uuid: <NOVA_STORAGE_SECRET_UUID>

    cinder_storage_client_key: <CINDER_STORAGE_KEY>
    glance_storage_client_key: <GLANCE_STORAGE_KEY>
    nova_storage_client_key: <NOVA_STORAGE_KEY>
    admin_storage_client_key: <ADMIN_STORAGE_KEY>

    cinder_storage_pool: cinder
    glance_storage_pool: object
    nova_storage_pool: nova

    ceph_fsid: <CEPH CLUSTER UUID>
    ceph_mon_initial_members: ceph-mon01,ceph-mon02,ceph-mon03
    ceph_mon_host: <STORAGE_FRONTEND_NET.CEPH_MON01_IP>:6789,<STORAGE_FRONTEND_NET.CEPH_MON02_IP>:6789,<STORAGE_FRONTEND_NET.CEPH_MON03_IP>:6789
    ceph_auth_client_required: cephx
    ceph_public_network: <STORAGE_FRONTEND_NET.CIDR>
    ceph_cluster_network: <STORAGE_BACKEND_NET.CIDR>

  ceph:
    client:
      keyring:
        object:
          key: ${_param:glance_storage_client_key}
        cinder:
          key: ${_param:cinder_storage_client_key}
        nova:
          key: ${_param:nova_storage_client_key}
        admin:
          key: ${_param:admin_storage_client_key}

To be able to mount Ceph backed volume, the nova compute nodes need to store the secret key of the client.cinder user in libvirt. So it will be stored behind the provided <NOVA_STORAGE_SECRET_UUID>. Ceph salt formula will be storing it using a secret.xml template file containing

<secret ephemeral='no' private='no'>
  <uuid>{{ compute.ceph.secret_uuid }}</uuid>
    <usage type='ceph'>
      <name>client.{{ compute.ceph.get('rbd_user', 'cinder') }} secret</name>
    </usage>
  </secret>

And to define and store the secret in libvirt

# virsh secret-define --file secret.xml
# virsh secret-set-value --secret NOVA_STORAGE_SECRET_UUID> --base64 <CINDER_STORAGE_KEY>

As we’ve seen earlier, you can get <CINDER_STORAGE_KEY> with

# ceph auth get-key client.cinder

On a compute node, you can list the existing libvirt secrets with

# virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 a5d0dd94-57c4-ae55-ffe0-7e3732a24455  ceph client.nova secret

But do not care too much, Salt will do all this boring stuff for you. Thanks Salt !!!

Ceph Cluster uuid (generated with uuidgen at Ceph install time) can be found in your ceph cluster in /etc/ceph/ceph.conf

You can grab the required keys for client.cinder, client.glance, client.cinder-backup, client.nova and client.admin like this

ceph auth get-or-create client.glance
ceph auth get-or-create client.cinder
ceph auth get-or-create client.cinder-backup
ceph auth get-or-create client.nova
ceph auth get-or-create client.admin
[client.admin]
   key = AQCdhCCCt2PmGRAAwww8oYhiiik80VQ==

You can also try authentication by requesting your cluster status

# ceph -s --name client.cinder --keyring /etc/ceph/ceph.client.cinder.keyring

Ceph Pools

We’ve configured the cinder, object, nova as the Ceph pools to be used by OpenStack. We need to create them on our Ceph Cluster

ceph osd pool create cinder 128
ceph osd pool create object 128
ceph osd pool create nova 128
ceph osd pool create backups 128 <.>

See Create a Pool for detail on specifying the number of placement groups for your pools, and Placement Groups for details on the number of placement groups you should set for your pools.

Verify the pools exists

ceph osd lspools

Ceph > Compute nodes <.>

We have to tell our compute nodes about Ceph and connect our hypervisor to the storage frontend network. The following code does most of the job for us. You just have to adapt it to your requirements.

vi classes/cluster/mb-staging/openstack/compute.yml <.>
classes:
  ...
  - system.ceph.client.single
  ...
parameters:
  linux:
    network:
      bridge: openvswitch
      interface:
      ...
        bond0.<STO_NET_VLAN>:
          enabled: true
          type: vlan
          proto: static
          address: ${_param:storage_address}
          netmask: <STO_NET_NETMASK>
          mtu: 9100
          use_interfaces:
          - bond0
  ...
  nova:
    compute:
      ceph:
        enabled: true
        ephemeral: yes
        rbd_pool: nova
        secret_uuid: ${_param:nova_storage_secret_uuid}
        client_cinder_key: ${_param:nova_storage_client_key}
        rbd_user: nova

The above nova section configure nova to use Ceph as a backend for ephemeral storage.

Ceph > Ctl nodes

Our control nodes need a bit more classes

vi classes/cluster/mb-staging/openstack/control.yml <.>
classes:
  ...
  - system.glance.control.storage.ceph
  - system.ceph.client.single

vi classes/cluster/mb-staging/openstack/control_init.yml 
classes:
  ...
  - system.cinder.control.backend.ceph
  - system.cinder.volume.backend.ceph

system.glance.control.storage.ceph configure Ceph RBD as a backend for Glance
system.cinder.control.backend.ceph configure Ceph as a backend for Cinder
system.cinder.volume.backend.ceph configure Ceph as a backend for Cinder

Note: if you don’t have these classes in your system repo, you may have to bump its version a bit.

Glusterfs > remove Glance volume

In our standard deployment we use a Glusterfs volume to store our Glance image. If you’ve switched to Ceph it’s a good idea to comment out this feature in your model

# vi classes/xxx/openstack/control.yml
...
#- system.glusterfs.client.volume.glance
...
#- system.glusterfs.server.volume.glance

If you’ve already ran the Glusterfs state, you’ll have to manually unmount and delete the Glance volumes which are shared by the infrastructure nodes. On each of your three controllers run

ctl01# umount /var/lib/glance/images

On one of your kvm node run

kvm01# gluster volume stop
kvm01# gluster delete volume glance

The above command does not delete the data, you have to do this manually too. <.>

git > commit

We now have to share our work to our team, commit your changes

# git add .
# git commit -m "Ceph backend for Nova, Glance and Cinder"
# git push

Now ssh to your MCP salt master to pull the updated model

cfg01# cd /srv/salt/reclass
cfg01# git pull

Salt states > provision Ceph infra

It’s now time apply our new model. First of all run the following state to create the ceph-mon01,ceph-mon02,ceph-mon03 node definition under nodes/_generated.

# salt 'cfg01*' state.sls reclass.storage 

Next you can create the four VMs: ceph-mon01, ceph-mon02, ceph-mon03 and decapod.

# salt 'kvm*' state.sls salt.control

Since you have new nodes, refresh pillars and sync_all

# salt '*' saltutil.refresh_pillar && salt '*' saltutil.sync_all

Run the following states on your ceph-mon’s and decapod minions, also run on kvm04, kvm05, kvm06 if you’ve added them.

# salt -C 'E@ceph-mon* and E@decapod*' linux,openssh,rsyslog,ntp,salt.minion

Salt states > Ceph OpenStack backend configuration

Now that we have your Ceph cluster up and running and healthy, we are ready to tell OpenStack to consume it.

Run linux node on your compute, infra compute and ctl nodes to update networking with new storage networks.

# salt -C 'E@kvm* and E@cmp* and E@ctl*' state.sls linux

Run nova state on your computes

# salt -C 'cmp*' state.sls nova

Run cinder|glance on your control plane

# salt -C 'I@cinder:controller' state.sls cinder -b 1
# salt -C 'I@glance:server' state.sls glance -b 1

That should be it. If I forgot something it will be added above soon, as of January 31, 2017 ;)

Glance Image

If we are lucky, we should now be able to upload a Glance image, remember that Ceph expect the image to be in RAW format if you want to boot from it, you can convert it from qcow2 like this

qemu-img convert -f qcow2 -O raw ubuntu-16.04-xenial-server-cloudimg-amd64-disk1.img ubuntu-16.04.raw

upload it

# source /root/keystonerc
# glance image-create --name ubuntu-16-04 --is-public=true --disk-format=raw --container-format=bare < ubuntu-16.04.raw

You can verify that the image is stored in Ceph by querying the image ID in the Ceph image pool

# rados -p object ls --name client.glance --keyring /etc/ceph/ceph.client.glance.keyring | grep -i id

Create a volume from it, to be able to boot an instance using copy-on-write

# cinder create --image-id <IMAGE_ID> --display-name <VOL_NAME> <SIZE>

You can also verify volume is stored within Ceph cinder pool

# rados -p cinder ls --name client.cinder --keyring /etc/ceph/ceph.client.cinder.keyring | grep -i id

Conclusion

Having the capability to model our infrastructure using Reclass and Salt from a git repository is really nice. As we’ve demonstrated in this cookbook article. It gives us the ability to adapt our model to our need, from Cinder iSCSI LVM to Ceph. Not only that, it also gives you a total visibility about who’s doing what over time on the versionned repository, it’s infrastructure as code, at its core.

At the same time leveraging hundreds of salt formulas to do anything from linux configuration to Kubernetes deployment is a great bonus. Our model could also be adapted to use Neutron ml2 Open vSwitch plugin instead of OpenContrail, but I think you see my point and that’s the subject of another cookbook article, maybe.

See you next time. In the meantime, take care !

PS: Before I leave I want to thank Ondrej Smola, Pavel Cizinsky and Jakub Pavlik for their great support :)