Kubernetes by Ansible

Kargo (a.k.a Kubespray) is an initiative to deploy a Kubernetes cluster using Ansible. It will contrast with our previous Step by Step article by showing that we can deploy a cluster with a single command, a bit like the newly integrated SwarmKit feature announced in Docker 1.12 docker swarm init.


As you’ve seen in our Step by Step article, deploying a Kubernetes cluster manually is a long, error prone and tedious process. This exercice was just meant as a learning exercice, to understand the different moving parts and how everything fits together.

Today our objective is to get a running Kubernetes cluster as fast as possible, to reach that objective we’ll be automating all of its deployment on top of vSphere with Kubespray which is using Ansible under the cover. It is like playing Kelsey Hightower recent challenge, where he asked engineers to deploy a cluster in less then 45’, without using or any other official helper.


Kargo support CoreOS, Debian, Ubuntu, Fedora and CentOS/RHEL, you’ll need at least 3 nodes running such OS. In my case, I’ll stick with CoreOS which is specifically built for that kind of use case.

These systems require Internet access to pull docker images and they require to allow SSH access from the Ansible (v2.x) Control Machine. The remote user (option –user) must be in the sudoers without password.

If you need details on how to prepare your Ansible control machine, consult my Ansible article.

Clone Kargo repository

Login to your Ansible Control Machine to clone the Kargo repository

cd ~
git clone

Ansible inventory

Still from your Ansible Control Machine, Create a ~/kargo/inventory/inventory.cfg inventory file

vi ~/kargo/inventory/inventory.cfg

k8s-01 ansible_ssh_host=
k8s-02 ansible_ssh_host=
k8s-03 ansible_ssh_host=





Replace above the nodename and IPs with your own. Each INI section define the group of machines deployed for the respective roles. Here we’ll have 2 Kubernetes master, a 3-node etcd cluster, and 2 worker nodes.

Prepare CoreOS nodes

In my case, I won’t be using any public cloud environment for my Kubernetes automated deployment but instead my own Bulb lab. So I need to boostrap three CoreOS VMs on vSphere.

Start by downloading the latest CoreOS OVA image

curl -LO

Upload this OVA to your vSphere datastore and deploy it three times by launching the Deploy OVF Template workflow from the vSphere Web UI.

Note: You don’t need to specify any parameters during the OVF deployment, you’ll provide them later by editing directly the VMX file of each VM. vApp parameters passing to VMX doesn’t seem to work so well for me :(

While the VMs are deploying create a cloud config file which contains your public SSH key

vi user_data


  - "ssh-rsa AAAAB......."

Replace the ssh key by the public key of your ansible control machine, to allow it to access coreOS nodes passwordless.

Encode it as a base64 string

base64 user_data && echo

Before powering any VM up, ssh to the ESXi host where it has been deployed and edit the corresponding vmx file to add configuration parameters using the guest info interface

ssh root@<ESXi_IP>
vi /vmfs/volumes/datastore-1/

And add the following content

guestinfo.hostname = ""
guestinfo.interface.0.role = "private"
guestinfo.dns.server.0 = ""
guestinfo.interface.0.route.0.gateway = ""
guestinfo.interface.0.route.0.destination = "" = "en*"       
guestinfo.interface.0.dhcp = "no"
guestinfo.interface.0.ip.0.address = "" = "I2Nsb3VkLWNvb..." = "base64"

Replace hostname, dns, gateway, mac, interface name, IP address with your own.

The configdata is the base64 encoded string you’ve got above.

To tell vSphere to reload that file you need to get the ID of your VM, look for your VM in the list output of

vim-cmd vmsvc/getallvms

Reload the VMX

vim-cmd vmsvc/reload <Vmid>

Repeat this process for all your nodes and power them up, once booted you can check the VMX info has been seen by your VM

core@k8s-01$ /usr/share/oem/bin/vmtoolsd --cmd "info-get guestinfo.interface.0.dhcp"
core@k8s-01$ /usr/share/oem/bin/vmtoolsd --cmd "info-get"

If you’ve got a DHCP assigned IP address, do not freak out, just reboot, it seems the VMX parameters saying we don’t want DHCP is set after an IP is already assigned, but that will get fixed after the second reboot.

CoreOS nodes require special care before you can start the Kargo provisionning.

First check that each node does have at least 400M of disk space in /opt/bin which should be writable.

sudo mkdir -p /opt/bin

Log back to your Ansible Control Machine to uncomment the variable ansible_python_interpreter in the file ~kargo/inventory/group_vars/all.yml

Finish the bootstrapping by running the Python bootstrap playbook

ansible-playbook -u core -e ansible_ssh_user=core -b --become-user=root \
                 -i inventory/inventory.cfg coreos-bootstrap.yml
.trucated output.
PLAY RECAP *********************************************************************
k8s-01                     : ok=10   changed=7    unreachable=0     failed=0
k8s-02                     : ok=10   changed=7    unreachable=0     failed=0
k8s-03                     : ok=10   changed=7    unreachable=0     failed=0        

If not already bootstrapped, this playbook install an alternative, fast, python interpreter, pypy in /opt/bin, install pip and httplib2 required Python module.

Kargo Installation

Kargo does provide a command line interface, it’s an Ansible wrapper, we won’t be using it here but you can install it on your Ansible Control Machine with pip

sudo pip2 install kargo

If you don’t want to override default value using kargocli command line, you can now edit them

vi /etc/kargo/kargo.yml

Kubernetes Cluster Deployment

The beauty of automation comes now, to deploy your k8s cluster first makes sure you have your private key loaded on your Ansible control machine

eval `ssh-agent -s`

Deploy your cluster with this single command

ansible-playbook -i ./inventory/inventory.cfg -u core -e ansible_ssh_user=core -b cluster.yml

-u define the user that will be used to connect over SSH to the remote system
-b tells ansible to become root on the target system

It should terminate with such a message after less then 30’

PLAY RECAP *********************************************************************
k8s-01                     : ok=151  changed=22   unreachable=0    failed=0   
k8s-02                     : ok=142  changed=3    unreachable=0    failed=0   
k8s-03                     : ok=71   changed=9    unreachable=0    failed=0   

If that’s not your case, re-run the above command.

Checking the k8s environment

To check that everything looks good you can login to your master node and list the worker node

ssh core@
kubectl get nodes
k8s-02    Ready     7m
k8s-03    Ready     7m

As you can see above we have two worker nodes ready which correspond to the thing we’ve declared in our Ansible inventory.

You can also check for services that are currently declared within your cluster

kubectl get svc
kubernetes   <none>        443/TCP   17m

List the pods in the kube-system namespace with

kubectl get po --namespace=kube-system
NAME                             READY     STATUS    RESTARTS   AGE
dnsmasq-3da6b                    1/1       Running   0          9m
dnsmasq-cvywv                    1/1       Running   0          9m
flannel-k8s-02                   2/2       Running   0          9m
flannel-k8s-03                   2/2       Running   0          9m
kube-controller-manager-k8s-02   1/1       Running   0          9m
kube-proxy-k8s-02                1/1       Running   0          9m
kube-proxy-k8s-03                1/1       Running   0          9m
kube-scheduler-k8s-02            1/1       Running   0          9m
kubedns-5m773                    4/4       Running   0          8m

Great everything looks good !!!

Kubernetes Dashboard

In my previous Step by Step installation, I couldn’t successfully install Kubernetes Dashboard, lets try again here.

The corresponding k8s manifest should look like that

vi kubernetes-dashboard.yml

kind: Deployment
apiVersion: extensions/v1beta1
    app: kubernetes-dashboard
    version: v1.1.0
  name: kubernetes-dashboard
  replicas: 1
      app: kubernetes-dashboard
        app: kubernetes-dashboard
      - name: kubernetes-dashboard
        imagePullPolicy: Always
        - containerPort: 9090
          protocol: TCP
          # Uncomment the following line to manually specify Kubernetes API server Host
          # If not specified, Dashboard will attempt to auto discover the API server and connect
          # to it. Uncomment only if the default does not work.
          # - --apiserver-host=http://my-address:port
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
kind: Service
apiVersion: v1
    app: kubernetes-dashboard
  name: kubernetes-dashboard
  type: NodePort
  - port: 80
    targetPort: 9090
    app: kubernetes-dashboard

Deploy the dashboard with

kubectl create -f kubernetes-dashboard.yml

Look for the assigned NodePort with

kubectl describe svc kubernetes-dashboard
Name:                   kubernetes-dashboard
Namespace:              default
Labels:                 app=kubernetes-dashboard
Selector:               app=kubernetes-dashboard
Type:                   NodePort
Port:                   <unset> 80/TCP
NodePort:               <unset> 30991/TCP
Session Affinity:       None
No events.

As you can see above you should then be able to connect on any node IP address on port 30991, so try to open

You should then see the following dashboard


That conclude our Kubernetes by Ansible article, as you’ve seen it’s a lot simpler and faster to automate the deployment of k8s.

Kargo also works in Google Compute, OpenStack or Amazon, which is great for application portability. Hybrid cloud nirvana is around the corner ;)