Salt Formulas
Always reinventing the wheel doesn’t pay off most of the time, so telling Salt what to do by creating Salt States again and again to install application components isn’t really efficient. Instead Salt Formulas brings convention and a bit of magic, and offer reusable bundles which package altogether all the necessary piece to automate a specific task, like deploying etcd, a distributed key value store cluster, which we will take as an example in this article.
Formulas have the objective of being simple enough, avoid repetition, and prevent you from having multiple places to update when changes comes. They should also be applicable to existing configuration. They pretty much reach these objectives but it’s somewhat difficult to understand how they work and the magic behind it. Like Ruby On Rails, most of Formulas construct lies behind conventions so let’s details all of this to clarify everything.
First of all a Salt Formula live in its own git repository which should look like this
foo-formula
|-- foo/
| |-- map.jinja
| |-- init.sls
| `-- bar.sls
|-- CHANGELOG.rst
|-- LICENSE
|-- pillar.example
|-- README.rst
`-- VERSION
They are mostly composed of State File (init.sls, bar.sls) which describe the end state of the system (declarative), plus added bonus to make it easily reusable. Salt Formulas are similar to Chef Cookbooks or Ansible Roles.
As of today, you’ll find around a hundred formulas on the official github formula repository.
Lets details the different files and functions, starting with the most important one, map.jinja, where the magic happens.
map.jinja
The map.jinja is the important piece, it sets data based on os_family
grain and merges Pillar data in. It’s a great place to centralize variables to avoid repetition.
{% set server = salt['grains.filter_by']({
'Debian': {
'pkgs': ['etcd', 'python-etcd'],
'services': ['etcd']
},
'RedHat': {
'pkgs': [],
'services': []
},
}, merge=salt['pillar.get']('etcd:server')) %}
First Line
set the server variable to grains.filter_by which match on os_family, our formula will work on Debian and RedHat but other OSs could be added. On a Ubuntu machine server.services will be set to etcd. Redhat section is not yet filled out :/
Line 2-9
a bunch of assignment which depend on what’s in the os_family grain.
Last Line
is a bit confusing, this will merge all the data from the Pillar etcd.server yaml section into the server variable. Pillar data will overwrite map.jinja assignment.
To use map.jinja into a State file, just import it
{% from "etcd/map.jinja" import server with context %}
This will import the Jinja template with context, meaning that variable will come over, server will contain values for pkgs and services with all the Pillar data merged into it.
You can then use the imported data in your State file
etcd_packages:
pkg.installed:
- names: {{ server.pkgs }}
We don’t need to use Grains any more in your State, all the data available is already pre-calculated based on os_family
in the map.jinja. You see the magic in action here ;)
You can also access Pillar data, which should lives in a file ’/srv/pillar/etcd.sls` with the same name as the formula, like this
{{ server.engine }}
or to create conditional block in your template based on it
{% if server.get('engine', 'systemd') %}
engine
is the key looked at, it should be declared in your pillar file within the etc.server yaml section
systemd
default value returned if key doesn’t exist in our server variable, meaning if it hasn’t been defined in map.jinja or our Pillar.
Pillar.example
At the root directory of each formula a pillar.example file should give an overview of how to use Pillar data to customize how the Formula States will perform.
Pillar data
Lets now look more closely at how to use Pillar data with formulas.
top file
Pillar gets assigned to minion in the /srv/pillar/top.sls
file. So to assign a /srv/pillar/etcd.sls
Pillar data file to 3 minions.
base:
'saltstack-m01,saltstack-m02,saltstack-m03':
- etcd
- match: list
I’m using a List match, for an etcd cluster I need at least 3 nodes. More information on Salt minion matching is available in my About SaltStack article.
pillar file
Now lets have a look at our Pillar content saved as /srv/pillar/etcd.sls
linux:
system:
name: {{ grains['fqdn'] }}
etcd:
server:
enabled: true
bind:
host: {{ grains['ipv4'][1] }}
token: $(uuidgen)
members:
- host: 172.16.52.101
name: saltstack-m01
port: 4001
- host: 172.16.52.102
name: saltstack-m02
port: 4001
- host: 172.16.52.103
name: saltstack-m03
port: 4001
Above you can override anything that exist in the map.jinja file with a single line of code. If you want to overrides services, just add the following line at the end
services: etcd-new-srvs-name
Data from Pillar have precedence over map.jinja variables.
All of the above data will then be available to the formula State file as
{{ server.bind.host }}
So Pillar are a really simple way to inject stuff into any formulas and even override some of their default settings.
README.rst
Should give an overview in restructured text of the way the Formula can be used and what it does.
CHANGELOG.rst
Each new version should have a line in that file which describe the deltas
VERSION
Should contain the currently released version of the particular formula. Could be a git repository tag which will become the package version as well, when formula will be packaged as debian pkg.
Formula are versioned according to Semantic Versioning
Convention and best practices
The etcd formula example is taken out of the tcp cloud repository, they are currently maintaining OpenStack-Salt. This formula isn’t 100% compliant with Salt Formulas conventions, it should have embedded the Pillar data into a lookup key so the map.jinja merge line should have been
}, merge=salt['pillar.get']('etcd:lookup')) %}
and the corresponding /srv/pillar/etcd.sls
file
linux:
system:
name: {{ grains['fqdn'] }}
etcd:
lookup:
enabled: true
bind:
host: {{ grains['ipv4'][1] }}
token: $(uuidgen)
members:
- host: 172.16.52.101
name: saltstack-m01
port: 4001
- host: 172.16.52.102
name: saltstack-m02
port: 4001
- host: 172.16.52.103
name: saltstack-m03
port: 4001
It would also have prevented the confusion between server the map variable and server the section of the YAML file !!! But it’s really just a convention.
Formula installation
All of this looks great but how can we then install the above etcd formula on our Salt Master. You have different options.
Filesystem storage
You can install the formula files to your master by cloning your forked repository
mkdir /srv/formulas
cd /srv/formulas
git clone https://github.com/planetrobbie/salt-formula-etcd/
Cloning into 'salt-formula-etcd'...
remote: Counting objects: 100, done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 100 (delta 20), reused 0 (delta 0), pack-reused 49
Receiving objects: 100% (100/100), 21.58 KiB | 0 bytes/s, done.
Resolving deltas: 100% (25/25), done.
Checking connectivity... done.
Now you just have to add the corresponding formula directory in the Salt Master /etc/salt/master
configuration file
file_roots:
base:
- /srv/salt
- /srv/formulas/salt-formula-etcd
And restart your master
service salt-master restart
It’s done :)
git storage backend
Another option would be to use gitfs to connect to your forked directory instead of cloning it locally by adding the following line in your /etc/salt/master
configuration file
fileserver_backend:
- root
- git
gitfs_remotes:
- https://github.com/<git username>/salt-formula-etcd.git
But for this to work, you need to install the dependency, pygit2 is the default provider if no other one, like Dulwich or GitPython are configured in gitfs_provider
in the above file.
So install this dependency
apt-get install python-pygit2
or for GitPython
apt-get install python-git
or for Dulwich
apt-get install python-dulwich
You should not connect your master directly to a 3rd party git repository or clone it directly, fork it instead to make sure you keep control over repository updates. It explain why we’ve put <git username>
in the URL above, git repository should be yours.
Applying to your minions
We have everything in place, our formula /srv/formulas/salt-formula-etcd
, our Pillar top file /srv/pillar/top.sls
and Pillar data file /srv/pillar/etcd.sls
. The last required bits is to assign our formula to our minion within /srv/salt/top.sls
which then contain
base:
'saltstack-m01,saltstack-m02,saltstack-m03':
- match: list
- hostfile
- etcd
hostfile
state will update the hostfile of each node, it contains, it’s necessary or etcd won’t be able to start
updating hostfile:
host.present:
- ip: {{ grains['ipv4'][1] }}
- names:
- {{ pillar.linux.system.name }}
grains['ipv4'][1]
IP Address of the minion
pillar.linux.system.name
hostname of your minion declared in the etcd.sls Pillar file.
It’s now time to apply our formula to our node, it is as simple as
salt '*' state.apply
And will converge the formula state to our three minion. After few minutes you should have a fully operational etcd cluster. You can easily repeat this pattern each time you need such a cluster or everything else that has been described in Salt Formulas. Great isn’t it !!!
If you don’t believe me, check your cluster status by SSHing to one of your minion and running
etcdctl cluster-health
member 970c12c81e0cde8 is healthy: got healthy result from http://172.16.52.103:4001
member 4caf2f327f93ec8f is healthy: got healthy result from http://172.16.52.101:4001
member d59d70c40eed3cba is healthy: got healthy result from http://172.16.52.102:4001
Try to store/retrieve keys/values
etcdctl set /version 1.0
etcdctl get /version
etcdctl rm /version
You can also store data into it using salt, refer to the documentation to see how to declare your cluster to Salt. etcd can also be used as a repository for Pillar data and other stuff !!!
Writing your own Formula
Instead of starting from a blank page, use this template. Then think about what is OS dependant and what might need to be expanded, don’t forget to set some variable to allow users to change it without giving them a ton of work. Put youself in the shoes of your formula users.
You can contribute back by asking one of the members over irc to create a repository under the saltstack github organisation, then fork it, and do a pull request to merge your stuff back in. Congrat you’ve published your first Salt Formula, make sure to maintain it too ;)
To help you build your next formula, in the next chapter, I share the main Jinja2 design patterns with some examples.
Jinja2 patterns
Jinja2, the default Salt templating engine (renderer), can do a lot, not to repeat what’s described in understanding Jinja lets focus on the patterns that improve your formulas.
filter_by
As we’ve said earlier, filter_by match on the os_family grain by default, but it can be changed by passing another grain as argument, see the function signature
salt.modules.grains.filter_by(lookup_dict, grain='os_family', merge=None, default='default', base=None)
lookup_dict
dictionary, keyed by a grain, current developement release (Carbon) allows to use globbing for the dictionary keys.
grain
name of a grain to match. Could be a list in the Carbon release. will return the lookup_dict value for a first found item in the list matching one of the lookup_dict keys.
merge
dictionary to merge with the results of the grain selection from lookup_dict
default
default lookup_dict’s key used if the grain does not exists or if the grain value has no match on lookup_dict. If unspecified the value is “default”.
base
lookup_dict key to use for a base dictionary. The grain-selected lookup_dict is merged over this and then finally the merge dictionary is merged. This allows common values for each case to be collected in the base and overridden by the grain selection dictionary and the merge dictionary. Default is unset.
You can list available grains with
salt '*' grains.items
Interresting ones are os, oscodename, osrelease which are for a Ubuntu 16.04 system: Ubuntu, xenial, 16.04
Still curious read the source code ;)
update
Once you’ve merged Jinja variables and Pillar data together, you can still update the resulting dictionary like this
{% set os_map = salt['grains.filter_by']({
'Debian': {
'config': '/etc/collectd/collectd.conf',
'javalib': '/usr/lib/collectd/java.so',
'pkg': 'collectd-core',
'plugindirconfig': '/etc/collectd/plugins',
'service': 'collectd',
...
...
},
...
...
}, merge=salt['pillar.get']('collectd:lookup')) %}
{% set default_settings = {
'collectd': {
'Hostname': salt['grains.get']('fqdn'),
...
...
}
} %}
{% do default_settings.collectd.update(os_map) %}
{% set collectd_settings = salt['pillar.get']('collectd', default=default_settings.collectd, merge=True) %}
In the above code, we first do the usual os dependent stuff and then we set a variable which contains a map with all the default collectd settings.
We then merge os_map into the settings dictionary with the update call. This pattern is usefull to differentiate OS dependent variable from default settings and put all of this in a single map.
The last line put all this together by merging configuration Pillar data that lives in the collectd key with our default_settings.collectd map. So when importing this map.jinja we’ll get everything required, os_dependent and default_settings overwritten by our pillar data in a single variable. To access it
{{ collectd_settings.pkg }}
or
{{ collectd_settings.Hostname }}
This example comes from collectd formula. In the 2015.5.0 Salt release, a base argument was added to the filter_by function. This formula can be simplified by setting up that argument to default_settings instead of doing an update later on ! But update can still be usefull in some cases.
import_*
It’s also possible to import and deserialize an external yaml file which is then made available as a Jinja dictionary. For example, in the apache map.jinja you’ll find
{% import_yaml "apache/modsecurity.yaml" as modsec %}
Other function like import_json or import_text can import their respective formats.
custom modules
To add a custom module in your formula, just create a _modules
directory at its root, store your python code inside it and call it from your templates like this
{{ salt['custom_module_name.function_name'](args...) }}
tcp cloud Formulas
tcp cloud is a company, acquired by Mirantis, which is specialized in deploying OpenStack and OpenContrail using Salt. They’ve built Formulas for all the required components like keystone, horizon, cinder, nova, etc..
On top of what has been said about formula conventions let see their guidelines.
directory structure
The directory structure looks pretty much the same as the one we’ve described in the introduction with the following added stuff
foo-formula
├── _grains/
| └── service.yml
├── _modules/
| └── service.yml
├── _states/
| └── service.yml
├── debian/
| ├── changelog
| ├── compat
| ├── control
| ├── copyright
| ├── docs
| ├── install
| ├── rules
| └── source
| └── format
├── doc/
├── foo/
| └── files/
| ├── config1.yml
| └── config2.yml
| ├── init.sls
| └── meta/
| ├── collectd.sls
| ├── heka.sls
| ├── iptables.sls
| ├── sensu.sls
| └── iptables.sls
| └── orchestrate/
| ├── init.sls
| ├── role1.sls
| └── role2.sls
| ├── _common.sls
| ├── role1.sls
| └── role2/
| ├── init.sls
| ├── service.sls
| └── more.sls
├── metadata/
| └── service/
| ├── role1/
| | ├── deployment1.yml
| | └── deployment2.yml
| └── role2/
| └── deployment3.yml
├── test/
├── Makefile
└── metadata.yml
_grains
optional grain modules
_modules
optional execution modules
_states
optional states modules
debian
APT package metadata
foo/files/
configuration files
foo/init.sls
allows the node catalog to be role agnostic by including roles when corresponding pillar.service.role[1|2] is defined
foo/meta/
declaration to support log, metric gathering, monitoring, firewalling, and documentation
foo/orchestrate/
information to orchestrate the deployment
foo/role1.sls
actual salt state resources that enforce service existence by installing pkg, configuring and starting it.
foo/role2/init.sls
used with more complex roles, uses further conditions to limit the inclusion of unecessary stuff.
foo/role2/service.sls
where oackage gets installed, configured and started for role2
metadata/service
reclass metadata
test
currently only syntax checking
metadata.yml
formula description, version and repository
services and roles
On a minion you can check services and roles
salt-call grains.item services
salt-call grains.item roles
reclass metadata files
Each of the files stored under metadata/service
serve as default reclass metadata for a given deployment.
Each role can have several deployments:
metadata/service/server/local.yaml
metadata/service/server/single.yaml
metadata/service/server/cluster.yaml
You can use parameters like ${_param:cluster_node01_hostname}
which will be interpolated at reclass merge time from the node declaration.
Testing
Testing your formula is crucial to insure it will work in different environments, but it could be tedious to provision so many operating system, install Salt, converge the States, run some tests and report the results. But don’t freak out, it’s possible to automate this workflow to insure tests are run easily and frequently.
Sometime it’s fair to recognize when other do things right, when it comes to Formula testing, test-kitchen, an awesome tool by Fletcher Nichol, from the Chef ecosystem seems to be the de-facto standard.
A provisionner, kitchen-salt has been created for Salt.
Lets use Test Kitchen to perform a suite of tests against a State by converging on a VM automatically provisionned by Vagrant.
Installation
To use this tool, you’ll need
- Salt
- Ruby 2.0+
- Git
- Vagrant
- VirtualBox or VMware workstation/Fusion (but with a non free driver for Vagrant)
- test-kitchen–1.2.1
- kitchen-vagrant - test Kitchen Driver for Vagrant.
Install Vagrant and VirtualBox or VMware Fusion/Workstation
In my case I’ll be using VMware Fusion as the backend for Vagrant using the following command to install the driver
$ vagrant plugin install vagrant-vmware-fusion
Installing the 'vagrant-vmware-fusion' plugin. This can take a few minutes...
Installed the plugin 'vagrant-vmware-fusion (4.0.12)'!
It’s been a while seen I’ve touched a license file, the driver isn’t free so here is the license step, download your license and
$ vagrant plugin license vagrant-vmware-fusion ~/Downloads/license.lic
Installing license for 'vagrant-vmware-fusion'...
The license for 'vagrant-vmware-fusion' was successfully installed!
Verify your installation
$ vagrant plugin list
Try to launch a Vagrant box to see if everything works as expected
$ vagrant init bento/ubuntu-16.04
I had to add the following line in my Vagrantfile to avoid a problem with the bento kernel
config.vm.synced_folder ".", "/vagrant", disabled: true
Provision a VM to see if Vagrant works well
$ vagrant up
After a while you should have a new VM running Ubuntu 16.04
Install Salt, on MacOS X used in our example just run, or consult About SaltStack for other OS.
$ sudo pip install salt
Now create the following Gemfile
source 'https://rubygems.org'
gem "test-kitchen", '>=1.2.1'
gem "kitchen-vagrant"
gem "kitchen-salt", ">=0.0.11"
Install all the dependencies with bundle
$ bundle install
Installing artifactory 2.5.0
Installing mixlib-shellout 2.2.7
Installing mixlib-versioning 1.1.0
Installing net-ssh 3.2.0
Installing safe_yaml 1.0.4
Installing thor 0.19.1
Using bundler 1.13.1
Installing mixlib-install 2.0.1
Installing net-scp 1.2.1
Installing net-ssh-gateway 1.2.0
Installing test-kitchen 1.13.2
Installing kitchen-salt 0.0.24
Installing kitchen-vagrant 0.20.0
Bundle complete! 3 Gemfile dependencies, 13 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
Check if test-kitchen it works
$ kitchen help
kitchen console # Kitchen Console!
kitchen converge [INSTANCE|REGEXP|all] # Change instance state to converge. Use a provisioner to configure one or more instances
kitchen create [INSTANCE|REGEXP|all] # Change instance state to create. Start one or more instances
kitchen destroy [INSTANCE|REGEXP|all] # Change instance state to destroy. Delete all information for one or more instances
kitchen diagnose [INSTANCE|REGEXP|all] # Show computed diagnostic configuration
kitchen driver # Driver subcommands
kitchen driver create [NAME] # Create a new Kitchen Driver gem project
kitchen driver discover # Discover Test Kitchen drivers published on RubyGems
kitchen driver help [COMMAND] # Describe subcommands or one specific subcommand
kitchen exec INSTANCE|REGEXP -c REMOTE_COMMAND # Execute command on one or more instance
kitchen help [COMMAND] # Describe available commands or one specific command
kitchen init # Adds some configuration to your cookbook so Kitchen can rock
kitchen list [INSTANCE|REGEXP|all] # Lists one or more instances
kitchen login INSTANCE|REGEXP # Log in to one instance
kitchen package INSTANCE|REGEXP # package an instance
kitchen setup [INSTANCE|REGEXP|all] # Change instance state to setup. Prepare to run automated tests. Install busser and r...
kitchen test [INSTANCE|REGEXP|all] # Test (destroy, create, converge, setup, verify and destroy) one or more instances
kitchen verify [INSTANCE|REGEXP|all] # Change instance state to verify. Run automated tests on one or more instances
kitchen version # Print Kitchen's version information
.kitchen.yml
Test Kitchen keeps it’s main configuration in .kitchen.yml
, it’s used to tell which platform you want to test, it’s a simple YAML file stored at the root of your formula. To quickly show you how to run tests on a formula lets clone one which already contain such a configuration file in our formula directory
$ cd /srv/formulas
$ git clone https://github.com/planetrobbie/influxdb-formula.git
Cloning into 'influxdb-formula'...
remote: Counting objects: 495, done.
remote: Total 495 (delta 0), reused 0 (delta 0), pack-reused 495
Receiving objects: 100% (495/495), 72.03 KiB | 0 bytes/s, done.
Resolving deltas: 100% (283/283), done.
Checking connectivity... done.
Configure the master configuration, /etc/salt/master
to add your formula directory
file_roots:
base:
- /srv/formulas/influxdb-formula
Test Kitchen configuration file contain
driver:
name: vagrant
provider: vmware_fusion
network:
- ["private_network", { ip: "192.168.33.33" }]
provisioner:
name: salt_solo
formula: influxdb
pillars-from-files:
influxdb.sls: pillar.example
pillars:
top.sls:
base:
"*":
- influxdb
state_top:
base:
"*":
- influxdb
- influxdb.cli
platforms:
- name: ubuntu-16.04
suites:
- name: default
driver
tells Test Kitchen what to use to create the test VM, in this section I’ve added the provider: vmware_fusion or by default Virtual Box is used.
provisioner
details about the provisioner to be used, kitchen-salt gem provides a Test Kitchen provisioner called salt_solo
pillars-from-file
which Pillar data to assign to our minion
platforms
different guest operating systems we’ll test against
suites
collection of attributes & tests to be run in conjunction. By default Test Kitchen store its tests below test/integration
.
To get what exactly get tested, look inside test/integration/default/serverspec
, it contains a description which use the serverspec testing framework
require "serverspec"
+set :backend, :exec
describe service("influxdb") do
it { should be_enabled }
it { should be_running }
end
+influxdb_ports = [8083, 8086, 8088]
for influxdb_port in influxdb_ports do
describe port(influxdb_port) do
it { should be_listening }
end
end
Ready to run the test
$ kitchen test
Test Kitchen will then create an environment to execute our formula in, kitchen-salt will make sure Salt is installed in our VM. Then salt-call will be executed and report its end status.
At the end of the salt-call execution, if everything ran successfully you should see
Service "influxdb"
should be enabled
should be running
Port "8083"
should be listening
Port "8086"
should be listening
Port "8088"
should be listening
Finished in 0.37356 seconds (files took 0.34517 seconds to load)
5 examples, 0 failures
Finished verifying <default-ubuntu-1604> (5m25.64s).
-----> Destroying <default-ubuntu-1604>...
==> default: Stopping the VMware VM...
==> default: Deleting the VM...
Vagrant instance <default-ubuntu-1604> destroyed.
Finished destroying <default-ubuntu-1604> (0m7.45s).
Finished testing <default-ubuntu-1604> (11m32.35s).
If you got a bad red message, you can converge again with
$ kitchen converge
or if convergence were successfull, just run the verification again with
$ kitchen verify
Wow, now you have a great testing framework in place :)
Conclusion
Formulas are the evolution of Salt States and bring modularity, reusability. Formulas are easy to hand off and you’ll have fewer files to manage. States can become complex by themselves and can become messy if you don’t pay attention. Breaking things up is a good practice for readability sake.
As we’ve seen, the next Salt release, Carbon, wil bring globbing capabilities for the dictionary keys and multiple grain matching. The base argument of the filter_by function exist now for while, to more easily merge config related stuff and os dependant information. Lets hope formula conventions and repositories will evolve to benefit from all of this.
A last word, each formula is third-party code running as root on your systems, so you need to be careful to read and understand every formula before applying them to your minions.
Links
- Salt Formulas on github
- tcp cloud Salt Formulas Guidelines
- oopss-salt formulas
- Vagrant
- Test Kitchen
- kitchen-salt
Video
- Salt Conf 14 Salt Formulas and States - talk by Forrest Alvarez
Documentation
- Salt Formulas documention
- Understanding Jinja
- Salt improving Jinja usage
- Installing Test Kitchen
- serverspec resource types