Rock IT

Managing multiple environments with Ansible - best practices

If you ever had a challenge of managing multiple servers, or at least you wanted to keep your infrastructure as a code (and that's a very good idea!), you probably heard about Ansible. Today I want to describe my own set of best practices for easy working with multiple environment configurations.

Short introduction to Ansible

(If you already know ansible, you should pass to Problems)

If I had to describe this tool in one sentence, it would be SSH on steroids :) That's the most important difference between Ansible and other similar tools, like Puppet or Chef - you don't need to install any agents on servers, everything is done using direct connections.

Ansible allows you to describe list of steps, called playbooks, that are executed through SSH on remote machines. They are written in .yaml files, in form of desired effect declarations. Here is a short code sample:

- name: update and upgrade apt-get
  apt: update_cache=yes upgrade=full

- name: install required packages
  apt: name={{ item }} state=present
  with_items:
    - python-pip
    - htop
    - vim
    - ufw
    - curl
    - fail2ban

- name: allow ssh
  ufw: rule=allow name=OpenSSH

Because introduction to Ansible is not the goal of this blog post, I encourage you to take a look at the official documentation or this great tutorial.

Problems

Imagine following situation. Almost two years ago, my previous team leader decided that we should create staging environment. And also virtual one, because why not. We had pretty extensive Ansible playbooks, but totally not prepared for differences between environments. I took that task and tried to crack it, as usual :P Oh boy, it was not a walk in the park. List of challenges that I've faced:

  • Rewriting whole codebase to support multiple configurations
  • Managing Environment-specific variables and secrets
  • Making creating new environment an easy task
  • Different amount of hosts (from 20 VPS on production to just 1 during tests)
  • Managing addresses of services (sometimes DB is on localhost, sometimes totally different DC)
  • Complicated Ansible variable precedence rules (list here)
  • Avoiding repeating myself (DRY!), yaml is not a programming language
  • Making faster testing iterations (it's really frustrating to wait 3 minutes after every change)

My solution

After reading lots of articles, looking into other projects and hard thinking, I came up with following project structure:

project/
├─ hosts/
│  ├─ shared-secrets.yml  # encrypted shared vars
│  ├─ shared-vars.yml     # not encrypted shared vars
│  ├─ production/
│  │  ├─ inventory        # inventory file with definitions of all hosts
│  │  ├─ secrets.yml      # encrypted vars for this environment
│  │  └─ groups_vars/     # same as in standard project
│  │  │  ├─ all.yml       
│  │  │  └─ group.yml
│  │  └─ host_vars/       # same as in standard project     
│  │     └─ host1.yml
│  ├─ staging/            # directory for other environment
│  │  └─ ...
│  ├─ vagrant/
│     └─ ...
├─ roles/                 # all ansible roles used in playbooks
│  ├─ role1/
│  │  ├─ main.yml
│  │  └─ ...
│  └─ role2/
│     └─ ...
├─ tasks/
|  └─ load_vars.yml       # special task for loading vars
├─ ansible.cfg            # ansible config file
├─ requirements.yml       # ansible galaxy requirements
├─ playbook1.yml          # here we put our playbooks
├─ playbook2.yml
└─ playbook3.yml

Explanation:
Each environment - specific data is located inside /hosts/environment_name directory. We keep there hosts addresses, all variables and settings.
Important! There is no other place where we differentiate between environments. If we store every single configuration variable inside that directory, and ensure that every decision in playbooks is based on them, we could easily add new environments. Shared variables should be stored inside hosts/shared_vars.yml and hosts/shared_secrets.yml. Secrets are in the same format as normal variables, but encrypted using ansible-vault encrypt hosts/shared_secrets.yml.

Other directories contain common data, like tasks, roles and playbooks, that makes use of defined values. Important2! You shouldn't access 'hostvars' directly in roles, but rather redefine them in group_vars/all.yml. It's because hosts amount and names are not constant, so adding additional abstraction layer really helps. Let's take a closer look at important files.

Example hosts/production/inventory file:

# we define hosts with name, so we can define variables
# with variable="{{ hostvars['database'].ansible_ssh_host }}"
database ansible_ssh_host=111.111.111.111 
redis ansible_ssh_host=111.111.111.112 

worker1 ansible_ssh_host=111.111.111.113 
worker2 ansible_ssh_host=111.111.111.114 
worker3 ansible_ssh_host=111.111.111.115 

[site]
worker1
worker2

[periodic]
worker3

# you can also define these in 'group_vars/site.yml 
[site:vars]
variable=true
docker=true

Example hosts/vagrant/inventory file:

# notice we have just a single host here!
# that's why we should avoid using 'hostvars' inside playbooks
vagrant ansible_ssh_host=111.111.111.111

[site]
vagrant

[periodic]
vagrant

[site:vars]
variable=false
docker=true

Example production/group_vars/all.yml file:

redis_address: "{{ hostvars['redis'].ansible_ssh_host }}"
database_address: "{{ hostvars['database'].ansible_ssh_host }}"

Example test vagrant/group_vars/all.yml file:

redis_address: "{{ hostvars['vagrant'].ansible_ssh_host }}"
database_address: "{{ hostvars['vagrant'].ansible_ssh_host }}"

Example shared_vars.yml

database_connection: "postgres://{{ database_address }}/db"

Load variables task (we need it due to variables precedence, there's no other way to load them in the way we wants)

- include_vars: "hosts/shared-secrets.yml"
- include_vars: "hosts/shared-vars.yml"
- include_vars: "{{ inventory_dir }}/secrets.yml"

A simple role:

name: install docker if required
package: name=docker state=latest
become: yes
tags: configuration
when: "{{ docker|bool }}"

# database_address can point to a different host on production
# and totally different for tests
name: print database connection string
debug: "{{ database_address }}"

And finally, playbook (notice pre-task where we load our variables in a proper order)

# could also be any group instead `all`, like `site` or `amazon`
- hosts: all
  pre_tasks:
   - include: tasks/load-vars.yml
  roles:
    - role: role1

Example commands:

# run against staging environment
ansible-playbook -i hosts/staging/inventory playbook1.yml

# run against production environment, limiting only to database
ansible-playbook -i hosts/production/inventory --limit database playbook2.yml

# skip certain tags, that you are sure won't change anything
# really helpful for speeding up testing
ansible-playbook -i hosts/vagrant/inventory --skip-tags configuration playbook3.yml

# if you have secrets, remember to add --ask-vault-pass flag

Conclusion

That's it! Using this approach, you can easily create extensible configurations supporting multiple environments. Example project structure is available in my Github repository. I'm using that base for almost every new project and it works very well for me.

There is also a tool called test-kitchen, that allows you to test playbooks against different systems in automated way, using Docker. Looks promising and may be helpful, so I'm sharing link here.

That post is much longer than usual, but I hope it wasn't boring. If you have any questions, ask in comments!

Author image
Warsaw, Poland
Full Stack geek. Likes Docker, Python, and JavaScript, always interested in trying new stuff.