Ansible on an immutable VM

06 February 2019 Marcus GustavssonAnsible , Containers , Docker , Dev-ops

We use a fair bit of Ansible here at DueDil, and as we try to investigate how to best manage workloads that for one reason or another we don’t want in Kubernetes, we've had to do some workarounds that might be helpful for others.

All of our applications are running inside docker containers, so it made sense to look into using Google’s Container OS as a fairly minimal and secured base to base our deployments on.

However, if you optimistically write a task containing something like this:

- name: "Start hello world service"
  docker_container:
    name: "helloworld"
    image: "nginxdemos/hello"
    ports:
      - 80:80
    state: started

Point it at a freshly spun up ContainerOS VM and run it, you will most likely get something like this in response:

TASK [Start hello world service] ***********************************************
fatal: [1.2.3.4]: FAILED! => {"changed": false, "msg": "Failed to import docker or docker-py - No module named docker. Try `pip install docker` or `pip install docker-py` (Python 2.6)"}

Right, silly me, obviously I need to install the required packages first. We'll add the installation instructions:

- name: "Install requirements"
  package:
    name: "python-docker"
    state: present
- name: "Start hello world service"
  docker_container:
    name: "helloworld"
    image: "nginxdemos/hello"
    ports:
      - 80:80
    state: started

Which gives us the output:

TASK [Install requirements] ****************************************************
fatal: [1.2.3.4]: FAILED! => {"changed": false, "msg": "Failed to find required executable equery in paths: /usr/bin:/bin:/usr/sbin:/sbin"}

Right. A bit optimistic here yes. As described in Container OS' documentation most of the filesystems on the OS are read-only for security. Which is great, but less of a help for us.

So, trying to move forward without going the painful route of writing our own ansible module to manage docker containers using only the existing shell command docker, we instead opted for trying to build an environment we could run the Ansible scripts on that contained the necessary libraries.

Ansible generally works by pushing a python file to the remote system (in ~/.ansible/tmp/ directory), then running it using the python interpreter specified in the variable ansible_python_interpreter. So if we were to spin up an environment on the VM inside a docker container (with appropriate mounts) and change the python interpreter Ansible uses to run inside that container, we should in theory get an environment that has all the requirements for Ansible whilst not changing the remote system.

To try this out, we'll do a quick hack:

- name: "Install pip in the docker container"
  shell: "docker run --name ansible_workspace python:alpine sh -c 'pip install docker-py'"
- name: "Save local image"
  shell: "docker ps -a | grep ansible_workspace | cut -f 1 -d ' ' | xargs docker commit"
  register: workspace
- name: "Save Ansible workspace hash"
  set_fact:
    docker_workspace: "{{ workspace.stdout_lines | first | regex_replace('^.*:', '') }}"
- name: "Remove temporary container"
  shell: "docker rm ansible_workspace"
- name: "Start hello world service"
  vars:
    ansible_python_interpreter: "docker run -ti --rm -v /var/run:/var/run:rw -v /home:/home:rw {{ docker_workspace }} python"
  docker_container:
    name: "helloworld"
    image: "nginxdemos/hello"
    ports:
      - 80:80
    state: started

So, first we launch an Alpine version of the Python docker image with the name ansible_workspace and run pip install docker-py in it.
Once it has completed, we find the image hash for the (now exited) container and commit it to an image hash.
We'll take the output of that, remove the hash algorithm to get something docker will take and register it as a fact called docker-workspace
Remove the temporary container.
And finally change the Python interpreter for Ansible to a docker run command mounting /var/run (for that great docker-in-docker action) and /home (to ensure the temporary files Ansible pushes to the remote system are available).

TASK [Install pip in the docker container] *************************************
changed: [1.2.3.4]

TASK [Save local image] ********************************************************
changed: [1.2.3.4]

TASK [Save Ansible workspace hash] *********************************************
ok: [1.2.3.4]

TASK [Whut] ********************************************************************
ok: [1.2.3.4] => {
    "msg": "6f8d3c4db652a25d1b774c7024ca154e19aae259051f245b24a873f8c24b3a8c"
}

TASK [Remove temporary container] **********************************************
changed: [1.2.3.4]

TASK [Start hello world service] ***********************************************
changed: [1.2.3.4]

PLAY RECAP *********************************************************************
1.2.3.4              : ok=6    changed=4    unreachable=0    failed=0

Great success!

The above naturally needs a fair bit of productisation (i.e push the ansible workspace image to our docker repo, ensure we re-use any existing workspaces rather than creating new ones all the time (or cleaning them up at the end of the run), etc.) but hopefully it'll help someone looking for how to get around the issue of an immutable OS environment.

Want to work in a great environment? Join us, we're hiring now.