Sunday, August 7, 2016

Deployments - Ansible and multiple clusters of servers

I have been digging on the internet for a solution to have Ansible variables per cluster of servers (environment in the Bamboo), by cluster variables I mean some sort of inventory vars but a bit easier to maintain.

You just put the code (inventory_vars.py) in the plugin folder - eg plugins\vars_plugins (where plugins folder is at same level with roles)

The plugin will allow you to have the following structure


While the plugin is basically very close to inventory variables concept, I found the original concept very complicated to maintain due to the inventory file format. Using the plugin will allow to use the regular yml dictionaries. The fact that you can have group vars or host vars per cluster is more of a nice to have for me.

In order to use the plugin you will need to have a folder under cluster_vars with exact same name as the original inventory file name.

Eg. if "qa" is my inventory file name, I can place a cluster with same name under cluster_vars. There are a few print statements to help you with debugging in case you run in trouble, if you are annoyed by them just comment the out.


# (c) 2016, Iulius Hutuleac

import os
import glob
from ansible import errors
from ansible import utils

from ansible.errors import AnsibleUndefinedVariable
from ansible.parsing.dataloader import DataLoader
from ansible.template import Templar
from ansible.utils.vars import combine_vars

import ansible.constants as C

def vars_file_matches(f, name):
    # A vars file matches if either:
    # - the basename of the file equals the value of 'name'
    # - the basename of the file, stripped its extension, equals 'name'
    if os.path.basename(f) == name:
        return True
    elif os.path.basename(f) == '.'.join([name, 'yml']):
        return True
    elif os.path.basename(f) == '.'.join([name, 'yaml']):
        return True
    else:
        return False

def vars_files(vars_dir, name):
    files = []
    try:
        candidates = [os.path.join(vars_dir, f) for f in os.listdir(vars_dir)]
    except OSError:
        return files
    for f in candidates:
        if os.path.isfile(f) and vars_file_matches(f, name):
            files.append(f)
        elif os.path.isdir(f):
            files.extend(vars_files(f, name))

    return sorted(files)

class VarsModule(object):

    def __init__(self, inventory):
        self.inventory = inventory
        self.group_cache = {}

    def get_group_vars(self, group, vault_password=None):
        """ Get group specific variables. """

        inventory = self.inventory
        inventory_name = os.path.basename(inventory.src())

        results = {}

        #basedir = os.getcwd()

        basedir = os.path.join(inventory.basedir(),"..")

        if basedir is None:
                # could happen when inventory is passed in via the API
            return
        inventory_vars_dir = os.path.join(basedir, "cluster_vars", inventory_name, "group_vars")

        inventory_vars_files = vars_files(inventory_vars_dir, group.name)
        print("Files for group ", group.name, ":",  inventory_vars_files)

        if len(inventory_vars_files) > 1:
            raise errors.AnsibleError("Found more than one file for host '%s': %s"
                                      % (group.name, inventory_vars_files))

        dl = DataLoader()

        for path in inventory_vars_files:
            data = dict()
            data.update( dl.load_from_file(path) )
            if type(data) != dict:
                raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
            if C.DEFAULT_HASH_BEHAVIOUR == "merge":
                # let data content override results if needed
                results = utils.merge_hash(results, data)
            else:
                results.update(data)

        return results

    def run(self, host, vault_password=None):
        print("Requested files for host ", host)
        return {}


    def get_host_vars(self, host, vault_password=None):
        """ Get group specific variables. """

        inventory = self.inventory
        inventory_name = os.path.basename(inventory.src())

        results = {}

        #basedir = os.getcwd()

        basedir = os.path.join(inventory.basedir(),"..")

        if basedir is None:
            # could happen when inventory is passed in via the API
            return
        inventory_vars_dir = os.path.join(basedir, "cluster_vars", inventory_name, "host_vars")

        inventory_vars_files = vars_files(inventory_vars_dir, host)
        print("Files for host ", host, ":",  inventory_vars_files)

        if len(inventory_vars_files) > 1:
            raise errors.AnsibleError("Found more than one file for host '%s': %s"
                                      % (host, inventory_vars_files))

        dl = DataLoader()

        for path in inventory_vars_files:
            data = dict()
            data.update( dl.load_from_file(path) )
            if type(data) != dict:
                raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
            if C.DEFAULT_HASH_BEHAVIOUR == "merge":
                # let data content override results if needed
                results = utils.merge_hash(results, data)
            else:
                results.update(data)

        return results

Saturday, August 6, 2016

IntelliJ, Ansible, Vaults and Windows ...

How to integrate ansible-vault in IntelliJ under Windows

Software prerequisites:

- install cygwin (I placed it directly on the C: drive)
- install pip inside cygwin: easy_install pip
- inside cygwin install ansible-vault using pip: pip install ansible-vault

Now we have the tools in place, we can switch to the setup:

- create a password file for example under C:\Users\<your user>\ansible_vault_pass

#!/bin/bash
echo -n "MyStrongPasswordSomething"

As we are on Windows, Ansible will detect the file as executable and try to run it in order to obtain the password.

- last create the 2 new external tools in IntelliJ as in the screenshots bellow:




Program:
c:\cygwin64\bin\bash

Parameters example:
-c "/bin/ansible-vault -vvv encrypt $FileName$ --vault-password-file /cygdrive/c/Users/XX/ansible-vault-pass"

Working directory:
$FileDir$

Tuesday, August 2, 2016

DSL plugin security change

For security reasons I have added the possibility to disable DSL processing when processing source code repositories.

If you would like to prevent DSL in a specific job in order to avoid random Bamboo task execution just disable it.

Bamboo Template Plans compatibility change

Starting version 3.1.4 I have made the necessary changes to support older versions of Bamboo up to 5.6.0 !