Fork me on GitHub

Ivan-Site.com

Auto-scaling on Amazon EC2 with Ansible

There are several documented ways of setting up autoscaling groups with the help of ansible, including the official ansible docs. EC2 auto-scaling does, however, assume that servers launched with the AMI will be ready to serve traffic, meaning that the AMI has to be pre-baked (with ansible or other tools) and tightly couples code with the AMI. So, you would have to re-build the AMI for every code release, which is less than ideal in certain environments - particularly when deploys happen more often and you have a large number of servers.

I posted how you could re-use the same AMI to bootstrap Chef in an EC2 auto-scaling group in a previous post, and it turns out you can apply the same concepts when bootstrapping nodes with ansible.

Ansible configuration

Lets start with creating and organizing an ansible configuration git repository. The recommended best practices still apply, however, you will also probably need access to your EC2 inventory. To do this we can use the ansible ec2 dynamic inventory plugin. I chose to include the plugin as part of the ansible configuration at the root like so:

|- ec2.ini
|- ec2.py
|- playbook.yml
|- requirements.txt
|- roles/

You can wget ec2.py and sample ec2.ini from:

Be sure to make ec2.py executable. Follow ansible docs on how to configure the plugin, basically it will use your boto config for AWS credentials.

Here's a minimalistic requirements.txt file to later be used when installing the packages through PIP on the server.

ansible==1.7.2
boto==2.34.0

IAM Profile/Role

We also need to create an IAM role that that will be assigned to all instances in your auto-scaling group. It will need to have at least these permissions:

  • read-only access to ansible-key bucket to get access to a private key used to checkout ansible config git repo.
  • ec2:Describe* is needed in order to have read-only access to your EC2 inventory
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:Get*",
        "s3:List*"
      ],
      "Resource": [
        "arn:aws:s3:::ansible-key",
        "arn:aws:s3:::ansible-key/*"
      ]
    },
    {
        "Action":[
            "ec2:Describe*"
        ],
        "Effect":"Allow",
        "Resource":"*"
    }
  ]
}

Launch config

Next, we'll create a launch config that will be able to bootstrap a server with our ansible configuration. The key part of this launch config will be "user-data portion, which is essentially a shell script that will run when the server boots up. Here's what it will need to do:

#!/bin/bash -v
apt-get update
apt-get upgrade
apt-get install -y python-pip git python-dev
# setup private key that will be used to clone git repo
pip install s3cmd==1.5.0-alpha3
s3cmd get s3://ansible-key/id_rsa /root/.ssh/id_rsa
chmod 600 /root/.ssh/id_rsa
(
cat << 'EOP'
bitbucket.org,207.223.240.182 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==
EOP
) > /root/.ssh/known_hosts
# clone ansible config git repo
git clone git@bitbucket.org:{{ my_project }} /root/ansible-config
# install requirements, should contain ansible and boto with their versions frozen
pip install -r /root/ansible-config/requirements.txt
# run ansible-playbook to configure this server
ansible-playbook -i /root/ansible-config/ec2.py -c local -l $(curl http://169.254.169.254/latest/meta-data/public-hostname) /root/ansible-config/playbook.yml
# should now be able to serve traffic
  • RSA private key that will be used to checkout the git repository with ansible configuration will be downloaded from s3 ansible-key bucket
  • git url of the configuration repo also needs to be specified
  • "-l $(curl http://169.254.169.254/latest/meta-data/public-hostname)" argument is needed to limit ansible-playbook to only execute the playbook once
  • this is essentially what ansible-pull does, but I thought I would illustrate it in more detail

Creating the launch config is possible through the AWS web console, cli, or with an ansible playbook. Here's an example cli call:

as-create-launch-config EXAMPLE --image-id ami-b8d147d1 --instance-type m1.large --monitoring-disabled \
    --iam-instance-profile IAM_PROFILE --group EC2_SECURITY_GROUP --user-data-file ansible-user-data.sh

and an example ansible task:

- ec2_lc:
    name: EXAMPLE
    region: us-east-1
    image_id: ami-b8d147d1
    security_groups: EC2_SECURITY_GROUP
    instance_type: m1.large
    instance_profile_name: IAM_PROFILE
    user_data: {{ user_data }}

Auto-scaling group

Lastly, you want to create an auto-scaling group with the launch config and start booting up instances. Once again, this can be done through web console, cli:

as-create-auto-scaling-group EXAMPLE --availability-zones us-east-1a,us-east-1b \
    --launch-configuration EXAMPLE --desired-capacity 2 --min-size 2 --max-size 10 \
    --load-balancers YOUR_ELB_NAME --health-check-type ELB --grace-period 300

or through an ansible task:

- ec2_asg:
    name: EXAMPLE
    launch_config_name: EXAMPLE
    health_check_period: 300
    health_check_type: ELB
    min_size: 2
    max_size: 10
    desired_capacity: 2
    availability_zones: us-east-1a,us-east-1b
    load_balancers: YOUR_ELB_NAME

Summary

Instances that are booted up as part of this auto-scaling group will have ansible automatically bootstrapped using this pull technique instead of ansible's default of push. If you would like to update code or playbook on any of these instances, you can re-run ansible-playbook from the instance using the same command we used to bootstrap it:

cd /root/ansible-config/
git pull
ansible-playbook -i /root/ansible-config/ec2.py -c local \
    -l $(curl http://169.254.169.254/latest/meta-data/public-hostname) /root/ansible-config/playbook.yml

One last note: unlike chef, ansible re-uses EC2 inventory directly. Thus, you don't need to do anything extra in order to synchronize your inventory between your configuration management system and ec2.

Posted Sun 26 October 2014 by Ivan Dyedov in AWS (Amazon Web Services, Ubuntu, ansible, autoscale, Python)