Create your own Heroku on EC2 with Vagrant, Docker, and Dokku

Heroku is an awesome component to the developer toolchain and lets you get something up and running easily. For me at least, it's transformed how I think about and architect applications. However, there are also a lot of reasons why you might choose an alternative. Maybe it's speed, control, or maybe you just like tinkering.

For us, the reason was cost--pure and simple. Clearbit has a huge cluster of machines crawling the web, and Heroku would have been prohibitively expensive. As it is, we use EC2 spot instances, scaling up our cluster based on demand at a fraction of the cost of most hosting providers.

Luckily there now exist some great alternatives to Heroku like Docker and Dokku that offer a lot of the benefit without some of the drawbacks.

In this article I'm going to take you step by step through building your own 'mini' Heroku on EC2 using Docker, Dokku and Vagrant. While by no means a replacement to the whole of Heroku (like the instant scaling), it'll get you some of the way there.

Tools

Firstly let's take a look at the tools we'll be using:

Vagrant

Vagrant abstracts building and starting machines behind a simple DSL. Their tagline is: create and configure lightweight, reproducible, and portable development environments. In practice though I've found Vagrant great in production, particularly with its AWS extension.

Docker

Docker provides operating-system level virtualization and isolates your application environment without having the overhead of a virtual machine. Docker images are similar to Heroku slugs, and you can distribute them amongst your machines without having to consider local environmental issues or conflicts.

We're not going to use Docker directly in this tutorial, but rather through an abstraction: Dokku.

Dokku

Dokku is a bunch of shell scripts written around Docker that emulates a lot of the behavior Heroku gives you, such as deploys over Git and auto-detection of your app environment. Indeed Dokku actually uses the same open source buildpacks that Heroku uses.

We're actually going to be using a variation of Dokku in this tutorial, the Dokku Alt project which is a more complete solution than the original Dokku, with plugins covering most use-cases.

Setup

Firstly, let's install Vagrant. Choose the appropriate package for your local OS you're developing on.

Next install the Vagrant AWS Provider:

$ vagrant plugin install vagrant-aws

That's all we need to install locally.

Key Pair

You'll need to create an AWS account and a EC2 Key Pair if you haven't already. Note down the name of the Key Pair, as we're going to need that later.

Security Group

Create a EC2 Security Group, I usually call my Security Groups after the environment they represent, say production. You'll need to at least allow incoming traffic from port 22 (SSH) and port 80 (HTTP).

Environmental Variables

You'll need to fetch your AWS credentials and generate a new Access Key. Set the AWS Access Key ID and Secret Access Key as ENV vars, either locally in a shell, in a dotenv file, or in your bash file.

export AWS_ACCESS_KEY_ID=123  
export AWS_SECRET_ACCESS_KEY=456  

Vagrantfile

Next we're going to create an EC2 image that will be the basis for all of our app servers. We'll use a Vagrantfile to describe our server so we can easily recreate it in the future.

Enter the following in a file called Vagrantfile inside your app's root directory. Here's a curl command for your convenience:

$ curl https://gist.githubusercontent.com/maccman/a2f52d067572dcb71dd3/raw/ad04a9e2acd2191e1b43db5e3b96d9154bb9d97d/Vagrantfile > Vagrantfile

And the full Vagrantfile:

Vagrant::configure('2') do |config|  
  config.vm.define :box1, autostart: false do |box|
    box.vm.box      = 'raring'
    box.vm.box_url  = 'https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box'
    box.vm.hostname = 'example.com'

    box.vm.provider :aws do |aws, override|
      aws.access_key_id     = ENV['AWS_ACCESS_KEY_ID']
      aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
      aws.ami               = 'ami-864d84ee'
      aws.instance_type     = 'm3.xlarge'

      # Override this to your keypair name
      aws.keypair_name      = 'aws'

      # Customize this to the security group you're using
      aws.security_groups   = 'production'

      # To mount EBS volumes
      aws.block_device_mapping = [
        {
          :DeviceName => "/dev/sdb",
          :VirtualName => "ephemeral0"
        },
        {
          :DeviceName => "/dev/sdc",
          :VirtualName => "ephemeral1"
        }
      ]

      override.ssh.username = 'ubuntu'

      # Customize this to your AWS keypair path
      override.ssh.private_key_path = '~/.ssh/aws.pem'
    end

    # To make sure we use EBS for our tmp files
    box.vm.provision "shell" do |s|
      s.privileged = true
      s.inline = %{
        mkdir -m 1777 /mnt/tmp
        echo 'export TMPDIR=/mnt/tmp' > /etc/profile.d/tmpdir.sh
      }
    end

    # To make sure packages are up to date
    box.vm.provision "shell" do |s|
      s.privileged = true
      s.inline = %{
        export DEBIAN_FRONTEND=noninteractive
        apt-get update
        apt-get --yes --force-yes upgrade
      }
    end

    # Install dokku-alt
    box.vm.provision "shell" do |s|
      s.privileged = true
      s.inline = %{
        export DEBIAN_FRONTEND=noninteractive
        echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list
        echo deb https://dokku-alt.github.io/dokku-alt / > /etc/apt/sources.list.d/dokku-alt.list

        apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
        apt-key adv --keyserver pgp.mit.edu --recv-keys EAD883AF
        apt-get update -y
        apt-get install -y dokku-alt
      }
    end
  end
end  

You'll need to set the hostname, keypair_name, security_groups and private_key_path variables to your domain, the name of the security group you configured, the name of your Key Pair, and the path to your Key Pair's private key.

The AMI referenced, ami-864d84ee, is just the default Ubuntu AMI. Although the box_url looks fake, it's actually correct - it's just a blank Vagrant box to coax Vagrant into working with AWS.

Notice that the box we defined is called box1.

Provisioning boxes

Now we're actually going to boot box1 using Vagrant. Run the following inside the directory you defined the Vagrantfile:

$ vagrant up box1 --provider=aws

If all goes well, Vagrant should have booted up your machine on EC2 and run all the configuration shell scripts to set up our environment. If you are prompted to 'open your web browser to finish configuration', press Ctrl-C since we're doing the installation manually.

Vagrant will also rsync your current directory straight over to the server under /vagrant as a convenience.

SSHing into the server

To SSH in your newly provisioned server, run:

$ vagrant ssh box1

Now you can interact with your server as you normally would. You can see the SSH configuration Vagrant is using by running:

$ vagrant ssh-config box1

Note down the IP specified by HostName -- we'll need this later when referencing the box.

Authorizing your public key

Let's go ahead and authorize our public key with the dokku user so we can do deploys. Run:

cat ~/.ssh/id_rsa.pub | vagrant ssh box1 -- sudo sshcommand acl-add dokku ${USER}  

Let's try interacting with dokku to make sure our configuration is set up right. Dokku pipes all SSH input to the dokku executable. In other words you can interact with Dokku, setting configuration vars for example, straight over SSH.

The ec2-box-address referenced below is the IP address you got when running ssh-config.

Run:

$ ssh dokku@ec2-box-address

    apps:disable <app>              Disable specific app
    apps:enable <app>               Re-enable specific app
    apps:list                       List app
    config <app>                    display the config vars for an app
    ....

Deploying

Dokku apps are deployed over Git--the same as Heroku. All you need to do is add a Git remote to the app and push.

Dokku will try and detect your app's language and download the appropriate buildpack. The Heroku node-js-sample is a good place to start.

First add the remotes. Notice the remote branch name node-js-app - Dokku uses this as the application's name.

$ git clone https://github.com/heroku/node-js-sample
$ cd node-js-sample
$ git remote add dokku dokku@ec2-box-address:node-js-app

And then push:

$ git push dokku master
Counting objects: 296, done.  
Delta compression using up to 4 threads.  
Compressing objects: 100% (254/254), done.  
Writing objects: 100% (296/296), 193.59 KiB, done.  
Total 296 (delta 25), reused 276 (delta 13)  
-----> Building node-js-app ...
       Node.js app detected
-----> Resolving engine versions

Since we haven't set up DNS yet, we'll need to configure the application's custom domain to that of our EC2 host. In practice you'll want to point a domain at your EC2 box, and maybe configure a subdomain wildcard. For now, so we can access our server, run:

ssh dokku@ec2-box-address domains:set node-js-app ec2-box-address  

Now you can visit your site!

$ open http://ec2-box-address

Further reading

This is just been a short primer on building your own PAAS, and I encourage you to read all the documentation for Vagrant and dokku-alt. In particular, here's a few things worth looking into:

Env vars

To set env vars use the config:set Dokku command. In this example we're using the dokku executable directly on the server, but you can run this command straight over SSH as before.

dokku config:set APP RACK_ENV=production  

Preboot

You can also instruct Dokku to preboot your server, warming up the application code and preventing any downtime during deployments.

dokku preboot:enable APP  

Multiple servers

Vagrant can support provisioning multiple servers at once, simply use a bit of Ruby to define multiple servers in your Vagrant file.

Vagrant::configure('2') do |config|  
  (1..20).each do |i|
    config.vm.define :"phworker#{i}" do |box|
      # Define box ...
    end
  end
end  

Vagrant can take a regular expression when booting up or provisioning servers, in this case booting up nine servers at once.

$ vagrant up /phworker[1-9]$/

Custom AMIs

At Clearbit we have a custom AMI that all our servers are based off. This means we don't need to build a server from scratch when we first boot it--the box is all ready to go. This allows us to scale with demand quickly.

Vagrant includes a useful little plugin called vagrant-ami which lets you create EC2 custom AMIs:

$ vagrant create-ami image1 --name my-ami --desc "My AMI"

Then you can replace the AMI ID in your Vagrantfile with your custom one.