Building an Email Testing Environment with Vagrant, Dovecot and Travis-CI

In my previous post I introduced the testing suite I created for Fetch. Here I want to go through exactly what needed to be done to put that together. This post is a bit longer, and you really don't need it to take advantage of this package, but it may provide some insight to anyone hoping to do something like this themselves.

For me testing has two major components- it should ease development on the local level by preventing regressions, and it should help me as the maintainer of an open source project by giving me trust in the pull requests that are coming in to me. Travis-CI, an application I've been using extensively for my projects now, makes that last problem much easier as it will run my test suites and tell me right in the pull request if any problems have come up. On the local level I decided to use Vagrant, as it makes adding virtual machines to source control (without adding a huge binary file) really simple.

Setting up Vagrant

Since it's a lot easier to test things locally it makes sense to start developing on Vagrant. The first thing we need to go is create a folder, which we'll simply name "vagrant", in the root of our testing environment. This folder be used by Vagrant to store temporary files when the virtual machines are running. It also contains the configuration file that tells Vagrant how to set things up, which we'll go over here.

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

Every Vagrant file has that same header, so we leave it untouched.

config.vm.box = "precise64"
config.vm.box_url = "http://files.vagrantup.com/precise64.box"

I chose to use the 64bit Ubuntu version Precise Pangolin for the Vagrant box, as that's what Travis is currently using and I wanted to keep things as similar as possible. One of the cool features of Travis is that you can add a URL to the config file so Vagrant can download the base box file automatically if it's not there already, which makes it a lot easier for other people to start testing.

config.vm.synced_folder "../resources", "/resources"

Later I plan on pulling in all of the scripts and email files that are shared between Travis and Vagrant. This resources directory will sit in the same testing folder that Vagrant does. This configuration value tells Vagrant that this folder should be available inside the virtual machine as "/resources".

config.vm.hostname = "tedivm.com"
config.vm.network :private_network, ip: "172.31.1.2"

We need the mail server to always have the same IP address, and it helps to give it a hostname. For the IP address I chose 172.31.1.2 because it's in the private IP space, and it seems to be less used than 192.168 or the 10 blocks. I chose "tedivm.com" as the domain because I own it and figure I can get away with it.

config.vm.provision "shell", path: "Initialize.sh"

We need to tell Vagrant how it should start setting things up. For now we're going to tell it to run the "Initialize.sh" file that exists inside of the vagrant directory. We'll also create this file, although we're going to leave it blank for the time being.

end

Finally, we close up the Vagrant.configure block and save the file.

If you head into the vagrant directory and run "vagrant up", it'll now download the box and create your first virtual server with this configuration. You can run "vagrant ssh" to ssh in, or "vagrant destroy" to shutdown and erase that specific instance.

Installing Dovecot

Now that we have a running virtual machine we need to setup the mail server itself. For this we're going to create a a script, resources/scripts/Provision.sh, to run through all of this for us. We're going to use this instead of the Vagrant specific "Initialize.sh" above because this script is also going to run on Travis.

#!/bin/sh

echo 'Provisioning Environment with Dovecot and Test Messages'

I'm a big fan of knowing what's going on, so I start by having the script echo a message out.

# Install and Configure Dovecot

if which dovecot > /dev/null; then
    echo 'Dovecot is already installed'
else

Since there's a chance this script could be run more than once I run a quick check to see if dovecot is already installed. I had originally run a check on the dovecot version itself to see if the command was there, but Ubuntu hijacked it to give me a friendly message saying which package contained the dovecot command. This is probably for the better as the "which" command really is the cleaner way to do it.

    echo 'Installing Dovecot'

    # Install Dovecot.
    # Pass the -y flag to suppress interactive requests.
    sudo apt-get -qq -y install dovecot-imapd dovecot-pop3d

    # Prepare the local.conf for custom values
    sudo touch /etc/dovecot/local.conf

    # Move Maildir to the users home directory.
    # This keeps things consistent across environments.
    sudo echo 'mail_location = maildir:/home/%u/Maildir' >> /etc/dovecot/local.conf

    # Enable plaintext for testing.
    # This is pretty awful for production environments.
    sudo echo 'disable_plaintext_auth = no' >> /etc/dovecot/local.conf

    # Running tests in isolation requires a lot of connections very quickly.
    sudo echo 'mail_max_userip_connections = 10000' >> /etc/dovecot/local.conf

    # Restart Dovecot so it gets it's new settings.
    sudo restart dovecot
fi

Now that we've messed with the configuration we need to restart dovecot, which is the last thing we need to do before closing out this block.

In order to be useful we need to have a user. I went with the infamous "testuser", giving it a password of "applesauce". Back to our Provision.sh file:

# Create "testuser"

# First check to see if the user already exits
if getent passwd testuser > /dev/null; then
    echo 'testuser already exists'
else
    echo 'Creating User "testuser" with password "applesauce"'
    sudo useradd testuser -m -s /bin/bash

    # Use chpasswd since we can pipe in a new password
    echo "testuser:applesauce"|sudo chpasswd
    echo 'User created'
 fi

We now have a script that will install Dovecot for us and set up our first user. Since we want this to run on our new vagrant box we should add it to our Initialize.sh script-

    #!/bin/sh

    /bin/bash /resources/Scripts/Provision.sh

Now whenever a new Vagrant instance is brought up it'll have Dovecot installed! To test this grab you current mail client (I use Mail.app) and connect using "172.31.1.2" as the host, "testuser" as the user and "applesauce" as the password. If everything went right you should be able to connect to and log into the server.

Creating The Emails

Having an email server is great and all, but it doesn't do much without emails. My first thought was to setup postfix on the test machine so I could send it emails directly. I went rather far with this, even scripting the postfix install process, before I remembered that there are multiple programs that allow you to sync mailboxes over imap.

This has some huge advantages. One is ease- it's a lot easier to setup a mailbox on an existing server than it is to set up a whole server. Another point on the ease factor involves getting emails from external servers- with a virtual server that's not setup to open itself to the internet is not easy. The key selling factor for me though was that I could have an actual public email address associated with this account, although one where I still had to manually sync it up before packaging it and releasing it. This lets me release the email address to the public for when they want to add new emails into the test configuration.

So I went ahead and setup "testuser@tedivm.com" and preceded to email it from a few accounts I had. I added a variety of different email types- with attachments, with the email as a cc, a bcc, with different priorities- and I added some folders, sorted some emails, and added a flags to a few. Then I connected to both my test server and the public account with my mail client (Mail.app) and literally dragged and dropped the emails from one account into the other.

Cheating? Probably. Effective and convenient? Completely.

Saving and Restoring the Emails

Earlier we set up "/resources" on the Vagrant machine to point to our "resources" folder in the project itself. We're going to use that folder to save the emails that we just copied so we can reuse them when the Vagrant machine gets reset.

sudo cp -Rp /home/testuser/Maildir /resources/

Next we update Provision.sh to refresh the Maildir every time it runs:

echo 'Refreshing the test mailbox- this could take a minute.'

sudo stop dovecot
[ -d "/home/testuser/Maildir" ] && sudo rm -R /home/testuser/Maildir
sudo cp -Rp /resources/Maildir /home/testuser/
sudo chown -R testuser:testuser /home/testuser/Maildir
sudo start dovecot

echo 'Test mailbox restored'.

This script deletes the existing Maildir and replaces it with the backed up copy in the resources directory. You may have noticed something different about this block of our Provision.sh script- it's the only one that isn't wrapped in a conditional. If Dovecot already exists we won't reinstall it, but every time we run this script we will reset our user's email.

Wrapping It Up and Preparing for Travis

All of the scripts above should work for Travis just as well as Vagrant, but so far we haven't built anything to actually start them. At the same time we can make the Vagrant start and stop process a bit less manual. For that lets start a new script, SetupEnvironment.sh. This script is ultimately the one that is going to run the whole process, and will be called directly before each test suite is run.

#!/bin/sh

# Save the directory of this file, since it may not be the cwd
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

# If we're in Travis the TRAVIS global will be set.
if [ -n "$TRAVIS" ]; then

    echo 'Travis config not yet written'

    # Copy the resources directory to /resources to match the Vagrant config.
    sudo cp -Rp $DIR/resources /resources

    # Just like Vagrant does with the Initialize.sh script, run Provision.sh
    sudo /bin/bash /resources/Scripts/Provision.sh

    # Uh oh! We need to run this to make Dovecot work over SSL
    sudo /bin/bash /resources/Scripts/SSL.sh

else

    # Since not in travis, lets load up a system with vagrant

    # Vagrant has to be called while inside the right directory
    cd $DIR/vagrant

    # If vagrant is running already, reprovision it so it has fresh email boxes.
    # Grep through the vagrant status output to see if vagrant is running.

    VAGRANTSTATUS=$(vagrant status)

    if echo "$VAGRANTSTATUS" | egrep -q "running" ; then
        # This provisions a box that already exists
        vagrant provision
    else
        # This creates a new box, or restarts a shutdown down.
        vagrant up --provision
    fi
    cd $DIR

fi

Here we do a neat little trick with vagrant to run the provisioning script on an already running machine rather than restarting with a fresh machine. This cuts each refresh down to seconds instead of minutes, which is really important for testing.

Now we need to add a ".travis.yml" configuration file to tell Travis to run SetupEnvironment.sh. The test script will be different for your test suite (typically something like phpunit or another testing suite), but for the sake of testing this package I made sure to add a quick test to see that the server was running.

before_script:
    - ./SetupEnvironment.sh

# Change this to your test script!
script: netstat -tuln | grep 143

SSL and Travis-CI

When testing this setup on Travis-CI for the first time I ran into a rather odd problem- SSL certificates were not being generated for Dovecot. Although the Dovercot package, installed through apt-get, was working just fine on the Vagrant install of Ubuntu I was using, the same version of Ubuntu was not generating the certificate for Travis.

Debugging issues with Travis-CI is not simple, as you generally don't have access to the machine itself. The folks over there were nice enough to give me a test virtual machine to log into, but that didn't help as it was difficult to reset. Ultimately I decided it was easier to just generate the missing certificates directly instead of attempting to fix the underlying install, so I copied a piece of Dovecot's own installer code and modified it to run here. If you read through the SetupEnvironment.sh script close you may have noticed a call to '/resources/Scripts/SSL.sh'-

#!/bin/sh

stop dovecot

STARTPATH=`pwd`

SSL_CERT=/etc/ssl/certs/dovecot.pem
SSL_KEY=/etc/ssl/private/dovecot.pem

echo "Creating generic self-signed certificate: $SSL_CERT"
cd /etc/ssl/certs
PATH=$PATH:/usr/bin/ssl
#MAILNAME=`cat /etc/mailname 2> /dev/null || hostname -f`

FQDN=tedivm.com
MAILNAME=tedivm.com

(openssl req -new -x509 -days 365 -nodes -out $SSL_CERT -keyout $SSL_KEY > /dev/null 2>&1 <<+
.
.
.
Dovecot mail server
$FQDN
$FQDN
root@$MAILNAME
+
) || echo "Warning : Bad SSL config, can't generate certificate."

chown root $SSL_CERT || true
chgrp dovecot $SSL_CERT || true
chmod 0644 $SSL_CERT || true
chown root $SSL_KEY || true
chgrp dovecot $SSL_KEY || true
chmod 0600 $SSL_KEY || true

start dovecot

Conclusion

At this point there was just some house cleaning to do, the biggest of which was to put this all on Github for you fine folk to use. I've added some changes since writing pieces of this up, including adding support for Composer (PHP's awesome package manager), auto shutoff in Vagrant, and cleanup to the code itself. If you have any suggests or requests feel free to fork this project or send an email!