I’ve been recently looking to improve the way in which I manage updates for the blog. In fact it was something long overdue, that I wanted to do months ago, but it wasn’t until last month that I decided to move away from Linode and into OVH, the reason being purely the competitive price offered, with the guarantee that the quality will be the same (this, thanks mostly to some peers that have been on it for quite some time now).
I took that chance to start looking more in detail into some of the Continuous Integration and Continuous Delivery aspects that I’ve been working with for about the last 5 years in my last team. In regards to maintaining the blog, I was mostly looking for a safe and automated way to be able to push codebase and database updates to the blog, from the comfort of my command line, without having to SSH into a server in order to pull code changes, grab backups manually before a deploy, deploy database updates, etc.
I’m also interested in getting more comfortable with Jenkins and get to know more of the ways in which it can be used, and since I want to leverage it at work, too, I made it a requirement to set up my CD pipeline for the blog with Jenkins. For the actual deployment logic, I wanted to write it in Python, as I want to get more comfortable with the language, too. And yes, I know I can use Ansible / Ansistrano, and that they’re easy to learn and use, but using those were not the goal.
Part of what I’ve come up with, I’ve called it wp_deploy, and it’s open sourced in github for anyone to use. While this has been done from scratch, it’s fair to mention that some of the concepts applied are heavily influenced by the years I’ve had the privilege to work near (virtually speaking) one of the best sysadmins on earth. A quick summary of what the script does:
- Grabs Codebase and database backups before deploying changes.
- Runs database updates after changes from the repository have been retrieved.
- Aborts process at different stages if one of the steps fails (e.g: unable to backup database), to ensure integrity and that site is kept up and running.
- Rolls back to the previous version of the code and database if the database update fails.
- Restart services (apache) deployment has been successful.
- Keeps the backups taken before the last deployment (just in case!).
It’d be incredibly quick and simple to write a simple shell script to update the codebase and the database, but that’s not enough to give me peace of mind. A database update could fail and leave the site in a bad state, and in that case I’d have to go into the server, grab the last backup taken, put it in place, revert to the previous version of the codebase, etc. Too many keystrokes for my taste.
The script requires just a little bit of initial setup, as explained in the github project page, but after that, it’ll do all its business just by executing it with the relevant parameters. In my case, I set up a Jenkins instance that checks for changes in the project repository, and will initiate the script remotely with the relevant arguments. That leaves me with nothing to do but update wordpress plugins or core in my local environment, and push them to the repo. In less than one minute they’ll be up and running on the live version of the blog.
This could be certainly improved in a number of ways, which I may be doing at some point in the future, although right now I’m pretty comfortable with what it does and how it does it. Nevertheless, the improvements, if moving this into a more professional environment:
- Some friends have suggested I could do this with Ansible. I was avoiding it on purpose this time. I just wanted to solve the problem within the problem domain, and not within the solution domain, but moving forward, it’s something I actually want to look into.
- The argument list can be simplified / cleaned a bit. It’s not too bad now, but it can be improved.
- The script requires sudo for a few things due to the logic it uses. I’d like to improve it so that it does not require it. In more advanced setup, the user for an automated deployment could be simply linked with an LDAP account that granted sudo access without password.
- Make it keep the last X builds. Right now it destroys the current LIVE dir and immediately recreates it. It’s a simple mv operation, which takes no time in my case, but for big sites is not the way to go! Plan is to clone the repo on each jenkins build and append a build number in the directory where it’s cloned. Then after all db updates and other actions are applied, symlink the live directory to the latest build. Safer approach, zero downtime.
- Include file permissions hardening as part of the script, just for extra safety.
- It’s python 2 (crowd boos and shouts in anger).
- In case this needed to be scaled to run on multiple servers (e.g: master branch to a dev server, and stage branch to a staging server), Fabric would be quite handy for that, too. Not needed if using Ansible, though.
Resources and links
I learned a lot of new things about Python and Jenkins while working on this. Here’s a list of some useful resources and links, as well as the link to the github project:
- wp_deploy in github.
- Introduction to Jenkins pipelines: https://jenkins.io/solutions/pipeline.
- Introduction to Continuous Delivery: https://continuousdelivery.com.
- A possible approach to do this for Drupal, with Ansible: https://mig5.net/content/using-ansible-deploy-drupal.
- Subprocess management in Python: https://docs.python.org/2/library/subprocess.html.
- Jenkins in Docker: https://engineering.riotgames.com/news/putting-jenkins-docker-container.
- Jenkins in Docker – (Quicker and simpler, useful for tests): https://wiki.jenkins.io/display/JENKINS/Installing+Jenkins+with+Docker.
- Passwordless sudo for specific user and command. Be careful here, folks: https://stackoverflow.com/a/11880102.
- Simple and robust way of parsing script arguments on Python: https://docs.python.org/3/library/argparse.html.
- Fabric project: http://www.fabfile.org/.