Post Deployment Smoke Testing Open Source Project

I have just released a new open source project called the Post Deployment Smoke Tester. This project has been a collaborative effort between a number of fellow developers in response to a typical enterprise software development problem, getting rapid feedback on the success of your deployments into production.

This tool has literally saved my teams bacon a number of times, and maybe it can help you too.

The credits for this toolset are as follows:

Open Source Release : Stephen Haunts

Original Author : Hugh Phoenix-Hulme

Contributors : Stephen Haunts, Jon-Paul Flood, Daniel Steele, Oliver Sintim-Aboagye, Mark Jones

The all Familiar Problem

For anyone working in an Enterprise Software environment, I am sure you have all encountered the same problems. You finish your development and testing. You package up a release and prepare all the release documentation for the operations team and submit the change request into your Change Request system for approval. Then the release is deployed during your standard outage/change window, normally at an ungodly hour of the night.

Smoke Test Config Editor - Results

Smoke Test Config Editor – Results

The people who do the deployments generally have no knowledge about what they are deploying, they will just follow the change instructions that you have written. You will normally only really know if the release has been a proper success once your users get their hands on it the next day. In some businesses that can be a big risk if an outage or problem with the system has a financial or customer service impact.

A Solution

What you ideally need is a very simple way for the release team to perform some standard checks to make sure everything is wired up and configured correctly. It is even better if you can automate these tests so they can be repeated by support teams when investigating an incident.

That is where this Post Deployment Smoke Testing Tools comes in. Don’t just take my word for it though. This is what Jez Humble and David Farley, authors of the book Continuous Delivery and Thoughtworks engineers have to say on the subject.

When you deploy your application, you should have an automated script that
does a smoke test to make sure that it is up and running. This could be as simple
as launching the application and checking to make sure that the main screen
comes up with the expected content. Your smoke test should also check that any
services your application depends on are up and running—such as a database,
messaging bus, or external service.

The smoke test, or deployment test, is probably the most important test to
write once you have a unit test suite up and running—indeed, it’s arguably even
more important. It gives you the confidence that your application actually runs.
If it doesn’t run, your smoke test should be able to give you some basic diagnostics
as to whether your application is down because something it depends on is not

Jez Humble and David Farley : Continuous Delivery – Addison-Wesley Professional

You may be lucky and work in a fully agile development shop where you do full continuous deployments directly into production. If so, I salute you. This tool is still very valuable as you can ensure the tests execute after each automated deployment and still get the benefits of this rapid feedback.

Smoke testing is there to give you rapid feedback on the success of a release. If you know instantly that there has been a deployment problem, the engineer doing the release can contact someone to get it fixed or roll back before it impacts the business.

When deploying into production, there are all manor of things that can happen that you didn’t expect, like firewall problems, database passwords being entered incorrectly, MSI installers going crazy (I hate it when they do that).

This smoke testing tools allows your development team to construct a list of test steps using a graphical interface. The tool is very easy to use and you can even use the tool to execute the tests. There is also a command line test runner that can be scripted as part of your deployment.

The whole system is extensible so that if there is not an included test type that you need, you can easily develop your own.

What Sort of Things Can You Test

Out of the box you can perform basic tests like the following :

    • Check certificates exist.
    • Check COM objects are registered.
    • Check SQL Server connection strings.
    • Check environment variables are set.
    • Check files exist.
    • Check folders exist.
    • Check HTTP connections return specified result codes.
    • Check minimum disk space requirements.
    • Check registry keys exist.
    • Check elements and attributes exist in XML files.

The system is extensible so it is very easy to add your own custom test types.

What’s Included

The Post Deployment Smoke Tester comes with 2 tools. There is a graphical user interface tool called the Test Configuration Editor for building / executing test chains and there is a command line tool for executing tests. The latter is great if you want to script the test execution. Lets take a little look at each tool.

Smoke Test Configuration Tool

The Test Configuration Tool allows you to set up and configure test scripts based on the available selection of test types. The tool is very easy to use so the test scripts don’t have to only be created by developers. It should be easy for your testing or DevOps team to build scripts.

Smoke Test Config Editor

Smoke Test Config Editor

The test configuration editor also allows you to execute the tests as shown below. When you run the tests you will get a basic traffic light system, similar to testing tools like NUnit. Green means the test passed successfully, and Red means the test failed. If a test does fail you get a message saying why so you can explore further.

Smoke Test Config Editor - Results

Smoke Test Config Editor – Results

Command Line Test Runner

There is also a command line tool that you can use to execute tests. If you run the tool on its own, it will pick up test scripts in that folder and allow you to run them from a menu, as shown below.

Run Smok Tests from  the Command Line

Run Smoke Tests from the Command Line

You can also run the command line tool and specify a particular test to run, as shown below.

Run Smoke Tests from the Command Line

Run Smoke Tests from the Command Line

Want to Contribute?

It is early days for this tool. I would like to see the range of available tests grow significantly. You can contribute in the following ways:

    • Develop a new test types. If you want to submit a new type of test, then what we require are tests that are general enough for anyone to use, so for example, tests for MSMQ, Rabbit MQ, various NoSQL databases. You may need tests that are very specific to your own enterprise, this is fine, but if it is only going to work on your system then it would not really be suitable for the overall project.
    • Help with our unit testing. The original version of this tool was written fairly quickly in response to some problematic deployments, so although there are quite a few unit and integration tests, we can always do better in increasing the code coverage.
    • Documentation. It would be great if people could contribute by writing tutorials and other use case documentation.
    • Use case Examples. If you use the tool in your company, it would be great if you could do a little write up about your particular usage scenario and submit some example test scripts and screen shots.

2 thoughts on “Post Deployment Smoke Testing Open Source Project

  1. Hugh Phoenix-Hulme

    You have two CommandLine example screenshots 🙂
    It’s also worth mentioning that every Test Type has a method that generates example configurations for that test. I wrote the the Command Line Application with the means to create an Example XML file, you should be able to run it with the /? switch to give you a full rundown on usage.
    There may also be some mileage in showing some of the generated XML here too.

    1. Stephen Haunts Post author

      Yeah the 2 screen shots show 2 different things. One is where it picks up all the files from the folder, the other is where you run a test set specifically with the Run command. I am working on more documentation which will be posted soon.

      Good point on the self created examples etc, I will put something more formal out about that.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s