Enemy of the state

Many moons ago, I worked for a bank. When you wanted to manually test something, your options were limited. Creating accounts in the big old mainframe needed a COBOL engineer to manually insert data into the db to create a test account. These accounts were shared around the team on a spreadsheet. The most useful of which … belonged to a non-existent person named `Sandra Brown`.

I’m sure you can imagine happens next, collisions, loads of them, people adding and removing accounts. It was not a pleasant experience. Our integration tests on the other hand, had no problems whatsoever, backed by an embedded MySql database, their state was completely predictable.

Enter end to end tests, sitting at or near the top of the test pyramid, they are often unreliable, one contributing factor to this instability? State. Similar to the `Sandra Brown` problem, we end up creating a series of entities and use them around tests, often causing collisions, leaving junk data behind. Let me ask a question to those who write E2E tests, if I truncated every table in your test db (excluding lookup tables, it’s only fair), how many of your E2E Tests would fail due to be depending on some sort of state in your database.

Running end to end often involves the introduction of unreliable state into out test systems. Let’s look at a common situation. I’m working on an eCommerce rest backend and I need to write an end to end test that verifies that the `/{userId}/orders` endpoint correctly returns a list of the users previous orders.

Poor solution

• Manually create a user, or use one that’s lying around
• Manually setup a collection of orders
• E2E test reads orders via API and asserts on the retrieved data

We’ve introduced 3 `states` to the system and a change to any will break the test

1. The user. Changes to the state of the user outside of your test can interfere with the orders endpoint
2. The orders. Changes to the state of an order can interfere with the orders endpoint
3. The test. The test knows about both the user and the orders it needs to assert against

We can drastically improve the stability of our tests by introducing a few changes.

• Create or allocate a new user each time. 
• Set the state you want to assert against in your test.
• Scrub all data associated with the user. 

Our improved solution:

• Create a new user
• Insert order records into your system
• Execute the test
• Delete both the user and records from the system. 

One of the major complaints against this approach is that the setup stage where information is inserted into a running system is slow and error prone. A partial cause of this instability is the reliance on clients to do both the setup and the execution of our tests. What if we only used the client for execution of our tests, but not the setup. Getting the system into the correct state is not the aim of the test, only a prerequisite step.

Does this make E2E testing completely stable or perfect? Nope. It’s a step closer though.

Reason #23 your acceptance tests are rubbish : Chekovs Gun

No, not that Chekov


[Ripped straight from the wikipedia article]
Remove everything that has no relevance to the story. If you say in the first chapter that there is a rifle hanging on the wall, in the second or third chapter it absolutely must go off. If it's not going to be fired, it shouldn't be hanging there.






It speaks to including unnecessary detail in acceptance criteria. Lets look at a simple example:

Given a list with 5 items
When I add a new Item
Then the list size should increase by 1

When you read this it implies that something special happens when you move between 5 and 6 elements, and if it does, then this criteria is completely valid. If nothing special happens, then it is misleading information. A simpler and clearer criteria would be :

Given a list
When I add a new Item
Then the list size should increase by 1

We don't really care how many items are there at the beginning, only that it increases by 1 when an element is added, the list can start with 0, 1 or more items, it's not important, so it's never mentioned.

Dogfooding Contract to test our internal tools

Our testing framework wouldn't be of any use to anyone else if we weren't willing to use it ourselves .... so, we went about dogfooding contract to test one of our internal tools.

A quick recap on Contract
Contract is an open source testing tool aimed at simplifying the testing of HTTP clients and servers, late stage integration testing is a nightmare that Contract is the perfect solution to. Server facing tests are be driven using a contract that specifies how the server is to react given a HTTP request. Client facing tests are be driven by an identical contract specifying that when a client makes a specific request, they should receive a specific response. With both sides of a client server relationship available for testing at a significantly earlier stage, we can improve the feedback of our testing cycles, and hopefully deliver value that little bit quicker.

What are we trying to test?

We have an application called Autodoc that we're eventually going to deploy to a servlet container. It exposes back end services via a REST API. We've a simple front end that acts as a UI to the back end services. We want the back end to be as stand alone as possible and maintain as high a degree of separation as possible from the front end. To that end, we will be doing no actual end to end testing of the application, we'll be leveraging Contract to test both halves of the application independently.  We use maven as a build tool and will be chaining together a few plugins to do the heavy lifting of our tests.

Step 1 : We use the maven cargo plugin to deploy our application, we override the connection details between the front and back end so we can introduce the contract server & client

Step 2 : Bring up a contract server for the front end to test against. This is the server the front end will actually be communicating with.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

<plugin> <artifactId>contract-maven-plugin</artifactId> <groupId>org.seekay</groupId> <version>${contract.version}</version> <executions> <execution> <id>client_facing_tests</id> <phase>pre-integration-test</phase> <goals> <goal>start-server</goal> </goals> <configuration> <gitSource>${git.url}</gitSource> <port>${back-end-port}</port> <username>${git.username}</username> <password>${git.password}</password> </configuration> </execution> </executions> </plugin>

We're using our new 'start-server' maven goal to bring up a contract server that'll remain active until the maven build finishes.

Step 3 : Execute the front end tests. We're using Cucumber & selenium to test our front end backed by our Contract server. We're executing the tests with the surefire plugin, nothing special to see here. 

Step 4 : Execute the back end tests. We'll be using the Contract client to execute tests against the actual back end. 

<plugin>
    <artifactId>contract-maven-plugin</artifactId>
    <groupId>org.seekay</groupId>
    <version>${contract.version}</version>
    <executions>
        <execution>
            <id>server_facing_tests</id>
            <phase>integration-test</phase>
            <goals>
                <goal>run-client</goal>
            </goals>
            <configuration>
                <gitSource>
                    ${git.url}
                </gitSource>
                <target>http://localhost:${front-end-port}</target>
                <username>${git.username}</username>
                <password>${git.password}</password>
            </configuration>
        </execution>
    </executions>
</plugin>

And we end up with completely independent symmetrical testing of a client server application with Contract.  

Contract is available from Maven Central and docs from github pages  here

Contract is open source 

Behavioral testing with Contract 0.0.6

A single stateless contract is never going to go very far in testing a HTTP system, which is why with version 0.0.6 of Contact, we're very pleased to announce 2 game changing new features to our framework.

A quick recap on Contract

Contract is an open source testing tool aimed at replacing a companion system in a HTTP client or server relationship so that shared specifications can be used to unit, integration or acceptance level test to ensure that both systems are on the same page. Server facing tests can be driven using a contract that specifies how the server is to react given a HTTP request. Client facing tests can be driven by an identical contract specifying that when a client makes a specific request, they should receive a specific response. With both sides of a client server relationship available for testing at a significantly earlier stage, we can improve the feedback of our testing cycles, and hopefully deliver value that little bit quicker.

Setup 

Being able to specify a single request response as part of a contract can only test so much of a system. Multiple requests are often required to get the system into a state where assertions can be made. For example, to test read functionality of a system, something must be there to be read. In 0.0.6 we've introduced a setup block to our contract. This allows for specification of any number of setup calls to be made to ensure the system is in the correct state before the contract is actually verified.

The JSON below is an example of the create then read contract applied to our key value example system.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

{ "info" : { "details": "The server should return a key-value pair for a valid get request" }, "setup": [ { "request": { "method": "POST", "path": "/kv/pair", "headers": { "content-type": "application/json" }, "body": { "key": "weight", "value": "220lbs" } }, "response": { "status": 201, "body": { "key": "weight", "value": "220lbs" } } } ], "request" : { "method" : "GET", "path" : "/kv/pair/weight" }, "response" : { "status" : 200, "body" : { "key": "weight", "value": "220lbs" } } }

This allows us to specify the creation of a key value pair before verifying the read functionality of the system

Variables and state

We introduced wildcards a few releases ago, allowing for specification of values when we didn't know exactly what they were going to be. With this release we're going a step further.

A common use case for testing a HTTP system is creation of a resource and then verification that said resource exists. Up until now, if that resource was assigned a unique id at creation time, we had no way of referring to this id and using it later. We're introducing variables to provide this exact functionality. 

Below is an example similar to the one above, its a contract for the creation of an entity on a server representing an unsecured git repository. The repository will be assigned a unique id at creation time and the id will then be used to verify that the repository information can be read. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

{ "info" : { "details": "The server should accept the details of an unsecured repo and store them" }, "setup" : [ { "request": { "method": "POST", "path": "/services/repositories", "body": { "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } }, "response": { "status": 201, "body": { "id": "${contract.var.positiveNumber.repoId}", "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } } } ], "request": { "method": "GET", "path": "/services/repositories/${contract.var.positiveNumber.repoId}" }, "response": { "status": 200, "body": { "id": "${contract.var.positiveNumber.repoId}", "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } } }

For more information on our variables expressions, have a look at our documentation here. Contract is available from Maven Central

Contract source

Parameterising contracts for interaction driven development

This is a follow on post about Contract, if you've not read any of the previous posts, this should serve as a good introduction. 

Having to specify single contract per file is a little painful, especially if you have a few scenarios to be tested. Lets look at a simple example, imagine I want a contract to verify that an endpoint exists and the correct status code will be returned. 

1 2 3 4 5 6 7 8 9 10 11 12

{ "info" : { "details": "The server should return the correct status for an endpoint" }, "request" : { "method" : "GET", "path" : "/service/car" }, "response" : { "status" : 200 } }

If we wanted to expand on this to include multiple endpoints, we would need to duplicate a lot of configuration. With parameters, we can reuse the same contract multiple times.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

{ "info" : { "details": "The server should return the correct status for an endpoint" }, "parameters" : [ { "endpointName" : "car", "expectedStatus": 200 }, { "endpointName" : "jeep", "expectedStatus": 200 }, { "endpointName" : "bicycle", "expectedStatus": 404 }, { "endpointName" : "pedestrian", "expectedStatus": 404 } ], "request" : { "method" : "GET", "path" : "/service/${contract.parameters.endpointName}" }, "response" : { "status" : "${contract.parameters.expectedStatus}" } }

A client, fed this contract will execute 4 requests, one for each parameter group. A server, fed this contract, will expand it into 4 contracts.

Links:

Github: https://github.com/harmingcola/contract

Google group: https://groups.google.com/forum/#!forum/seekay-contracts

Documentation: http://harmingcola.github.io/contract/

Contract 0.0.4, delivering expressions

This week saw the release of Contract 0.0.4 and the first months stats from maven central. About 30 unique downloads, I'm chuffed if even half of those people who downloaded it actually used it. Ok, on to what we did this week.

Wildcard expressions:

Having to hard code absolutely everything is never going to be very good for testing systems. People don't want to have to fill out contracts in that level of detail. There are also times when people cant predict what will be returned when a request is made to a server. With that in mind, we added the wildcard expression functionality.

We'd dipped our toes into this functionality 2 releases ago with the ${contract.timestamp} in a limited amount. We've expanded on that in our 0.0.4 release to include strings and numbers. We've limited its use to within string fields within JSON bodies for now.

These wildcards can be used to replace parts of the path, query parameters, headers and body. Looking at an example, lets use an online streaming service, and an API endpoint that is used to register new content. Lets write the contract for creating a new TV show.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

{ "request" : { "method" : "POST", "path" : "/webmovies/library/tvshow", "body" : { "name" : "Imprisioned Construction", "year" : "2005", "productionCompany" : "BOX" } }, "response" : { "status" : 201, "body" : { "id" : "${contract.anyNumber}", "name" : "Imprisioned Construction", "year" : "2005", "productionCompany" : "BOX", "addedOn" : "${contract.timestamp}" } } }

This contact specifies that when a POST request is made to the 'webmovies/library/tvshow' url with the body specified. Then the server will respond with the CREATED response code, the same body as was in the request enriched with id and addedOn fields.

We can create a contract for reading a TV show from the service where we know effectively nothing and the test server will fill in the blanks.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

{ "request" : { "method" : "GET", "path" : "/webmovies/library/tvshow/${contract.anyString}" }, "response" : { "status" : 200, "body" : { "id" : "${contract.anyNumber}", "name" : "${contract.anyString}", "year" : "${contract.anyNumber}", "productionCompany" : "${contract.anyString}", "addedOn" : "${contract.timestamp}" } } }

Here any request valid request sent to the server will be given a valid response. 

Want to give it a try? Download our fat runnable jar from here and run it with :

1	java -jar contract-all-0.0.4.jar -s -p 9005 -g https://github.com/harmingcola/webmoviesContracts

Make a GET request to http://localhost:9005/__configure and the contracts loaded into that server will be listed.

Links:
Contract source
Contract docs
Webmovies contracts

Taking contract for a field test with Jenkins and Dashing

I've been working on Contract for a while now, and I'm starting to get that feeling that things are going a little too well on the build front ...... I want to take the project for a field test and see what happens.

The plan
I've been wanting to build a Dashing dashboard for my contract Jenkins builds. And instead of using and out of the box dashboard, I'd develop one myself. So, here our build pipeline :

Our code needs to pass through 5 stages before we're happy with it. However, to alert us to a problem, we want our dashboard to be red for any failure per project.

The Requirements

• I want a Dashing dashboard for my Jenkins builds.
• I want to display one tile per group of jobs.
• Groups should be defined in the dashboard config.
• If any job in the group is red, the group is red.
• If all are green, the group is green.

The rules

• Once i've the contract set, I cant go back to Jenkins. I can only develop the dashboard against the ContractServer. 

The Implementation

Before I can start with the dashboard I need to write my contracts. I have no idea what Jenkins payloads look like or what URLS to get them on. After having a look around I've decided to go with this endpoint /job/contract_build/lastBuild/api/json where 'contract_build' is the name of the job. 

Contracts have been written, one for each of my 5 build jobs for Contract, and another one to prove that the failure case works too. Because my builds are absolutely never never ever red.

Dashing projects aren't maven based, As part of Contract 0.0.3, well be releasing a fat runnable jar for running servers from the command line. Running the jar looks like :

I'm off to develop the dashboard, wish me luck ..... My ability to misspell 'Contract' as 'Contact' knows no bounds. Dashboard didn't take long to write. I know neither ruby nor coffeescript, so I've got that going for me. With the dashboard completed, I've got something that looks like the image below. This is exactly what we expect, all jobs in the contract group are green, and the 'interceptor_build' job in the interceptor group is failing.

Lets plug the dashboard into our real Jenkins and see what clicks.

Perfect!! Our contract group stays green because all the jobs are actually green. The interceptor group turns orange because it doesn't exist on our real Jenkins server. I think this can definitely be marked as a successful field test.

Contract docs are here and contract source is here.

Contract 0.0.3 has just been released to maven central. The stars of this release are certainly the maven plugins and fat jar. And by stars, I mean the only thing in the release