Dogfooding Contract to test our internal tools

Our testing framework wouldn't be of any use to anyone else if we weren't willing to use it ourselves .... so, we went about dogfooding contract to test one of our internal tools.

A quick recap on Contract
Contract is an open source testing tool aimed at simplifying the testing of HTTP clients and servers, late stage integration testing is a nightmare that Contract is the perfect solution to. Server facing tests are be driven using a contract that specifies how the server is to react given a HTTP request. Client facing tests are be driven by an identical contract specifying that when a client makes a specific request, they should receive a specific response. With both sides of a client server relationship available for testing at a significantly earlier stage, we can improve the feedback of our testing cycles, and hopefully deliver value that little bit quicker.

What are we trying to test?

We have an application called Autodoc that we're eventually going to deploy to a servlet container. It exposes back end services via a REST API. We've a simple front end that acts as a UI to the back end services. We want the back end to be as stand alone as possible and maintain as high a degree of separation as possible from the front end. To that end, we will be doing no actual end to end testing of the application, we'll be leveraging Contract to test both halves of the application independently.  We use maven as a build tool and will be chaining together a few plugins to do the heavy lifting of our tests.

Step 1 : We use the maven cargo plugin to deploy our application, we override the connection details between the front and back end so we can introduce the contract server & client

Step 2 : Bring up a contract server for the front end to test against. This is the server the front end will actually be communicating with.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

<plugin> <artifactId>contract-maven-plugin</artifactId> <groupId>org.seekay</groupId> <version>${contract.version}</version> <executions> <execution> <id>client_facing_tests</id> <phase>pre-integration-test</phase> <goals> <goal>start-server</goal> </goals> <configuration> <gitSource>${git.url}</gitSource> <port>${back-end-port}</port> <username>${git.username}</username> <password>${git.password}</password> </configuration> </execution> </executions> </plugin>

We're using our new 'start-server' maven goal to bring up a contract server that'll remain active until the maven build finishes.

Step 3 : Execute the front end tests. We're using Cucumber & selenium to test our front end backed by our Contract server. We're executing the tests with the surefire plugin, nothing special to see here. 

Step 4 : Execute the back end tests. We'll be using the Contract client to execute tests against the actual back end. 

<plugin>
    <artifactId>contract-maven-plugin</artifactId>
    <groupId>org.seekay</groupId>
    <version>${contract.version}</version>
    <executions>
        <execution>
            <id>server_facing_tests</id>
            <phase>integration-test</phase>
            <goals>
                <goal>run-client</goal>
            </goals>
            <configuration>
                <gitSource>
                    ${git.url}
                </gitSource>
                <target>http://localhost:${front-end-port}</target>
                <username>${git.username}</username>
                <password>${git.password}</password>
            </configuration>
        </execution>
    </executions>
</plugin>

And we end up with completely independent symmetrical testing of a client server application with Contract.  

Contract is available from Maven Central and docs from github pages  here

Contract is open source 

Behavioral testing with Contract 0.0.6

A single stateless contract is never going to go very far in testing a HTTP system, which is why with version 0.0.6 of Contact, we're very pleased to announce 2 game changing new features to our framework.

A quick recap on Contract

Contract is an open source testing tool aimed at replacing a companion system in a HTTP client or server relationship so that shared specifications can be used to unit, integration or acceptance level test to ensure that both systems are on the same page. Server facing tests can be driven using a contract that specifies how the server is to react given a HTTP request. Client facing tests can be driven by an identical contract specifying that when a client makes a specific request, they should receive a specific response. With both sides of a client server relationship available for testing at a significantly earlier stage, we can improve the feedback of our testing cycles, and hopefully deliver value that little bit quicker.

Setup 

Being able to specify a single request response as part of a contract can only test so much of a system. Multiple requests are often required to get the system into a state where assertions can be made. For example, to test read functionality of a system, something must be there to be read. In 0.0.6 we've introduced a setup block to our contract. This allows for specification of any number of setup calls to be made to ensure the system is in the correct state before the contract is actually verified.

The JSON below is an example of the create then read contract applied to our key value example system.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

{ "info" : { "details": "The server should return a key-value pair for a valid get request" }, "setup": [ { "request": { "method": "POST", "path": "/kv/pair", "headers": { "content-type": "application/json" }, "body": { "key": "weight", "value": "220lbs" } }, "response": { "status": 201, "body": { "key": "weight", "value": "220lbs" } } } ], "request" : { "method" : "GET", "path" : "/kv/pair/weight" }, "response" : { "status" : 200, "body" : { "key": "weight", "value": "220lbs" } } }

This allows us to specify the creation of a key value pair before verifying the read functionality of the system

Variables and state

We introduced wildcards a few releases ago, allowing for specification of values when we didn't know exactly what they were going to be. With this release we're going a step further.

A common use case for testing a HTTP system is creation of a resource and then verification that said resource exists. Up until now, if that resource was assigned a unique id at creation time, we had no way of referring to this id and using it later. We're introducing variables to provide this exact functionality. 

Below is an example similar to the one above, its a contract for the creation of an entity on a server representing an unsecured git repository. The repository will be assigned a unique id at creation time and the id will then be used to verify that the repository information can be read. 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

{ "info" : { "details": "The server should accept the details of an unsecured repo and store them" }, "setup" : [ { "request": { "method": "POST", "path": "/services/repositories", "body": { "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } }, "response": { "status": 201, "body": { "id": "${contract.var.positiveNumber.repoId}", "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } } } ], "request": { "method": "GET", "path": "/services/repositories/${contract.var.positiveNumber.repoId}" }, "response": { "status": 200, "body": { "id": "${contract.var.positiveNumber.repoId}", "name": "kvServerContracts", "url": "https://github.com/harmingcola/kvServerContracts.git" } } }

For more information on our variables expressions, have a look at our documentation here. Contract is available from Maven Central

Contract source

Parameterising contracts for interaction driven development

This is a follow on post about Contract, if you've not read any of the previous posts, this should serve as a good introduction. 

Having to specify single contract per file is a little painful, especially if you have a few scenarios to be tested. Lets look at a simple example, imagine I want a contract to verify that an endpoint exists and the correct status code will be returned. 

1 2 3 4 5 6 7 8 9 10 11 12

{ "info" : { "details": "The server should return the correct status for an endpoint" }, "request" : { "method" : "GET", "path" : "/service/car" }, "response" : { "status" : 200 } }

If we wanted to expand on this to include multiple endpoints, we would need to duplicate a lot of configuration. With parameters, we can reuse the same contract multiple times.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

{ "info" : { "details": "The server should return the correct status for an endpoint" }, "parameters" : [ { "endpointName" : "car", "expectedStatus": 200 }, { "endpointName" : "jeep", "expectedStatus": 200 }, { "endpointName" : "bicycle", "expectedStatus": 404 }, { "endpointName" : "pedestrian", "expectedStatus": 404 } ], "request" : { "method" : "GET", "path" : "/service/${contract.parameters.endpointName}" }, "response" : { "status" : "${contract.parameters.expectedStatus}" } }

A client, fed this contract will execute 4 requests, one for each parameter group. A server, fed this contract, will expand it into 4 contracts.

Links:

Github: https://github.com/harmingcola/contract

Google group: https://groups.google.com/forum/#!forum/seekay-contracts

Documentation: http://harmingcola.github.io/contract/

Contract 0.0.4, delivering expressions

This week saw the release of Contract 0.0.4 and the first months stats from maven central. About 30 unique downloads, I'm chuffed if even half of those people who downloaded it actually used it. Ok, on to what we did this week.

Wildcard expressions:

Having to hard code absolutely everything is never going to be very good for testing systems. People don't want to have to fill out contracts in that level of detail. There are also times when people cant predict what will be returned when a request is made to a server. With that in mind, we added the wildcard expression functionality.

We'd dipped our toes into this functionality 2 releases ago with the ${contract.timestamp} in a limited amount. We've expanded on that in our 0.0.4 release to include strings and numbers. We've limited its use to within string fields within JSON bodies for now.

These wildcards can be used to replace parts of the path, query parameters, headers and body. Looking at an example, lets use an online streaming service, and an API endpoint that is used to register new content. Lets write the contract for creating a new TV show.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

{ "request" : { "method" : "POST", "path" : "/webmovies/library/tvshow", "body" : { "name" : "Imprisioned Construction", "year" : "2005", "productionCompany" : "BOX" } }, "response" : { "status" : 201, "body" : { "id" : "${contract.anyNumber}", "name" : "Imprisioned Construction", "year" : "2005", "productionCompany" : "BOX", "addedOn" : "${contract.timestamp}" } } }

This contact specifies that when a POST request is made to the 'webmovies/library/tvshow' url with the body specified. Then the server will respond with the CREATED response code, the same body as was in the request enriched with id and addedOn fields.

We can create a contract for reading a TV show from the service where we know effectively nothing and the test server will fill in the blanks.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

{ "request" : { "method" : "GET", "path" : "/webmovies/library/tvshow/${contract.anyString}" }, "response" : { "status" : 200, "body" : { "id" : "${contract.anyNumber}", "name" : "${contract.anyString}", "year" : "${contract.anyNumber}", "productionCompany" : "${contract.anyString}", "addedOn" : "${contract.timestamp}" } } }

Here any request valid request sent to the server will be given a valid response. 

Want to give it a try? Download our fat runnable jar from here and run it with :

1	java -jar contract-all-0.0.4.jar -s -p 9005 -g https://github.com/harmingcola/webmoviesContracts

Make a GET request to http://localhost:9005/__configure and the contracts loaded into that server will be listed.

Links:
Contract source
Contract docs
Webmovies contracts

Taking contract for a field test with Jenkins and Dashing

I've been working on Contract for a while now, and I'm starting to get that feeling that things are going a little too well on the build front ...... I want to take the project for a field test and see what happens.

The plan
I've been wanting to build a Dashing dashboard for my contract Jenkins builds. And instead of using and out of the box dashboard, I'd develop one myself. So, here our build pipeline :

Our code needs to pass through 5 stages before we're happy with it. However, to alert us to a problem, we want our dashboard to be red for any failure per project.

The Requirements

• I want a Dashing dashboard for my Jenkins builds.
• I want to display one tile per group of jobs.
• Groups should be defined in the dashboard config.
• If any job in the group is red, the group is red.
• If all are green, the group is green.

The rules

• Once i've the contract set, I cant go back to Jenkins. I can only develop the dashboard against the ContractServer. 

The Implementation

Before I can start with the dashboard I need to write my contracts. I have no idea what Jenkins payloads look like or what URLS to get them on. After having a look around I've decided to go with this endpoint /job/contract_build/lastBuild/api/json where 'contract_build' is the name of the job. 

Contracts have been written, one for each of my 5 build jobs for Contract, and another one to prove that the failure case works too. Because my builds are absolutely never never ever red.

Dashing projects aren't maven based, As part of Contract 0.0.3, well be releasing a fat runnable jar for running servers from the command line. Running the jar looks like :

I'm off to develop the dashboard, wish me luck ..... My ability to misspell 'Contract' as 'Contact' knows no bounds. Dashboard didn't take long to write. I know neither ruby nor coffeescript, so I've got that going for me. With the dashboard completed, I've got something that looks like the image below. This is exactly what we expect, all jobs in the contract group are green, and the 'interceptor_build' job in the interceptor group is failing.

Lets plug the dashboard into our real Jenkins and see what clicks.

Perfect!! Our contract group stays green because all the jobs are actually green. The interceptor group turns orange because it doesn't exist on our real Jenkins server. I think this can definitely be marked as a successful field test.

Contract docs are here and contract source is here.

Contract 0.0.3 has just been released to maven central. The stars of this release are certainly the maven plugins and fat jar. And by stars, I mean the only thing in the release

I'm not the first person to have this problem

I'm going to try to post once a week to document what I'm doing with contract and anything else that happens to cross my mind ...... 

And because these blog posts look better with at least one photo.

Surely I'm not the first person to have this problem ???

This question has saved me more wasted effort and wheel re-inventing than any other. The ability to sit back, look at a problem and acknowledge the fact that people have come before us and that there is something to be learned from them. There's no place were this is truer than the software development industry. So, the next time you go to put finger to keyboard say to yourself "Am I the first person to have this problem??"  Often the answer is no. There's something we can reuse to make our lives a little easier. Then we can concentrate on the truly cool stuff we can build. 

How did we end up like this?? I can only speak for myself. I blame my college education. I'm not speaking ill of the Dublin Institute of Technology, they did a fantastic job of educating me. In college we were taught to do everything from scratch. You need something sorted, you wrote quick sort yourself, arrays we're used for everything and an IDE was the work of the devil. You leave college and you're presented with Java Collections and wonder why you were never taught about them. But that old mentality of 'Do everything yourself' was still there. So every chance you get. Ask yourself that one simple question and see where it leads you. 

Hows contract going?

Grand, ticking over nicely. I spent the weekend hardening my Jenkins pipeline to maven central (it was a little flaky, still is, the kvServer tests are a nightmare to run well), added the ability to handle current timestamps (it was an early feature request for servers that return the time a request was processed at). Allowing for bodies to be specified in native json as opposed to escaped strings. 

I'm going to aim to release once a week, lets see how that goes. 

Contract github

Introducing, Contract 0.0.1

Two years ago, I started asking myself that question "surely, I'm not the first person to have had this problem". I was working on a green field project and was looking for a better way to test the interaction between services that communicate over HTTP. I ended up using a project called Wiremock to replace one of the systems for testing and everything worked quite well. But it wasn't perfect.

A year ago, I found myself in the same position again. This time I stumbled across consumer driven contracts and pact-jvm. We were developing the server at the time and the client was being developed in another department we couldn't dictate their test approach. We build our own contract driven frame work on top of Wiremock and again, it worked quite well. But it wasn't perfect.

I started working on 'Contract' a few months ago. Replacing Wiremock with a embedded tomcat and using the lessons learned from both stalled attempts to build this tool. Contract will provide a contract-server that will respond based on the supplied contracts to a client system and a contract-client that will send requests and validate the responses from a server under development. Combine the two and you've got test symmetry.

After battling with Jenkins for the last 2 days, 'Contract' is now up in maven central. What it currently lacks in features, it makes up for in simplicity.

Github: https://github.com/harmingcola/contract

Google group: https://groups.google.com/forum/#!forum/seekay-contracts

Documentation: http://harmingcola.github.io/contract/

And now that the basics are in place, its feature time. Watch this spot for more features!!!

Testing the test tool

Going back to working on my HTTP contract testing tool, I've basically realized for the most part that I'm done!! Well, the minimal viable version is done. No bells and whistles just yet. So, why the lack of confidence in releasing it?

I've nothing in the region of acceptance tests. Over the last couple of years I've gotten exceedingly comfortable with this safety net to catch my numerous fuck-ups. My project is unit tested to an inch of its life. I'm comfortable that my coverage of 95% is exactly that, and worthwhile at the same time (maybe i'll add some mutation testing at some stage to see how accurate that 95% actually is).

In order to give myself that safety net and feel comfortable releasing my project I went about testing a test tool. Something I've never done before. A bit of background. Contract is a JVM testing tool designed to replace mocks and test clients in HTTP client server tests and provide identical yet separate test suites for both.

If I were testing a web-app, the first port of call would be to try to think like a user and use the system. Why not the same for a test library. I gave myself 2 users. A client and a server. I designed the simplest server I could think of, a server side map, send a key value pair it saves it for later retrieval. Paired with a client to provide java based access to the same server. I wrote the contracts to define expected behaviours. To follow my own best practices, I put the contracts in a third location so they were accessible to both.

The results: things didn't work exactly as i'd expected. My confident lack of confidence paid off again. Time to fix me some bugs!!!!

What went wrong exactly: when I left out the 'Content-Type' header the server responded with a 415. In the contract that defines this behaviour I left the body blank, I wasn't really concerned with what the body sent back, its usually garbage anyway. NullPointerException, bodies in the response of a contract are mandatory apparently.

Some links:
Contract: https://github.com/harmingcola/contract/
KvServer: https://github.com/harmingcola/kvServer
KvClient: https://github.com/harmingcola/kvClient
KvContracts: https://github.com/harmingcola/kvContracts
Docs: http://harmingcola.github.io/contract/index.html