Enemy of the state

Many moons ago, I worked for a bank. When you wanted to manually test something, your options were limited. Creating accounts in the big old mainframe needed a COBOL engineer to manually insert data into the db to create a test account. These accounts were shared around the team on a spreadsheet. The most useful of which … belonged to a non-existent person named `Sandra Brown`.

I’m sure you can imagine happens next, collisions, loads of them, people adding and removing accounts. It was not a pleasant experience. Our integration tests on the other hand, had no problems whatsoever, backed by an embedded MySql database, their state was completely predictable.

Enter end to end tests, sitting at or near the top of the test pyramid, they are often unreliable, one contributing factor to this instability? State. Similar to the `Sandra Brown` problem, we end up creating a series of entities and use them around tests, often causing collisions, leaving junk data behind. Let me ask a question to those who write E2E tests, if I truncated every table in your test db (excluding lookup tables, it’s only fair), how many of your E2E Tests would fail due to be depending on some sort of state in your database.

Running end to end often involves the introduction of unreliable state into out test systems. Let’s look at a common situation. I’m working on an eCommerce rest backend and I need to write an end to end test that verifies that the `/{userId}/orders` endpoint correctly returns a list of the users previous orders.

Poor solution

• Manually create a user, or use one that’s lying around
• Manually setup a collection of orders
• E2E test reads orders via API and asserts on the retrieved data

We’ve introduced 3 `states` to the system and a change to any will break the test

1. The user. Changes to the state of the user outside of your test can interfere with the orders endpoint
2. The orders. Changes to the state of an order can interfere with the orders endpoint
3. The test. The test knows about both the user and the orders it needs to assert against

We can drastically improve the stability of our tests by introducing a few changes.

• Create or allocate a new user each time. 
• Set the state you want to assert against in your test.
• Scrub all data associated with the user. 

Our improved solution:

• Create a new user
• Insert order records into your system
• Execute the test
• Delete both the user and records from the system. 

One of the major complaints against this approach is that the setup stage where information is inserted into a running system is slow and error prone. A partial cause of this instability is the reliance on clients to do both the setup and the execution of our tests. What if we only used the client for execution of our tests, but not the setup. Getting the system into the correct state is not the aim of the test, only a prerequisite step.

Does this make E2E testing completely stable or perfect? Nope. It’s a step closer though.