e2e tests vs contract tests

In order to discuss this I need to define my terms as e2e tests have many meanings. Here I base them upon outside in tests of the system that you support, tested mostly. through the user interface or public api.

For contract tests I mean systems like Pact which ensure that for a given set of api calls a certain response is given.

The system that I mainly work on has a react UI served by graphql endpoints. The e2e tests use Playwright to automate the UI and perform checks. The frontend code calls the real backend code. I have had to mock out some of the external dependencies (we can’t have an e2e test making credit card payments).

The frontend code has unit tests. The backend code has unit tests. The e2e tests ensures that one can talk to the other in a useful manner.

Contract tests can prove that a given contract will still return what is expected where as an e2e test can also check. that the call is actually being made.

e2e tests do have downsides.:

  • Slow
  • Unstable
  • Non specific errors

The main part of the slow is the setting up of the environment to run the tests. Only test enough to prove that the services are connected. Never loop through all inputs in an e2e test as this will be slow and will enhance any statistical failures.

Unstable is that these will break if the UI flow changes. They can also break if any of the external mocks change.

There are many ways for a test to fail so you don’t always get the cause shown.

The only way to overcome all of these is to have a way to run the same setup locally.
e2e tests also force you to understand all of the dependencies that you are using. Keep the boundaries as tight as you can. Sometimes this can be painful if you are consuming details from many services.

Web clients are fundamentally insecure

You can’t trust that a web client has not been compromised.
The only safe bet is to assume that any API that you expose to your web client is being directly used as an API.

The client side javascript code for a site makes great documentation for attacking your server.

A simple `wget -r URL will give you the html, css and javascript of most of the site.

The internal urls are stored in the javascript along with any of the graphql queries that you are using.

Developer tools in the browser plus a simple graphql client tool can give you access to far more than you expect.

HMRC Online Tax Return a review

This is an annual ritual in the Uk if you have been asked to fill one in.
The deadline is end of January but it actually covers the year to the previous April.

You are asked to enter data from various forms you have been sent over the year.
It fails to carry forward simple information from the previous year.
My name has not changed nor has my married status.

My employers name is longer than the maximum length that can be input!
The form is multipart asking you in early steps to determine how much of it you actually need to complete.
It is lacking a “hang on I need to ask someone for one field” option. This means you have to pause the process if you find you are missing a P11D.

Given that there are multiple steps I would like to be able to fill in the capital gains section even if I can’t complete the employee section.

The joke is that they already have all this information. The only important part is making you state that you believe this to be correct.

I don’t want to do all this in the new app that they created for it.

Beyond Equality: Compare

There are lots of ways to check for equality. Funx even defines a protocol for it.

This is useful in a number of situations and combines well with other protocols.

There are many situations in which you don’t merely want to say that two things are different you need to know what is different. For this I typically use a compare function that either returns :ok or a list of Validation errors.

Having a simple contract like this makes them easily composable into rulesets. Rulesets can know when it is appropriate to apply a given compare. Using this approach allows a system to provide actionable feedback. Its more than a computer says no, its a these are the problems that I have found.

This is similar to the changeset that Ecto provides for databases but at the domain model level.

Extracting composable compares and rulesets makes the domain logic explicit and testable. This will make extensions easier.