E2E tests with mono repositories

Over the past couple of years during which I have been writing software utilising microservices, I found out that writing code that just works is simple. Writing maintainable and testable code, on the other hand, has never been trivial. End-to-end tests, especially, have always been a maintenance nightmare due to potential changes in all dependencies which cannot be tracked down using conventional methods (e.g. by using static type checks).

What if there was a way to decrease the level of dependencies between tested microservices so we are able to write E2E tests efficiently?

Using one programming language for projects

  • define contracts between services in a manner that is verifiable by the type system of the chosen language —which saves a lot of debugging time when writing conventional E2E tests,
  • share commonly used utility functions between the services (components),
  • execute multiple instances of different services by executing a single file containing dependencies — allows for simpler setup than the old-fashioned E2E tests.

I believe the choice of the programming language for a project should be based on what the project in question is supposed to do and what are the possible options of achieving that goal. It is worth noting that taking testability into account might turn the tide in favour of a different technological stack.

Mono repository

  • inclusion of a lot of copied boilerplate code (project setup and common utilities),
  • mismatches in adherence to inter-service API contracts (this might be mitigated by E2E tests),
  • project dependencies’ divergence between similar microservices.

The aforementioned nuisances can be dealt with by making a step back and returning to a single RCS repository. Even though the very concept of a mono repository does not require depending on a single programming language for the codebase, there exists a lot of advantages should such a decision be made as different microservices will remain independently deployable but project setup, common utilities and tests (including E2E tests!) will be shared.

The way a mono repository is structured is highly dependent on the infrastructural and architectural choices, including the choice of services, languages, libraries, scopes of tests and deployment pipelines.

Architecture

  • a single programming language for the codebase,
  • a mono repository setup of the project,

we can leverage the following architectural patterns:

Inversion of Control

Monolithic / Microservice Architecture

Event-driven Architecture

Infrastructure

  • pollution of (multiple) databases if the cleanup methods were not correctly written or not executed after the tests have failed,
  • there is no control of the process whatsoever (no possibility of checking if certain changes were done unless there’s a direct API exposed to check them or we explicitly inspect the database to verify it).

With leveraging a single repository with a single programming language we could write all the necessary E2E tests for all the required microservices (also, many instances of them, if needed) that would run isolated as a part of a single process. Were we not to mock databases or stream processors or message brokers, we can put the tests inside e.g. a Docker container and connect it to real external systems also existing within other Docker containers.

Databases

  • booting up a test database takes time and resources,
  • running parallel tests against a single database might result in some non-deterministic behaviour, including, but not limited to, a different order of operations, non-repeatable reads and eventual consistency effects,
  • the test setup might not reflect the production setup in a slightest (limited number of replicas, different network layout, server versions, etc.).

Certain scenarios are either not reliably testable or require a lot of setup to become testable to begin with. Generally, testing SQL-compliant databases should be easier than their NoSQL counterparts.

For most basic testing of backend interactions with a SQL-compliant database using an ORM, a software developer could do the following:

  • switch to a in-memory or text database for tests to speed up the running time (please note that this might result in losing the ability to test code that uses non-standard SQL features),
  • execute the creation of database schema once,
  • for tests executed one by one, truncate all the tables after each test (easy) or execute each test in a transaction that is rolled back after the test finishes (more complex),
  • for parallel testing it might be beneficial to have a separate database for each thread.

From the architectural standpoint the following steps are highly recommended:

  • using an ORM should shield against differences between different SQL flavours,
  • data access objects (a.k.a. repositories) should expect an ORM interface as their dependencies and never expose this interface or structures coming from it (for instance, active records should not be used),
  • services that operate on a database should perform their duties only through data access objects (DAOs).

By utilising the aforementioned tips, we can track the entire flow of microservices and their interactions with different databases.

Stream Processors and Message Brokers

  • mocking them away (by mocking the very code components that connect to them) or
  • booting them up alongside the executed test setup (one can use Docker and/or Docker-Compose to connect one to the other).

Due to distributed nature of some of these systems, tests that connect to live instances must explicitly await completion of requested operations within a reasonable timeframe — that might pose a challenge when testing systems that require polling to gather the status of operation completion instead of setting up webhooks or publish/subscribe pipelines.

Summary

  • switching projects to a mono repository,
  • using a single programming language,
  • using the Inversion of Control and related software design patterns,
  • mocking external services away or providing lighter alternatives in test environments.

An avid reader may see that providing a more sound alternative based on recent developments in software development might result in E2E tests that are easy to write, execute and maintain.

Full-stack Software Developer that loves building products.