E2E tests with mono repositories

Over the past couple of years during which I have been writing software utilising microservices, I found out that writing code that just works is simple. Writing maintainable and testable code, on the other hand, has never been trivial. End-to-end tests, especially, have always been a maintenance nightmare due to potential changes in all dependencies which cannot be tracked down using conventional methods (e.g. by using static type checks).

What if there was a way to decrease the level of dependencies between tested microservices so we are able to write E2E tests efficiently?

Using one programming language for projects

Despite the fact that is not always possible due to different business constraints, reducing the number of programming languages in use allows to:

• define contracts between services in a manner that is verifiable by the type system of the chosen language —which saves a lot of debugging time when writing conventional E2E tests,
• share commonly used utility functions between the services (components),
• execute multiple instances of different services by executing a single file containing dependencies — allows for simpler setup than the old-fashioned E2E tests.

I believe the choice of the programming language for a project should be based on what the project in question is supposed to do and what are the possible options of achieving that goal. It is worth noting that taking testability into account might turn the tide in favour of a different technological stack.

Mono repository

The worldwide push for microservice-based infrastructures replaced monolithic projects and allowed codebases to be split into smaller chunks, usually organised under distinct RCS (Revision Control System) repositories. Self-contained microservice codebases are easily deployed on its own, however, they are prone to:

• inclusion of a lot of copied boilerplate code (project setup and common utilities),
• mismatches in adherence to inter-service API contracts (this might be mitigated by E2E tests),
• project dependencies’ divergence between similar microservices.

The aforementioned nuisances can be dealt with by making a step back and returning to a single RCS repository. Even though the very concept of a mono repository does not require depending on a single programming language for the codebase, there exists a lot of advantages should such a decision be made as different microservices will remain independently deployable but project setup, common utilities and tests (including E2E tests!) will be shared.

The way a mono repository is structured is highly dependent on the infrastructural and architectural choices, including the choice of services, languages, libraries, scopes of tests and deployment pipelines.

Architecture

Provided we have chosen:

• a single programming language for the codebase,
• a mono repository setup of the project,

we can leverage the following architectural patterns:

Inversion of Control

By using this pattern (e.g. implemented by the simple dependency injection or the more advanced strategy design pattern) we gain possibility of writing tests using mocks which can be implicitly passed down to code components. This becomes extremely useful when entire microservices are mocked and tested in one single test file in order to verify a particular flow of information.

Monolithic / Microservice Architecture

Even though we have a mono repository (this is the monolithic part) we can have the shared parts (common setup and utilities) applied to the microservice part of the described architecture. As mentioned before, we are able to write E2E tests that check entire microservices by simply executing their boot-up routines in test files.

Event-driven Architecture

For some projects that require more scalability, we might be forced to consider moving to event-driven systems. Data contracts between microservices (enforced usually by serializers and deserializers) can be shared in the utilities segment of the monolith repository. Functionality of any message broker can be mocked away in tests.

Infrastructure

Traditionally, E2E tests operate on existing pieces of an infrastructure (both test and production instances of various microservices). Testing in this fashion falls prey to at least two fundamental issues:

• pollution of (multiple) databases if the cleanup methods were not correctly written or not executed after the tests have failed,
• there is no control of the process whatsoever (no possibility of checking if certain changes were done unless there’s a direct API exposed to check them or we explicitly inspect the database to verify it).

With leveraging a single repository with a single programming language we could write all the necessary E2E tests for all the required microservices (also, many instances of them, if needed) that would run isolated as a part of a single process. Were we not to mock databases or stream processors or message brokers, we can put the tests inside e.g. a Docker container and connect it to real external systems also existing within other Docker containers.

Databases

Testing database operations is fickle as:

• booting up a test database takes time and resources,
• running parallel tests against a single database might result in some non-deterministic behaviour, including, but not limited to, a different order of operations, non-repeatable reads and eventual consistency effects,
• the test setup might not reflect the production setup in a slightest (limited number of replicas, different network layout, server versions, etc.).

Certain scenarios are either not reliably testable or require a lot of setup to become testable to begin with. Generally, testing SQL-compliant databases should be easier than their NoSQL counterparts.

For most basic testing of backend interactions with a SQL-compliant database using an ORM, a software developer could do the following:

• switch to a in-memory or text database for tests to speed up the running time (please note that this might result in losing the ability to test code that uses non-standard SQL features),
• execute the creation of database schema once,
• for tests executed one by one, truncate all the tables after each test (easy) or execute each test in a transaction that is rolled back after the test finishes (more complex),
• for parallel testing it might be beneficial to have a separate database for each thread.

From the architectural standpoint the following steps are highly recommended:

• using an ORM should shield against differences between different SQL flavours,
• data access objects (a.k.a. repositories) should expect an ORM interface as their dependencies and never expose this interface or structures coming from it (for instance, active records should not be used),
• services that operate on a database should perform their duties only through data access objects (DAOs).

By utilising the aforementioned tips, we can track the entire flow of microservices and their interactions with different databases.

Stream Processors and Message Brokers

Similar to regular database management systems, stream processors and message brokers can be tested by either:

• mocking them away (by mocking the very code components that connect to them) or
• booting them up alongside the executed test setup (one can use Docker and/or Docker-Compose to connect one to the other).

Due to distributed nature of some of these systems, tests that connect to live instances must explicitly await completion of requested operations within a reasonable timeframe — that might pose a challenge when testing systems that require polling to gather the status of operation completion instead of setting up webhooks or publish/subscribe pipelines.

Summary

Today we have learned how to build more manageable E2E tests by:

• switching projects to a mono repository,
• using a single programming language,
• using the Inversion of Control and related software design patterns,
• mocking external services away or providing lighter alternatives in test environments.

An avid reader may see that providing a more sound alternative based on recent developments in software development might result in E2E tests that are easy to write, execute and maintain.