In this blog, I will not detail all existing test strategies, but at least provide a high level understanding of why and when one test strategy makes sense.
Tests are often associated with CI/CD, so let’s start by clarifying what CI/CD is all about.
CI/CD
What everyone usually knows is that CI means Continuous Integration.
What is sometimes not as clear is what Continuous Integration is good for (here is a long list of tasks).
In a few words, CI is often used to:
1. Run some static code analysis.
2. Compile the code in such a way that no local files are present and no locally modified environment can be set.
3. Run some tests.
4. If all is green, trigger some code review.
My recommendation is to start CI from the very first line of code.
CD is usually more obscure; it has actually two meanings:
- Continuous Deployment
- Deploy your artefacts to some test environments
- Test your artefacts automatically
- Continuous Delivery
- You are able to often deliver your artefacts in production.
- Some tests should be applied if you can make sure you will not create side effects to the customer experiences.
- Usually Blue/Green, Canary are good approaches to make sure your system is running without impacting 100% of the users.
Now let’s review the various test approaches and strategies.
Unit tests
Unit testing is the most well known test strategy. It consists of testing all public (potentially protected) methods.
In unit testing we make sure that an isolated portion of code is tested with mocked dependency thus making sure that dependencies cannot fail.
Say A and B are 2 classes and A has a dependency on B.
Unit tests allow us to make sure the intrinsic logic of A and the intrinsic logic of B is working, but it provides no information regarding the possible side effects of B not reacting as A was expecting.
Unit tests:
- Should be kept simple to improve understanding of the code.
- Are technical tests.
- Are very sensitive to refactoring (main disadvantage).
From my own experience: costs of 80% of unit tests coverage = additional 40% effort to the development effort.
So, if a planned development effort of a feature needs 10 days:
- 14 days are needed to have the feature developed with 80% of unit test code coverage.
- However, unit tests make the 14 day deadline more predictable (Without unit tests the real deadline might easily be 14 days too… but without having unit tests… ).
- For any refactoring I need this additional 40% of effort.
Unit tests are usually run as part of the Continuous Integration pipeline.
Integration tests
In this step, some or all the dependencies are no longer mocked. In other words, integration tests will be activating several, but not all parts of the program.
Say A and B are 2 classes and A has a dependency on B.
Say A spawns threads that are all using a shared instance of B.
And say B is not properly thread safe.
The integration test should be designed to make sure A is able to rely on B.
Integration tests – compared to unit tests – are usually a little more robust when it comes to refactoring.
Integration tests:
1. Can hold states
Even though an API is based on a stateless design, this does not mean that the program never stores any state in DB, for example.
2. Are usually more complex than unit tests
Defining a rate of effort to achieve 80% of test coverage (like I provided for unit testing) is not possible because of the potential combinations of states.
3. Usually define a behaviour
Integration tests are usually run as part of the Continuous Integration pipeline, often using Docker and sometimes Docker Compose.
System tests
System tests expect the whole program and its dependencies to be up and running (potentially with mocked third party dependencies).
In the cloud, system tests are sometimes relying on bypassing the API, thus activating the program with inputs that are different to the ones the end user will be using.
The advantage of this approach is that potential misconfiguration of the API, latencies, throughput or connection problems will not break this test, thus allowing us to narrow down some problems.
These tests can be very complex, especially in the case of stateful applications.
Stateful applications seen from the interface usually means a state is stored in an in memory DB. In stateful applications the test must replicate the logic of the system it tests.
In other words:
- If an API leading to a state (for example registerUser returning a userId) is called,
- Then the logic of the test must know what kind of data is stored by the tested system.
- Later, if a second API is called using the userId returned by the first API, then the logic of the test must be able to analyse whether the reply of the second API makes sense (taking into account the various inputs provided to the first API calls).
If the system has a complex business logic, the tests should verify it.
System tests are usually very robust when it comes to refactoring.
They are usually run in:
- the Continuous Integration pipeline using Docker Compose
- the Continuous Deployment pipeline to verify a deployment
End to end tests
End to end tests are designed to verify the system as a whole, including its API and all dependencies.
This is very similar to a system test except that it takes into account side effects like latency, throughput, connection problems, etcetera.
They offer the advantage of testing not only the software, but also all of its dependencies.
Smoke tests are a special example of an end to end test.: tests demonstrate the basic functionality of a system. However, if you say only “end to end” tests, one usually assumes a test suite that is more than smoke tests.
These tests are usually run in the Continuous Deployment pipeline and can be used as monitoring techniques against production.
Behavioral tests
Behavioral tests are usually used to write the following tests in a more readable and maintainable way:
- Integration tests
- System tests
- End to end tests (short running Behavior Driven Development (BDD) tests can be used for monitoring)
It is not good practice to write unit tests with BDD because unit tests are technical whereas BDD describes behavior.
Non regression tests
When a bug occurs, we need to make sure the bug will not occur a second time. For this, once the bug is corrected we need to write a test that:
1. Fails with the old implementation
2. Passes with the new implementation
In the cloud, it is safer to have such bugs covered by end to end tests. However, sometimes the complexity is so high that we need to narrow down to integration or even unit tests.
Performance tests
These tests are based on some tools like Gatling or JMeter.
I usually use a couple of end to end tests for the validation. Such tests are usually run in expensive dedicated environments and on demand.
Learn more about testing
If you would like more information on test categories and their applications, please contact Cloudreach.
