...
For the last 6 months, there has been a Cypress happy path e2e test implemented in the LiteFarm codebase and it is set up to run on a GitHub hosted runner on pull requests to the integration branch. The statement coverage for this test is currently just under 60%, and it was is the QA team’s goal to progressively increase this coverage metric.
Goals
Tests should be refactored to be more flexible (guidance from Mika Saryan). Some goals to aim for with our tests:
The tests should be organized into smaller chunks with specific functionality: create task, create account, create farm, etc
The architecture has to be designed to allow flexibility in how you run the tests
Only run a single language
Allow to run only "fast" tests. You can tag the tests as fast and slow, this allows to run subset of the tests (e.g. when testing a commit vs integration into master)
Use "fixtures" (they may not be called this in Cypress). There is concept of fixtures in automated testing, which allow to do certain pre-requisites before running a test (e.g. create an account). This allows you to run 1 test or 10 tests, but create an account only once.
Eliminate any Sleep or fixed time based Wait. Usually this is the first thing that causes flaky and slow tests
Add profiling. You need to know how long each test takes which will allow to figure out bottlenecks in the test or in the backend
Issues with the current implementation
It has been the experience of both the engineering and QA team that cypress tests that have been stabilised stabilized and run consistently on local environments are still flaky when run in the CI/CD pipeline on the github Github hosted runner. This has made progress increasing test coverage quite challenging.
...
Maintenance: Running E2E tests on a dedicated CI runner requires additional maintenance and management overhead. Teams must manage and maintain the infrastructure, which can be time-consuming and costly.
Resource utilisationutilization: Running E2E tests on a dedicated CI runner can consume significant resources, which can be expensive or require additional infrastructure scaling.
...
Next step | Owner | Due |
|---|---|---|
Explore custom runner pricing and configuration options. | Mwaya | |
Investigate CircleCI’s free-tier’s ability to build our respective Docker containers (custom runner?) | Mwaya |
Suggested next steps
After further consultation with one of the contributors(KK) the following amendments and additions to the current strategy were suggested:
An upgrade to a larger Github hosted runner(https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners ) to explore whether lack of resources is the cause of flakiness of Cypress tests in the CI/CD pipeline
Run the app using Docker, if the above does not yield any positive results
Ensuring that engineers fix tests that become broken
Standardized .env files across developers local environments and test environments
Breaking test suites down into smaller, focused test suites. Aim to keep each test suite focused on a specific area or feature of your application. Avoid creating overly large or complex test suites that test multiple unrelated features. Smaller, focused test suites are easier to manage, debug, and maintain.