Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Introduction

For the last 6 months, there has been a Cypress happy path e2e test implemented in the LiteFarm codebase and it is set up to run on a GitHub hosted runner on pull requests to the integration branch. The statement coverage for this test is currently just under 60%, and it was the QA team’s goal to progressively increase this coverage metric.

Issues with the current implementation

It has been the experience of both the engineering and QA team that cypress tests that have been stabilised and run consistently on local environments are still flaky when run in the CI/CD pipeline on the github hosted runner. This has made progress increasing test coverage quite challenging.

Some possible solutions to this issue are as follows:

Solution 1

Run cypress tests on the developers local environments. And enforce said local testing by setting up cypress tests to run on a husky pre-push hook

Pros:

  1. Stability: Tests runs have proven more stable on developers local environments

  2. Speed: Tests run faster on developers local environments(there is the added time of setting up and tearing down of the test environment on a CI runner )

  3. Cost: No added cost associated with providing a test environment(Github hosted runner in this case)

  4. Faster feedback: Automated tests can provide feedback to the developers on the quality of their code almost instantly, allowing them to make changes and adjustments as needed.

  5. Reducing errors: Developers may forget to run tests or may accidentally run the wrong tests using an npm script. By running E2E tests on a pre-push hook, the testing process is automated and reduces the likelihood of errors in the testing process

Cons:

  1. Limited test coverage: Running E2E tests on a pre-push hook may not cover all possible scenarios and edge cases that can occur in a production environment. It is important to ensure that the tests cover a wide range of scenarios and edge cases to ensure that the code is thoroughly tested.

  2. Environment inconsistencies: Developers may encounter environment inconsistencies when running automated tests in their local environment, which can lead to false positives or false negatives.

Solution 2

Incorporate E2E tests into the deploy script, this assumes the current flakiness of E2E tests in CI is due to the test environment(the github runner)

Pros:

  1. Immediate feedback: Running E2E tests in the production environment provides immediate feedback on the new code's performance and functionality. This helps identify any issues or bugs that may have been missed during development and ensures that the code is functioning as expected in the production environment.

  2. Improved quality assurance: E2E tests help ensure that the entire application is functioning correctly, including integrations with external systems and services. Running these tests during deployment ensures that the application is fully functional in the production environment, reducing the risk of bugs or errors that could negatively impact users.

  3. Faster resolution of issues: If E2E tests fail during deployment, the GitHub Action workflow can be configured to automatically notify developers and relevant team members. This ensures that any issues can be identified and resolved quickly, reducing downtime and improving the overall user experience.

Cons:

  1. Increased deployment time: Running E2E tests during deployment can increase the time it takes to deploy new code changes. This is because E2E tests typically take longer to run than unit tests or integration tests. This increased deployment time can cause delays in delivering new features or updates to users.

  2. Limited customization: GitHub Actions provide a limited set of customization options for E2E testing. Teams may have specific requirements or preferences for how their E2E tests are run that cannot be accommodated within the constraints of a GitHub Action workflow.

  3. Security: Running E2E tests in the production environment using a GitHub Action workflow may raise security concerns, as it requires giving access to the production environment to external systems.

Solution 3

Optimise the CI runner and test environment to more accurately mimic the production environment

Pros:

  1. Customization: Running E2E tests on a dedicated CI runner provides more flexibility and customization options. Teams can configure the environment, tools, and dependencies to suit their specific needs.

  2. Control: Running E2E tests on a dedicated CI runner provides greater control over the testing process. Teams can manage the resources used for testing, and ensure that external dependencies are available and configured correctly.

  3. Security: Running E2E tests on a dedicated CI runner provides an additional layer of security, as it allows teams to isolate the testing environment from the production environment.

Cons:

  1. Maintenance: Running E2E tests on a dedicated CI runner requires additional maintenance and management overhead. Teams must manage and maintain the infrastructure, which can be time-consuming and costly.

  2. Resource utilisation: Running E2E tests on a dedicated CI runner can consume significant resources, which can be expensive or require additional infrastructure scaling.

  • No labels