Flaky tests in CI: how to find the cause and fix it
By the QAAnywhere team · Published November 8, 2026
Why flakiness is a career skill
A flaky test passes and fails on the same code. One flaky test is an annoyance; twenty make a team ignore red builds, and then the test suite stops protecting anyone. Fixing flakiness is one of the most valuable things a test automation engineer does, and one of the most common interview topics.
Remote teams depend on their pipelines more than most. On 8 November 2026, of the 99 remote QA listings on QAAnywhere with a full description, 52 mentioned CI/CD and 11 mentioned flaky tests by name. When nobody is in the room to say "that one always fails, ignore it", a trustworthy pipeline is how a distributed team knows whether it can release.
The most common causes
Almost every flaky test falls into one of these groups.
1. Timing and waiting
The classic cause. The test acts before the page, API or database is ready: it clicks a button that is not yet enabled, reads a value before it updates, or asserts on a list still loading.
Fix: wait for a specific condition (element visible and enabled, response received, text present), never a fixed time. Modern tools such as Playwright and Cypress auto-wait for many actions and retry assertions; in Selenium, use explicit waits. A sleep(3) is a flaky test waiting to happen: too long on a fast machine, too short on a busy CI runner.
2. Shared state between tests
Test A creates a user named "test-user"; test B deletes all users; run in a different order or in parallel, one fails. Shared state includes database rows, files, browser storage, caches, feature flags and global variables.
Fix: make every test independent. Each test creates the data it needs (ideally through an API), uses unique names (append a random suffix or timestamp), and cleans up after itself. A good check: does the suite pass when run in a random order, and in parallel?
3. Test data and environments
Tests that depend on data someone created by hand in a shared staging environment break when that person changes it. Tests that assume "there are exactly 10 products" break when someone adds an eleventh.
Fix: create data per test, or seed a known dataset into an isolated environment (for example a containerised database started for the pipeline). Assert on what your test controls.
4. Environment differences
Passes on your laptop, fails in CI. Common reasons: a different browser version, screen size (an element hidden behind a cookie banner on a small viewport), time zone or locale (date formats, daylight saving), slower CPU, missing fonts, or network access to services that are only reachable from your machine.
Fix: run tests locally in the same configuration as CI, ideally the same container image; pin browser versions; set time zone and locale explicitly in the test configuration.
5. Asynchronous back-end work
The UI says "saved", but a background job finishes the work a second later; or an email or event is sent through a queue. Tests that check the result immediately fail some of the time.
Fix: poll for the end state with a timeout (for example, check every half second for up to ten seconds), or test the background process separately at a lower level.
6. Real external services
Calls to payment providers, email services or third-party APIs from tests fail when those services are slow or rate-limit you.
Fix: stub or mock external services in most tests, and keep a few clearly labelled contract or smoke tests against the real service.
7. Genuine bugs
Sometimes the flakiness is the product: a race condition, a caching bug, a real timeout under load. Do not assume the test is wrong. A test that fails one time in fifty may be telling you that one in fifty customers sees an error.
How to investigate a flaky test
- Collect evidence from failing runs. Screenshots, videos, Playwright traces, browser console, network logs and server logs. Configure CI to keep them on failure; without them you are guessing.
- Reproduce the failure rate. Run the test many times in a loop, locally in CI-like conditions and in CI itself. Knowing it fails 3 times in 100 tells you whether a fix worked.
- Run it alone and in the suite. Passes alone but fails in the suite points to shared state or ordering. Fails alone too points to timing, data or environment.
- Change one thing at a time. Slow the CPU, run in parallel, change the order, use a fresh environment. Note what changes the failure rate.
- Fix the cause, then verify. Run the loop again. A fix you cannot measure is a hope.
Retries: a tool, not a fix
Automatic retries (rerunning a failed test once or twice) keep pipelines moving, and most frameworks support them. Used carelessly, they hide real problems, including real product bugs.
A healthy policy:
- allow at most one retry, and report tests that passed only on retry;
- track those "passed on retry" tests as flaky, and fix the worst ones regularly;
- never retry a test that has started failing consistently.
Keeping a suite trustworthy
Teams that keep flakiness low usually agree on a few rules:
- Quarantine. A known-flaky test is moved out of the blocking pipeline into a separate job, with a ticket and an owner, so it cannot block releases or be ignored forever.
- Measure. Track the flaky test rate and the suite's run time on a dashboard.
- Review tests like code. Waits, data creation and independence are checked in pull requests.
- Delete without guilt. A test nobody can make reliable, covering something already tested elsewhere, should go.
Talking about flakiness in interviews
"A test passes locally but fails in CI; what do you do?" is one of the most common automation interview questions. Answer with the investigation steps above, and if you can, tell a real story: what the cause was, how you proved it and what the failure rate was before and after. "Found that two tests shared a hard-coded user; made test data unique; flaky rate dropped from 8% to 0.5%" is exactly the kind of result hiring managers remember.
For other common interview topics, see QA interview questions for remote roles. The roles below are the newest on the board; each listing shows where you can work from.