Flaky tests, the ones that pass one run and fail the next with no real change in the code, are the clearest symptom that a test automation program is in trouble. They rarely start as a tooling problem. A flaky test is what you get when one of five decisions gets made carelessly early in a project: a bad test case, data tied to one environment, the wrong framework, weak script review, or no maintenance plan. Fix those five, and flaky tests mostly disappear. Replace the tool instead, and they come right back.
What Counts as a Flaky Test
A flaky test is any automated test that produces different results, passes one time, fails the next, against the same code and the same conditions. It’s not a test that correctly catches a real bug. It’s a test whose outcome depends on something other than the thing it claims to verify: timing, leftover data, environment state, or a race condition in the script itself.
Teams often treat flaky tests as background noise. Someone reruns the pipeline, it goes green, everyone moves on. That habit is the actual danger. Every flaky test that gets waved off with “rerun it” is a test the team has quietly agreed to stop trusting, and a suite full of tests nobody trusts stops being automation. It becomes a very expensive coin flip.
Why This Keeps Happening
Every automation vendor pitch sounds the same. Faster releases, fewer regressions, a dashboard full of green checkmarks. Teams buy the pitch, buy the license, and six months later the suite is producing flaky tests faster than anyone can triage them, until it gets ignored, half broken, or actively distrusted by the people running it.
Industry surveys on automation ROI keep landing in the same place: most programs never hit the coverage or stability they were funded to deliver. That’s not a licensing problem. Licenses work fine. Flaky tests come from decisions nobody wrote down, and nobody owns.
I’ve sat inside enough of these programs to know the pattern repeats regardless of company size, industry, or tool stack. Here’s where flaky tests actually come from.
Reason 1: Bad Test Cases Are the First Flaky Test You’ll Ever Ship
The mistake: Automating a test case or user story that was never solid to begin with.
Teams treat automation as a translation exercise. Someone hands over a user story with vague acceptance criteria, or a manual test case written three product versions ago that nobody updated, and the automation engineer is told to “script it.” The engineer scripts exactly what’s written, because that’s the job. The result is a fast, reliable, perfectly automated test that verifies the wrong thing, or verifies it inconsistently depending on how the ambiguity gets interpreted that day.
This is the one that stings, because it’s rarely the automation team’s fault and they take the blame anyway. A test case with no clear expected result, no defined data state, and no edge case coverage doesn’t get better because you wrapped it in Selenium or Playwright. It just produces flaky tests faster and more often.
What it costs: False confidence. A green suite that tells leadership everything is fine while the actual defect slips through because the test never checked for it. That’s worse than no automation at all, because now people stop looking.
Do this instead: Automation should never start from a test case that hasn’t passed a review gate. Before a single line of script gets written, the test case or story needs a clear precondition, a clear expected result, and a clear pass or fail boundary. If the manual test case is ambiguous, fix the test case first. Automating ambiguity just makes ambiguity run faster and flakier.
Reason 2: Environment-Tied Data Is the Number One Cause of Flaky Tests
The mistake: Scripts hardcoded or loosely tied to data that only exists in one environment, so they collapse the moment they run anywhere else.
This is the single biggest source of flaky tests I see, and it has almost nothing to do with the automation tool. A script gets built against a specific customer ID, a specific order number, a specific account balance that happened to exist in the test environment on the day it was written. It works. It gets checked in. Three weeks later, that record gets refreshed, migrated, or deleted, and the script fails with an error that has nothing to do with the actual application behavior. That failure gets logged as a flaky test, rerun, and forgotten, when it was never flaky at all. It was a test data management problem wearing a flaky test costume.
Teams then spend hours debugging a failure that isn’t a bug. It’s a data dependency nobody flagged. Multiply that across a suite of a few hundred scripts and you get a QA team that spends more time on test data management than on testing the product.
What it costs: Time, mostly. And trust. Once a team gets burned twice by a “flaky” failure that turns out to be stale data, they stop trusting red results, and a suite nobody trusts is a suite nobody looks at.
Do this instead: Scripts need to create or provision the data they need at the start of the run, not assume it exists. That means test data setup as part of the script, not a manual step someone does once and forgets. A basic test data management strategy- provision on run, tear down after, own your own state- eliminates most environment-tied flaky tests without touching the framework at all. If your scripts can’t run cleanly against a freshly built environment with zero prior state, they’re not automated. They’re scheduled to fail eventually.
Reason 3: The Wrong Framework Makes Every Flaky Test Harder to Diagnose
The mistake: Picking a tool because it’s popular, because a competitor uses it, or because a vendor gave a good demo, without checking whether it fits the actual application under test.
This decision usually gets made by someone who won’t be maintaining the suite. A manager reads that a certain framework is trending, or a vendor demo looks smooth on a sample app, and six months later the automation team is fighting a tool that doesn’t handle the application’s shadow DOM, doesn’t support the mobile hybrid views the product actually ships, or requires a licensing tier the company never budgeted for. Every one of those mismatches shows up later as an intermittent, hard-to-reproduce failure, because the framework is fighting the application instead of testing it.
Framework selection is a due diligence problem, not a preference problem. Before committing, someone needs to test the actual candidate frameworks against the actual application, its real component structure, its real authentication flow, and its real CI pipeline. A proof of concept against a sample to-do app tells you almost nothing about how a framework behaves against a five-year-old enterprise system with iframes, shadow DOM, and a login flow that involves three redirects, which is exactly the kind of environment that produces chronic flaky tests.
What it costs: Sunk cost. Teams that picked wrong rarely admit it early. They keep patching around the tool’s limitations for a year, treating every flaky test as a one-off, before anyone is willing to say the selection was wrong.
Do this instead: Run a two- to three-week proof of concept against your actual application, not a demo app, before you commit budget. Score the framework against your specific technical constraints: your front-end stack, your CI tooling, your team’s existing skill set, and your maintenance capacity. The best framework on paper is the wrong choice if your team doesn’t have the skills to keep it stable.
Reason 4: Poor Script Review Lets Flaky Tests Multiply
The mistake: Treating test automation code as lower stakes than product code, so it never gets the same review rigor.
This is the quiet one, and it’s the most direct producer of flaky tests on this list. Product code goes through pull requests, code review, sometimes pair programming. Automation code, written by the same engineers, often gets merged with a glance or no review at all, because “it’s just test scripts.” That mindset is exactly how a suite ends up full of hardcoded waits, duplicated logic across two hundred scripts, and assertions that check the wrong thing without anyone noticing for months.
A script with a hardcoded five-second wait instead of a proper synchronization check will pass locally and fail intermittently in CI. That’s a flaky test born the day it was written, and nobody caught it because nobody reviewed it. A script with an assertion that checks the page loaded instead of checking the actual business outcome gives you a green result on a broken feature. That’s not a hypothetical. That’s the majority of flaky test tickets I’ve seen closed with “reran and it passed,” which is not a fix; it’s a shrug.
What it costs: Flaky suites. Flaky suites get ignored. Ignored suites stop catching real defects, and the whole investment quietly dies without anyone deciding to kill it.
Do this instead: Automation code needs the same pull request standard as product code. Same reviewers, same checklist, same bar for readability and maintainability. If your product team wouldn’t accept a hardcoded wait or a copy-pasted function in application code, don’t accept it in test code either. Most flaky tests are caught in a five-minute review, not a five-hour debugging session.
Reason 5: No Maintenance Strategy Turns Occasional Flaky Tests Into a Flaky Suite
The mistake: Building the suite and assuming it runs itself once it’s built.
This is the reason most automation programs die slowly instead of failing loudly. Nobody budgets time for maintenance because maintenance isn’t a feature, it doesn’t show up on a roadmap, and it’s the first thing that gets cut when a sprint runs long. The application changes every release. The suite doesn’t get updated at the same pace. Within two or three release cycles, a meaningful chunk of the suite is testing UI elements that moved, flows that changed, or features that got deprecated, and every one of those mismatches surfaces as a new flaky test.
Teams then face a choice nobody likes: spend a sprint catching the suite up, or keep shipping features and let the automation debt grow. Most choose the second option, because features have a deadline and test maintenance doesn’t, until the ratio of flaky tests to trustworthy tests flips and rebuilding the suite looks cheaper than fixing it.
What it costs: The entire investment. This is the reason automation programs get killed and restarted every two to three years instead of compounding in value like they’re supposed to.
Do this instead: Maintenance needs a line item, a percentage of every sprint, and an owner. A reasonable starting point is treating ten to fifteen percent of QA capacity per sprint as standing maintenance time, not something you request when things break. Suites that get maintained continuously keep their flaky test rate near zero. Suites that get maintained in emergency sprints become the thing everyone quietly stops relying on.
The Pattern Underneath All Five
None of these five reasons are technical failures, and none of them are bad luck. They’re decisions made without enough scrutiny, at the exact moment scrutiny would have been cheap. A bad test case costs five minutes to catch in review and five sprints to catch after it’s automated and quietly flaky. A wrong framework choice costs three weeks to evaluate properly and a year to unwind if you skip that evaluation. Every flaky test in your suite right now traces back to one of these five moments. Every one of these failures is cheap to prevent early and expensive to fix late.
What to Do This Week to Get Rid of Flaky Tests
- Pull your last ten test cases before you automate anything else. Check if each one has a clear precondition, expected result, and data state. If you can’t answer those three questions from reading the test case alone, fix the test case before you script it.
- Audit five random scripts in your current suite for hardcoded data. Search for specific IDs, names, or values that look like they belong to one environment. Every hit is a future flaky test waiting for that environment to refresh.
- If you’re mid framework evaluation, stop and run it against your actual application, not a sample app. Two weeks now saves you a year of patching around a tool that doesn’t fit.
- Check your last month of pull requests for test automation code. If review comments are thinner than they are on product code, that’s your gap. Apply the same standard starting this week.
- Pull your last 20 flaky test reruns and sort them by root cause. Data, timing, assertion, or environment. That list tells you exactly which of the five reasons above is hurting you the most right now.
- Put a maintenance percentage on your next sprint plan. Even ten percent, protected and non-negotiable, changes the trajectory of a suite within two quarters.
FAQs on Flaky Tests
What are flaky tests?
A flaky test is an automated test that gives different results, passing one run and failing the next, without any real change in the code or the feature it’s testing. The instability comes from something outside the test’s actual purpose: timing, leftover data, environment state, or a race condition in the script.
How do you handle flaky tests in Selenium or Playwright?
Start by replacing hardcoded waits with proper synchronization, waiting for a specific element state instead of a fixed number of seconds. Then check whether the test depends on data that isn’t provisioned fresh at the start of the run. Most flaky tests in both Selenium and Playwright trace back to one of those two issues, not the framework itself.
Is it worth automating if my test cases are already messy?
No. Fix the test case first. Automating a bad test case just means you get the wrong, and often flaky, answer faster and with more confidence than a manual tester would have given it.
How much of a suite typically needs rework due to environment-dependent data?
It varies by team, but any suite with hardcoded IDs, names, or account data is carrying risk that shows up the moment that environment gets refreshed or migrated. Treat it as a when, not an if.
Can a good framework fix a bad process?
No. A framework is a tool. It executes what you build with it faster and more consistently, including your mistakes. Framework selection matters, but it’s reason three on this list, not reason one.
How do we get leadership to fund maintenance time when it doesn’t produce visible output?
Frame it around the cost of the alternative. A suite that isn’t maintained doesn’t stay flat; it degrades, and the eventual cost of rebuilding it from scratch is far higher than a steady maintenance allocation. Show the trend in flaky test count over time, not just the current state.
What’s the single highest leverage fix on this list?
Review. It’s the cheapest to implement, and it catches problems from all four other reasons before they ship. A reviewer who knows to ask “what data does this depend on” or “is this wait hardcoded” catches a flaky test before it exists.
The Bigger Picture
If there’s one thread running through all five of these, it’s that flaky tests are not a QA inconvenience to be shrugged off with a rerun button. They’re the earliest, cheapest warning sign that something upstream- a test case, a data strategy, a framework fit, a review habit, or a maintenance plan- was never solid. The teams getting real, compounding value from automation are the ones who treat every flaky test as a diagnostic, not a nuisance. Everyone else is going to keep rebuilding their suite every two years and wondering why the tool didn’t fix it.
It was never the tool.
