Writing Tests Got Easier. How Do We Run All of Them?

AI has made it much easier to add tests.

Not long ago, adding tests meant listing cases by hand, preparing fixtures, and writing the same scaffolding over and over. Now I can generate a first pass alongside the implementation and ask AI to find boundary cases I missed. Our test suite started growing at what felt like an exponential rate.

The catch is that only the cost of writing tests went down. The CPU needed to run them, the work required to prepare a database, and the time spent investigating failures did not disappear. Those costs accumulated on every CI run.

For us, that gap showed up as flaky tests. The failures looked like the same timeout, but there were actually two separate causes:

1. Our MongoDB tests were creating the same indexes repeatedly.
2. CI jobs and test workers were competing for CPU and getting throttled.

The symptoms were nearly identical. The problems were not.

As the test suite grew, CI started to wobble

We run web and mobile applications built with TypeScript and MongoDB in a monorepo. Our CI runs on GitHub Actions with self-hosted runners on Kubernetes. To keep infrastructure costs down, we also use spot instances for the nodes that host those runners.

At first, this worked without much trouble. The test suite was small, CI was fast enough, and intermittent failures were rare.

That changed as AI helped us add tests faster. The suite took longer to run, and MongoDB-related tests started hitting unexplained timeouts.

They did not fail consistently. The same code could pass on one run and time out on another. A rerun might turn green. It was a textbook flaky test.

First cause: creating indexes over and over

Our MongoDB tests used Mongo Memory Server. It gave us an environment close to a real MongoDB instance for testing queries and persistence logic, but its setup cost grew along with the suite.

For a while, I considered removing the Mongo Memory Server tests entirely. Replacing them with mocks would be faster and simpler. I wondered whether stabilizing CI meant giving up the confidence of those integration tests.

Once we narrowed down where the time was going, however, Mongo Memory Server itself was not the real problem. Index creation was happening repeatedly during the test run.

That cost was easy to miss when the suite was small. As the number of test groups grew and more workers ran at the same time, index setup accumulated. Under light load it finished before the timeout. When the environment slowed down even slightly, it crossed the line.

The fix was not to remove the MongoDB tests. It was to change how they ran.

We moved the Mongo Memory Server tests into a dedicated Jest group. Instead of recreating expensive indexes in the middle of individual tests, we set them up once when that group started. The initial setup could take the time it needed, while the tests themselves no longer paid the same index cost repeatedly.

The cost did not vanish. We moved it from many small payments to one upfront payment.

That fixed the MongoDB timeouts. I thought we were done.

Then tests unrelated to the database started failing

Later, a different set of timeouts appeared. These tests did not use MongoDB at all. Tests that normally completed comfortably within the limit would occasionally time out in CI.

It was tempting to treat this as more of the same. The error message still said timeout, and rerunning the job often made it pass.

The important difference was that tests with no connection to the database were slowing down too. That pushed us to look beyond the test code and inspect the environment running it.

The decisive clue was CPU throttling.

Second cause: two layers of parallelism

When too many CI jobs landed on the same Kubernetes host, we saw CPU throttling. As a container approaches its CPU limit, the kernel restricts how much CPU it can use. The Kubernetes documentation describes CPU limits as being enforced through throttling.

The key was that parallelism existed at more than one layer.

Concurrent CI jobs × workers per test runner
= actual concurrency

In our case, GitHub Actions ran several test jobs in parallel, and Jest created several workers inside every job. Neither setting looked extreme in isolation. Multiplied together, they told a different story: in the worst case, around 120 test workers could compete for CPU at once.

More parallelism sounds like it should always make a suite faster. Once it exceeds the available CPU, the opposite can happen. Every process gets a smaller slice, while throttling and scheduling delays increase. A test that normally finishes within ten seconds can occasionally cross the timeout without anything being wrong in the test itself.

We capped both CI jobs and test workers

We reduced parallelism at both layers.

For our GitHub Actions matrix, we set max-parallel to 3. This option limits how many matrix jobs can run simultaneously. The behavior is documented in the GitHub Actions documentation.

strategy:
  max-parallel: 3
  matrix:
    # test jobs

Inside each job, we capped Jest at four workers with maxWorkers. In Jest, this setting limits the size of the worker pool used to run tests. The Jest documentation specifically notes that it can be useful to adjust this value in resource-constrained environments such as CI.

export default {
  maxWorkers: 4,
  testTimeout: 10_000,
};

Together, those changes reduced our possible concurrency from roughly 120 workers to 12.

3 GitHub Actions jobs × 4 Jest workers = 12 at most

The numbers 3 and 4 are not universal defaults. The lesson is to stop looking at CI parallelism and test-runner parallelism separately. Whether the stack is GitLab CI with Vitest, Jenkins with another JavaScript test runner, or something else entirely, the multiplication is the same.

We stopped raising timeouts and standardized on ten seconds

Before fixing the underlying issues, our timeout settings had started to drift. Whenever a test failed, one limit became ten seconds, another became thirty.

That can make CI green for a while, but the cause remains. The next burst of host load can break the thirty-second limit too. Longer timeouts also mean waiting longer to discover a test that has genuinely stalled.

After fixing MongoDB index setup and CI parallelism separately, we standardized the default timeout at ten seconds. We could also treat the necessarily slow startup phase separately from the limits applied to individual tests.

The intermittent failures disappeared.

Telling two identical timeouts apart

In hindsight, both problems hid behind the same error message, but their clues were different.

If you run into a similar problem, this is the order I would investigate it:

1. Check whether failures are concentrated in database tests.
2. Look for setup, migrations, schemas, or indexes being recreated for every test.
3. Check whether unrelated tests slow down during the same period.
4. Multiply concurrent CI jobs by workers per test runner to find the real concurrency.
5. Inspect CPU throttling when failures occur.
6. Fix setup and parallelism before raising timeouts.

Running tests now needs as much design as writing them

AI has made tests much cheaper to write. More tests are a good thing. We can cover boundary cases we previously skipped and build a denser safety net around every change.

But the bottleneck has moved. Designing how thousands of tests are isolated, initialized, scheduled, and parallelized is becoming more important than the time it takes to type the test code.

In our case, one timeout message hid two causes. One lived inside the test architecture; the other lived in the infrastructure running it. As test suites grow faster, both need to be treated as part of the same testing system.