r/javascript • • 5d ago

Flashwright: the fastest e2e testing framework, up to 4x faster than Playwright

https://flashwright.dev/

If you're interested in the details, here's a blog post: https://codebeast.io/blog/flashwright

67 Upvotes

30 comments sorted by

39

u/Recoil42 5d ago

TLDR:

The biggest single cost in a lot of suites isn’t Playwright’s code at all. It’s the browser compiling your app’s JavaScript again in every fresh renderer. Playwright gives each test a brand-new browser context, which is great for isolation, but it means every test pays for a cold start of your app.

So the trick that matters most is keeping one context per worker and giving each test a new tab in it, then resetting everything a test could have changed: options, permissions, cookies, every visited origin’s storage (local, session, IndexedDB, Cache Storage, service workers), caches, routes, init scripts, bindings, downloads and the clock. The previous test’s pages are stopped before anything gets cleared, so a late script can’t leak state into the next test. That reset is the part that really has to be right, and it’s what tests/page/recycle-isolation.spec.ts checks.

48

u/dakruzz 5d ago

Doesn't sound safe for parallel testing

23

u/TheThingCreator 5d ago

doesn't even sound like it would work in many contexts. lots of my testing requires fresh browser profiles or controlled browser profiles. i woudlnt be able to work like that.

7

u/DuckDatum 5d ago

I don’t run tests in browsers but, what experience I do have tells me, at minimum, it would add a constant concern that the test is failing potentially because of leaked shared state.

Though, tests which don’t rely on client state could see a bump in speed.

I wonder if this could be designed as an automated optimization, for whenever a test can be programmatically detected as idempotent. Then playwright could do the same thing, no?

5

u/skt84 5d ago

Failing or passing. If I can’t trust a failure is caused by shared state then I can’t trust the passes either.

2

u/Jebble 5d ago

In of our apps specifically (legacy that's being replaced) uses a lot of cookies and opening it in multiple apps is troublesome. This wouldn't ever be able to help us test it.

1

u/dunderball 5d ago

I don't get why this is needed when playwright has very good sharding capabilities

4

u/JimmyKillem 5d ago

Because someone wanted to get online clout from their vibe coded project

2

u/RubbelDieKatz94 🇩🇪 Sr. Fullstack / TS / Py / 10+ years 2d ago

I recently tried sharding - Zero speedup because of runner constraints.

1

u/dunderball 2d ago

Can you elaborate? I've seen time savings as we've increased our shards. It's not linear but it has been nice to not wait around forever for e2e tests

1

u/RubbelDieKatz94 🇩🇪 Sr. Fullstack / TS / Py / 10+ years 2d ago

Imagine you have one runner with a CPU capacity of 0.1. This runner runs your 10 browser tests in 100s.

Now you shard into 2 shards. Each shard downloads cache, artifacts, etc. and runs 5 browser tests. For that it gets 0.05 CPU and takes 120s.

In total, your test run now takes 120s instead of 100s.

Cluttered runners simply play by different rules.

1

u/dunderball 1d ago

Just run 1 shard on 1 runner? Works fine in GitHub and Bitbucket

1

u/RubbelDieKatz94 🇩🇪 Sr. Fullstack / TS / Py / 10+ years 1d ago

That works if you have multiple runners with uncontested CPUs.

We have 2 runners that process 20 MRs at a time.

8

u/thinkmatt 5d ago

Kind of like why jest is slow as balls because they insist on reimporting every js file for each test file

5

u/aicis 5d ago

At certain scale it becomes a problem. No one likes to wait 20 minutes for test suite to finish.

2

u/heavenparadox 2d ago

Speak for yourself. That's break time, baby!

1

u/xcodebeast 5d ago

Thanks!

16

u/fireatx 5d ago

seems cool, similar to vitest's isolate: false config option, but i feel like this could probably just be a PR on the playwright repo adding a new config option. does this need to be an entirely new package?

3

u/xcodebeast 5d ago

Thanks. I agree, that particular optimization could be a PR on playwright repo but I think all the other optimizations + the QoL features I added justified a new package. I might push some PRs to Playwright though.

4

u/fireatx 5d ago

you definitely should put up some PRs! at my work we use playwright in several places, and having these optimizations available would be a game changer. i can't justify replacing playwright entirely, but could definitely make use of these speedups if they were available within playwright.

19

u/Positive_Method3022 5d ago

The strategy can lead to false positives/negatives. Sandboxing test execution by context is a must.

0

u/xcodebeast 5d ago

It depends a lot on the project and what you're testing as u/caboosetp said. Flashwright allows you to turn off each individual optimization so you can use only what actually fits your project.

1

u/caboosetp 5d ago

Depends on what you're testing.

Many of the tests i write are testing static components like navigation which shouldn't depend on what other stuff has done. If not being in a sandbox breaks that, there's a problem anyways. The chance of a false positive is also almost zero. 

Being able to choose to group tests like that to use a resetting context would be fine. Especially if you're just grouping things like static components. 

But I agree the ability to put tests in a sandbox context is also very important, especially for anything dynamic. Unfortunately it's the dynamic stuff that generally has the most tests, so i don't even know if grouping just some things is going to see real gains on most projects

2

u/jobRL 5d ago

In your example about navigation, what in that context are you testing?

12

u/OpiumOpossum 5d ago edited 5d ago

Not sure but this seems like it could have memory leaks if you use listeners, network interceptors or logging. Otherwise, I am pretty sure this is a similar way to how Vitest optimizes their test runner.

1

u/RubbelDieKatz94 🇩🇪 Sr. Fullstack / TS / Py / 10+ years 2d ago

Vitest is slower than Jest btw, in its default config. It has some pretty advanced isolation that makes it very stable, and rather slow.

3

u/Discordbothost 5d ago

there's an isolation: 'strict' option for a fresh context per test. default recycling is per worker, so the concern is state leaking into the next test on that worker, not parallel workers sharing one context. https://github.com/xcodebeast/flashwright#differences-from-playwright

3

u/LukaRozman27 5d ago

beware if you rely on Playwright's built-in tracing, Flashwright skips it entirely to hit those speed claims.

3

u/DarkGhostHunter 5d ago

Tested with Pest and the gains are… negligible. Barely 5%, margin of error.

2

u/tomByrer 5d ago

Neat! How well does it work vs headless browsers, esp LitePanda & co? Much improvement there?