r/backgammon • u/CalebWetherell • 15h ago
I built a new backgammon engine, Vulcan. I believe it's the best engine in the world. Details inside, and you can play and analyze with it for free.
Hi all,
I've built a backgammon engine from scratch, programmed in Rust, Vulcan, with its own neural networks. I believe it's now the strongest backgammon engine in the world, and I'd like to make that case as simply as I can: two results, then the site where you can use it for free, then all the detail for anyone who wants to check the work.
1. The strongest network we could find to test against
Take away all search and let each engine play with its neural network alone. In cubeful money games, Vulcan's network beats every other engine's network we could get our hands on, by a wide margin:
- vs XG 2.10's network: +0.058 points per game
- vs GNU Backgammon 0-ply: +0.084
- vs HedgeHog FOX: +0.058
- vs HedgeHog Aureus: +0.037
(20,000 game pairs each; every 95% interval is about ±0.003, so none of these is close to level.)
And Colossus 2.0? It isn't publicly playable, so nobody outside HedgeHog can play it. But HedgeHog published how Colossus 2.0's network scores against these same four opponents, with the same settings, and Vulcan's margin is bigger against every one of them. Against Aureus, HedgeHog's own second-best network, Vulcan wins by +0.037 and Colossus 2.0 by +0.021: Vulcan beats HedgeHog's No. 2 by almost twice as much as HedgeHog's No. 1 does.

HedgeHog also published 25,000 games of Colossus 2.0 against XG. On the 74,639 decisions in those games where Vulcan, looking as far ahead as Colossus did, would have played differently, rollouts favour Vulcan's play (+0.0004 per disagreement; Vulcan's play was clearly better in 17,160 decisions, Colossus's in 15,323). Those rollouts were played by Vulcan's own networks, so treat this as supporting evidence rather than proof.
2. Vulcan's 2-ply beats XG's 4-ply, even though XG looks one ply further ahead
Plies count differently between programs: Vulcan's 2-ply looks as far ahead as XG's 3-ply. XG's 4-ply, the setting strong players use to check their games, searches one ply deeper than Vulcan's 2-ply. I played them against each other anyway, with full cube decisions on both sides and XG using its opening book:
- Money games: Vulcan +0.0049 points per game over 200,000 game pairs (luck-adjusted, 95% interval +0.0038 to +0.0059; the raw result is level).
- 7-point matches: Vulcan wins 50.27% of 80,000 matches (interval 50.22-50.32%).
- Vulcan thinks a quarter to a third as long per move as XG 4-ply.
At equal lookahead (Vulcan 2-ply vs XG 3-ply) the gap is much wider: +0.0125 points per game.

That's the case: a better network than any engine I could test, more than Colossus 2.0's published margins against the same opponents, and a 2-ply that holds its own against XG one ply deeper. Every run was planned and written down before it started, and raw results are reported beside the adjusted ones (details below).
Use it free on Gammon Forge
Vulcan runs everything on gammonforge.com, and all of it is free: no ads, no subscription, one Google sign-in (you can play one game without an account).
- Play Vulcan at five levels, from Beginner (about 1100) to Full strength (about 1950; the ratings are rough estimates), in single games, matches or money sessions with the cube. Climb a ladder of 5-point matches, or play rated games against people, live or by correspondence.

- Analyse any position. Paste an XGID or GNU BG ID and Vulcan ranks every play, going deeper while you watch. Close calls are settled with free rollouts. Here's a 1-3 backgame where the top four plays are within 0.01 of each other at 3-ply; a quick rollout keeps 8/4 on top and drops the hit 13/12*/9 to an inaccuracy:



- Every game you finish is reviewed: PR for both players, every mistake with the better play drawn on the board, a win-chance graph and the luck of each roll. You can also upload .mat files from XG or GNU BG.


- 84,233 puzzles from Vulcan's games against itself, with a puzzle rating, a daily puzzle, streaks and sprints, plus lessons on the rules, the opening moves and the cube.

Verification
I've done my best to validate and second-guess myself before making this post. The project has been a year in the making. There's detailed results showing engine-vs-engine analysis that I conducted personally, so there's an element of trust here. I would be happy to work with someone who wants to set up a third party engine evaluation system, as I think that would be the best way to prove the claims I am making. Also, I encourage strong players (not me!) to test it out, share feedback, critique flaws you find. I'd love positions where you think Vulcan plays badly, and questions about the testing.
More detail, for those interested
Every headline run
| Vulcan 2-ply vs | Format | Games | Luck-adjusted [95% CI] | Raw [95% CI] |
|---|---|---|---|---|
| XG 2.10, 4-ply checker + cube, XG's opening book | Money | 200,000 pairs | +0.0049 [+0.0038, +0.0059] ppg | -0.0039 [-0.0112, +0.0033] |
| XG 2.10, 4-ply checker + cube | 7-point matches | 80,000 matches | 50.27% [50.22, 50.32] | 50.21% [49.90, 50.52] |
| GNU Backgammon 1.08.003, 2-ply checker + cube | Money | 100,000 pairs | +0.0214 [+0.0197, +0.0230] ppg | +0.0150 [+0.0040, +0.0261] |
| GNU Backgammon 1.08.003, 2-ply | Cubeless | 200,000 pairs | +0.0119 [+0.0114, +0.0124] | +0.0099 [+0.0065, +0.0133] |
Every equal-lookahead run, with Colossus 2.0's published cells
Cubeful money, HedgeHog's benchmark settings (Jacoby and beavers off, no opening books).
| Vulcan depth | Opponent (same lookahead) | Pairs | Vulcan, ppg [95% CI] | Colossus 2.0, same opponent (published) |
|---|---|---|---|---|
| 0-ply | GNU BG 0-ply | 20,000 | +0.0837 [+0.0811, +0.0864] | +0.0708 [+0.0689, +0.0728] |
| 0-ply | HedgeHog FOX | 20,000 | +0.0580 [+0.0554, +0.0606] | +0.0388 [+0.0368, +0.0407] |
| 0-ply | XG 2.10 1-ply | 20,000 | +0.0577 [+0.0550, +0.0603] | +0.0499 [+0.0479, +0.0518] |
| 0-ply | HedgeHog Aureus | 20,000 | +0.0371 [+0.0346, +0.0396] | +0.0209 [+0.0190, +0.0229] |
| 1-ply | XG 2.10 2-ply | 20,000 | +0.0360 [+0.0336, +0.0385] | +0.0327 [+0.0232, +0.0422] (level with Vulcan's) |
| 1-ply | GNU BG 1-ply | 20,000 | +0.0506 [+0.0482, +0.0529] | not published |
| 2-ply | XG 2.10 3-ply | 4,000 | +0.0125 [+0.0082, +0.0169] | not published |
| 2-ply | GNU BG 2-ply | 3,000 | +0.0292 [+0.0131, +0.0452] | not published |
Depth names: Vulcan's 0-ply = GNU BG 0-ply = XG / HedgeHog "1-ply" (the network alone); Vulcan's 1-ply = XG 2-ply; Vulcan's 2-ply = XG 3-ply. Colossus's cells are HedgeHog's raw results over 10M games per pair (its XG 2-ply cell is against real XG 2.10 on HedgeHog's own harness). Vulcan's are luck-adjusted with Vulcan's networks as the reference; on the GNU BG row, GNU BG's networks give the same answer (+0.0841 vs +0.0837).
5-point matches with public dice. One run took its dice from a public randomness beacon (drand) whose round I named in a public gist before that round existed. Vulcan 1-ply vs XG 2.10 2-ply, 10,000 5-point matches: 51.29% match wins [51.15, 51.43] (raw 50.81%). HedgeHog's published Colossus 2.0 vs XG 2.10 in the same format: 50.68% [50.44, 50.93]. Every match file replays and its dice can be checked.
Colossus 2.0's own decisions. Rollouts of the decisions in Colossus's 25,000 published games vs XG where Vulcan would play differently: at equal lookahead (Vulcan 1-ply, an earlier network from Nov 1), +0.0004 [+0.0002, +0.0006] per disagreement over 74,639 decisions; with Vulcan's 2-ply (one ply deeper than Colossus played), +0.0051 per disagreement over 70,091. Rollouts: cubeful, 1-ply policy, played by Vulcan's own networks.
Speed
Vulcan's ply costs more than XG's or GNU BG's: in cube-aware play it runs its full network on every reply it considers, where they screen replies with a small network first. Its 2-ply still thinks a quarter to a third as long as XG's 4-ply. The neural nets behind Vulcan are larger than the networks in XG, with numerical optimizations applied that compress the network for speed, while retaining as much intelligence as possible.

How it was measured
- Paired games. Each pair of games uses one dice sequence twice with the seats swapped.
- Luck adjustment. Before each roll a fixed reference network values all 21 rolls; the roll's luck is its value minus the average. Subtracting it can't shift the expected result (the average luck is zero by construction); it removes noise, about 7x less spread per pair here, so 200,000 pairs measure what would otherwise take millions. In the headline runs and the 5-point run the reference is GNU Backgammon's networks, not Vulcan's. Raw results are shown beside every adjusted one.
- Pre-registered. Sample size, settings and the decision rule were written down before each run; runs weren't stopped early, and the one extension (a second 100,000-pair money run against XG 4-ply on fresh dice) was registered as its own run and both are reported.
- The XG opponent. Engine-vs-engine play needs an automated XG, so these runs use a native port of XG 2.10's engine: its networks, move filters, cube logic and opening book. To check it, real XG 2.10 analysed the port's games: at 2-ply it flagged 0 of 14,233 decision records in 100 matches, and at 4-ply 0 of 7,384 in 40 matches from the 7-point run above. XG agrees with every move and cube decision the port made.
- GNU BG is the real program (1.08.003), with Jacoby off and cubeless checker play for the cubeless run (its defaults undervalue gammons there).
Looking forward
A bigger research version of Vulcan's network (too slow to serve today) beats the current one by a further +0.012 points per game head to head. So, I'm confident that there is more room for Vulcan (or others) to improve further.
If you've made it this far, thank you! Please give it a spin and let me know what you think.
-Caleb Wetherell