Fix waves 1 and 2, and a development pipeline 4× faster
Nineteen root causes fixed by two fix teams, restarts cut from 5 min to 1 min, a platform that reproduces a bug on an exact clone of a character, and a disk incident resolved.
- Commit
- 1f53a0b
- Build
- Build success (4 min) · Target worldserver
Wave 1
- Workshop loop: a cooking fire from an inactive event (Harvest Festival) was still selected; now ignored.
- Deaths after resurrection (226 of 421 deaths within 2 min of a resurrection): waiting for resurrection sickness to wear off, graveyard resurrection with broken weapons, hearthstone no longer interrupted.
- Wrongful quest abandons (80%): only time actually spent on the quest counts.
- Distant turn-ins set aside during gryphon flights: fixed.
Wave 2
- Vendor / bank / supplies loop, main cause of an evening XP drop: the delay between visits only yields during the visit itself; a visit with no sale, purchase or repair sets that vendor aside for 10 min.
- Route danger measured better (aggressive monsters only, at their max level): 39% of deaths on the way to a quest are now anticipated, vs 6%.
- Casters without a wand: "Shoot" is no longer attempted without the weapon (238 failures out of 247 for one character).
- Hunters without a pet: travel to another continent for the taming quest.
- Unreachable portal, deliveries dropped en route, water and bread re-sold at auction, unreachable battleground target: fixed.
Development pipeline
- Restart in ≈ 66 s (stop 2 s, world ready in 6 s, 62 characters in game in 31 s), vs ≈ 4 min 40.
- Hot settings: about twenty thresholds tunable without restart.
- Reproduction platform: cloning a production character in its exact state on a separate server, disposable characters on demand, automatic validation of a build before production.
- Continuous deployment, detection every minute, immediate alert on critical bugs, automatic bug dossiers and fixes linked to the registry.
Incident
Between 21:40 and 22:11 UTC no client could enter the game: the (spinning) disks were saturated by test jobs and a database write took 5 s. Database tuned for this hardware, test jobs at lowest disk priority, a connection probe every 5 min with automatic restart.