Murloc Cult

Back to the log

Fix waves 1 and 2, and a development pipeline 4× faster

Nineteen root causes fixed by two fix teams, restarts cut from 5 min to 1 min, a platform that reproduces a bug on an exact clone of a character, and a disk incident resolved.

Commit
1f53a0b
Build
Build success (4 min) · Target worldserver
Wave 1
  • Workshop loop: a cooking fire from an inactive event (Harvest Festival) was still selected; now ignored.
  • Deaths after resurrection (226 of 421 deaths within 2 min of a resurrection): waiting for resurrection sickness to wear off, graveyard resurrection with broken weapons, hearthstone no longer interrupted.
  • Wrongful quest abandons (80%): only time actually spent on the quest counts.
  • Distant turn-ins set aside during gryphon flights: fixed.
Wave 2
  • Vendor / bank / supplies loop, main cause of an evening XP drop: the delay between visits only yields during the visit itself; a visit with no sale, purchase or repair sets that vendor aside for 10 min.
  • Route danger measured better (aggressive monsters only, at their max level): 39% of deaths on the way to a quest are now anticipated, vs 6%.
  • Casters without a wand: "Shoot" is no longer attempted without the weapon (238 failures out of 247 for one character).
  • Hunters without a pet: travel to another continent for the taming quest.
  • Unreachable portal, deliveries dropped en route, water and bread re-sold at auction, unreachable battleground target: fixed.
Development pipeline
  • Restart in ≈ 66 s (stop 2 s, world ready in 6 s, 62 characters in game in 31 s), vs ≈ 4 min 40.
  • Hot settings: about twenty thresholds tunable without restart.
  • Reproduction platform: cloning a production character in its exact state on a separate server, disposable characters on demand, automatic validation of a build before production.
  • Continuous deployment, detection every minute, immediate alert on critical bugs, automatic bug dossiers and fixes linked to the registry.
Incident

Between 21:40 and 22:11 UTC no client could enter the game: the (spinning) disks were saturated by test jobs and a database write took 5 s. Database tuned for this hardware, test jobs at lowest disk priority, a connection probe every 5 min with automatic restart.