What the logs already knew

A tour of sotalogparser, and of the wrong answers that shaped it.

Shroud of the Avatar writes everything down. Every swing, every catch, every failed masterwork, every word anybody said near you, timestamped, in plain text, in a folder you already own. A player of two years is sitting on a few million lines of it.

Almost nobody reads them, because a chat log is not a record — it is the raw material for one. This is an app that does the reading: a cross-platform desktop application that parses that folder into a queryable index and reports on it thirteen ways.

The rules were derived against a real corpus of 3.84 million lines and recognise 99.91% of them. That corpus is the reason the interesting parts of this post exist. Every non-obvious decision below was a plausible answer first, and the corpus is what proved it wrong.

The shape of the thing

Three layers, strictly one-directional.

.NET 10, Avalonia 12, MVVM. One codebase produces self-contained single-file builds for six runtimes — Windows, macOS and Linux, x64 and arm64 apiece. Everything worth a test lives in Core, which is why there are 522 of them and none require a window.

The pipeline

LogLineReader splits [timestamp] message and folds continuation lines. ChatLogParser runs an ordered list of rules, first match wins. Whatever nothing claims becomes an UnrecognizedEvent.

That last part is a rule, not a fallback. Never drop a line. The unrecognised bucket is the only mechanism by which a gap in the rule set is visible at all — silently discarding what doesn't match would hide exactly the information needed to fix it. The app surfaces the coverage percentage and lets you filter a log down to just the unclaimed lines, which is how 99.91% got to be 99.91%.

The index

One log file gets a LogSummary. Everything across the whole corpus goes through a SQLite index: parse, attach scene context, write fact rows, one transaction per file, re-reading only files whose size or modification time changed. 3.84M lines index in about twenty seconds into roughly 19 MB.

The index is a disposable cache and never a source of truth. Schema or parser changes bump a version, and the file is deleted and rebuilt rather than migrated. Nothing in it can't be regenerated from the logs, so nothing in it is worth the risk of a migration bug.

Two things about it that took a corpus to learn.

The parser is stateless per line, so only a zone-change line knows where anything happened. A tracker latches the last scene entered and resets on a session start. Roughly a quarter of real events occur before any Entering line — they land in an (unknown) scene, which is a real bucket the reports must show, not a gap to be filtered away.

Combat has a direction. A quarter of attack lines in the corpus are mobs hitting the player, and their abilities look exactly like the player's. Without splitting on direction, "your skills" lists skills the player has never used.

Thirteen reports

The rail groups them. Each loads lazily on first open and reloads when the date, scene or character filter changes; everything exports to CSV.

Overview and Log browser

What the index covers, and every line of a single log as the parser saw it. The browser exists for the coverage rule above — filter to Unrecognized and you are looking at the parser's remaining blind spots, in order.

Combat — Skills vs mobs

How each skill performs against each target, and where a pairing beats what the rest of the data predicts. 65 skills crossing 282 targets is a two-way table, and the interesting cell is the one that beats both its row and its column. Three choices, each of which was wrong first:

• Median polish, not means. With means, one exceptional cell drags up its own skill row and its target column, so every neighbour reads as resistant. One real finding, six invented ones.

• Unweighted. Weighting the median lets any cell holding over half its row's observations define its own baseline — which is precisely the heavily-farmed pairing you most want to measure.

• A spread gate as well as a sample gate. A skill used against only two targets has no independent baseline, so the gap between its two cells reads as an enormous interaction. Against the real corpus this single gate is the difference between one defensible finding and eight artifacts.

Combat — What hits you

The mirror question, and the newest report: which creatures attack you, with what, for how much, and how often your defences stop it. It is emphatically not the previous report with the arrow reversed — reusing the outgoing machinery would have been wrong in four ways, every one of which produces a plausible number rather than a visible failure.

More than half the incoming fire lands on the pet.

And they behave nothing alike: 52% of blows aimed at the player connect for nothing, against 0.5% of blows aimed at a pet. Pooled, the defensive profile belongs to neither. Pets are excluded — and their share reported, because "this covers 44% of what was thrown at you" is something a reader is owed.

The obvious mitigation figure omits the majority of mitigation. The outgoing report's formula counts dodges, blocks, parries and glancing blows, and comes to 18% incoming. But 52% of incoming attacks are connected, dealt nothing — excluded by that formula. The honest figure is 66.2%. Meanwhile blocks and resists are absent from incoming lines entirely — zero of each in 3.84M lines — so those columns are omitted rather than shown as structural zeroes, because a column that is always zero reads as "you have never blocked anything".

A dodge never names what was dodged. All 7,120 incoming dodges and all 17,474 outgoing ones sit in the unnamed bucket whatever provoked them. So every per-ability rate divides by the attacks that did name an ability, and the dodge rate lives on the per-creature roll-up where it is true.

And ranking is by damage per swing, because the three plausible orderings barely agree:

Rank by frequency and the report opens on a crocodile doing two damage. Rank by damage per hit and it opens on an event that happens one time in twenty. Only per-swing folds in the connect rate and answers what is this going to cost me.

Combat — Records

Personal bests: biggest critical per skill, hardest blow taken, each traceable to the line it came from. This is the one report that must undo a roll-up the rest of the app depends on. Pet damage is credited to the owner, which is right for totals and wrong for a record:

More blows land on the pet than the player — 64,245 against 44,329 — so "the hardest hit you took" is somebody else's bruise unless you check. Bests are per character and never pooled, ties go to the earliest, and damage-over-time is excluded: the biggest of those in the corpus is 13,555, which beats every attack, and folding it in silently answers a different question.

World — Scene activity

Where the time went, and what each place was good for. Three things it has to get right:

• A rate needs a denominator you can trust. Three minutes in Arlia collected one 10,000 XP award — 206,897 an hour, second place ahead of a spot farmed for eighty hours. The per-scene figure is still shown, beside the minutes it came from, but the rankings apply a 30-minute floor and say how many scenes that excluded.

• Kills and deaths are the same lines read from opposite ends. 281 death lines are 246 kills, 22 deaths, 15 bystanders and 2 self-kills. Counting rows reports a fifth more kills than were made and hides every death.

• Corpse loot is combat; harvesting is gathering. Tanglemire logs 743 attacks, 1,305 harvests and no corpse loot at all. One combined loot bucket labels it by whichever total wins, so a fishing spot with wolves in it comes out as a combat zone.

World — Loot and Fishing

Where drops came from and what each place actually yields; every catch pivoted by where you fished, by the water it came from, and by species.

Craft — Recipes

Exceptional rates per recipe, how far a masterwork can be pushed, and what it cost. Intervals are Wilson's score interval, not the normal approximation — which puts the upper bound above 1 at seven successes in eight and collapses to a point at nought in twelve. Three more:

• An attempt is one craft. The two completion lines are alternatives, never both. The corpus gives the mistake away immediately: Dark Red Dye logs 73,231 ordinary and 73,433 exceptional, so treating exceptional as a subset of attempts yields a success rate above 100%.

• A masterwork failure names the item one step below where the attempt was made. Tallying by the printed number invents a smooth curve out of nothing.

• Significance and effect size are separate quantities. The interval decides whether a recipe really differs from the overall rate; the observed rate decides whether it differs enough to mention. Conflating them flagged one of three food recipes clustered near 45% and left its two neighbours unmarked.

More — Progression

Experience earned and skills trained. Quiet slices are kept — a series built only from days something happened draws a month off as an unbroken run, the exact opposite of what the reader opened it to find out.

And steps are not levels. The game does not announce every level: Tailoring runs 44, 48, 51, 54, so 85 announcements cover 117 levels. The report shows steps and the observed range, and marks rows where more levels went by than were announced. "Levels gained" is simply not in the log, and the report says so rather than guessing.

The related trap, corpus-wide: experience readouts are running totals, and there are four different ones per pool — lifetime earned, the character pool, the shared pool, and their sum, written together as a block. The readouts sum to 1.57 trillion against 187 million actually awarded. Awards are selected positively by kind, never as "anything that isn't a readout", because that phrasing silently admits every readout kind added later.

More — Conversations

The flat chat stream threaded back into exchanges. There is no conversation marker in the log at all, so a run is split wherever the talking stopped for ten minutes. Two rules earn their place: a run with nobody in it but the player is not a conversation (otherwise a session spent advertising on Traders fills the report with one-sided "exchanges"), and a message with no timestamp keeps its file position (sorting nulls to the front opens a gap the width of the whole conversation).

"Me" is resolved per row against the log's own character — a corpus with six characters has six answers to who am I.

More — Economy

Mostly notable for what it refuses to compute. The log has no prices and no purchase lines, so units sold are knowable and revenue is not. The only gold figure anywhere is You picked up N gold., which has nothing to do with your vendor. The report says so, because a gold number printed beside a sales table will be read as takings.

What it does surface is the standing-order split — order-fulfilled against browsed — which is the one thing in the data a seller can act on, and which came out cleanly divided in the corpus: every raw material 100% orders, every refined good 100% browsed.

More — Trades

Player-to-player trades folded back together. A trade is a run of lines — opened, things put up and taken back, closed — and no single line says what changed hands. Four things:

• Gold is a running total, not a change. The line states the new figure. A real trade in the corpus runs gold to 99,000 and back to 0 before completing; summing the lines reports ninety-nine thousand gold that never moved.

• The two sides are logged asymmetrically. The game records the partner adding and removing items, but only the player adding — never withdrawing. So the player's side is "what was offered" and can overstate; the partner's side is exact. The report says which is which.

• Most "trades" are with yourself. 64 of 106 completed trades are between the player's own alts, and where both alts' logs are present the same trade is stored twice, once from each side. Flagged and filterable.

• The partner name is sometimes mangled by the game — Zealoty,,1]Zealoty. That is in the log, not something the parser does. The repair matches only that exact doubled shape, because real names contain commas.

The companion site

The app can optionally sign in to a companion website and publish four datasets: personal bests, fishing catches, skill-vs-target cells, and monster abilities.

The last two are unlike the first two in a way that has to be respected. They are cumulative totals, so a re-publish replaces rather than accumulates — which means publishing must always be unfiltered by date and scene, because under replace semantics a filtered slice does not send weaker figures, it deletes the fuller ones. And a character over the server's cell cap is skipped whole rather than truncated, for the same reason.

What goes up is the sufficient statistics — count, Σln(1+damage), Σln(1+damage)² — rather than the fitted result, because median polish does not pool across players and sums do. They are additive, which is what makes exact community pooling and constant-time leave-one-out possible at all. (The 1 + is load-bearing: the inverse is exp(x) − 1, and reading them back with plain exp is under a percent off at a few hundred damage and a third off at three. Plausible everywhere and wrong everywhere. It is how the site's first implementation shipped.)

One thing the corpus taught here too. An ability is largely a property of its creature — 22 of 74 abilities belong to exactly one, and 169 of 272 creatures have two or fewer. So there is no two-way model to fit; what replaces it compares an ability against itself only where three or more creatures use it, leave-one-out, so the creature holding most of an ability's observations cannot define its own baseline and report itself as ordinary. Earth Elemental holds 2,955 of Earthquake's 3,637 swings and comes out at 0.08×; a self-inclusive pool would have read about 1.0 and said nothing.

The account is optional and stays optional. Restoring a session reads a file and makes no request. The first packet leaves the machine when you click Sign in or Publish, never before. No report takes a dependency on being signed in, and the publish controls are hidden when signed out rather than shown disabled.

Privacy

Chat logs contain other players' messages, and that fact constrains the code rather than being mentioned in a policy.

The index stores raw message text — that is the conversations report. It lives in your app data directory, outside the repository, and nothing from it is ever uploaded. No future payload should change that without asking the user in as many words.

What publishing does send is character names, skill and creature names, figures, scenes and times. One of those deserves care: the log never says whether an attacker was a creature or another player, so an attacker name can be a real person's. That is why publishing shows the actual rows once per dataset before the first upload rather than only counting them. Uploading is outward-facing and cannot be undone from here.

Real logs are git-ignored; test fixtures are hand-redacted.

Where it is

Working end to end. Thirteen reports, no placeholders left in the rail, 522 tests, six release targets. The parser has 99.91% of the corpus and the remaining 0.09% is visible in the app rather than swept up.

If there is a thread running through all of it, it is that the corpus was right and the reasoning was wrong, over and over. Every heading above that begins "three things" is a list of numbers that looked correct until somebody checked them against three million lines of what actually happened. That is the whole method: check the pattern against the corpus rather than reasoning about what the game probably writes.