Reading a community when nobody can read all of it
A large, active community produces more feedback in a day than any team can read by hand. Here is how I built a pipeline that turned that volume into something the dev team actually used.
The first real problem with a large community is not moderation. It is that the community starts saying useful things faster than anyone can listen.
At Amazing Seasun Games the Mecha BREAK community ran into the tens of thousands of members, and on a busy day it produced tens of thousands of messages. Buried in that volume was the exact thing the development team wanted: honest, specific reactions to the build they had just shipped. The problem was that the signal arrived mixed into everything else, in real time, with no structure. A bug report sat next to a meme sat next to an argument about balance. Nobody had the hours to read all of it, so most of it was never read at all.
The instinct in that situation is to ask people to file feedback in a form. That helps, but it only captures the members who are willing to stop and fill out a form, which is a small and unrepresentative slice. The most useful reactions are the ones people type into chat the moment they feel them, and then move on. If you only collect the structured submissions, you miss the part that matters most.
So I built the pipeline to pull from both sides. It took the structured bot submissions and it also watched the passive chat, and it ran everything through an LLM step whose only job was to separate constructive signal from noise. Not to judge whether feedback was right, just whether it was the kind of thing a developer could act on. A message that says “the dash feels bad” gets kept. A message that is two players insulting each other does not.
The hard part was never the model. It was deciding what to keep. Set the filter too loose and you have rebuilt the original problem, an inbox nobody opens. Set it too tight and you quietly drop the one comment that would have mattered. I tuned it against real days of real chat until the output was something I would have been comfortable handing to an engineer with no edits.
On top of the filtering I built reporting on a fixed cadence, daily, weekly, and monthly, each one structured the same way so the development team could skim it in the same place every time. During the closed and open beta tests, the busiest stretches, a single report would cover hundreds of distinct data points. The whole thing processed thousands of submissions a month, and the team got a steady read on what the community was actually saying instead of a feeling that there was feedback out there somewhere.
One detail I would point to for anyone building something similar. The translation step mattered more than I expected. Accepted feedback was auto-translated into Simplified Chinese for the development team, which meant the distance between a player typing a reaction and a developer reading it was a few minutes and one pipeline, not a meeting and a backlog. Shortening that distance is most of the value. The model is a commodity. The judgment about what to forward, and the discipline to deliver it the same way every day, is the part that made people trust it.
If there is a general lesson, it is that “we have too much feedback to read” is not a content problem. It is a routing problem. You do not need to read everything. You need a reliable way to decide what reaches the people who can do something about it, and then you need to run that decision every single day without a person in the loop. Build that, and a community that was too loud to hear becomes the best research instrument you have.