Case Study: A Four-Person Team Ships Version One in Nine Days

Case Study: A Four-Person Team Ships Version One in Nine Days

The team in this story is invented, but anyone who has worked in a delivery group will recognise the people. Four engineers at a mid-sized company were asked to build a tool that scores incoming support requests by urgency, so the human agents could handle the pressing ones first. Their plan, drawn up carefully, ran for six months.

The six-month plan

The plan had phases. First, collect and label two years of historical tickets. Second, train and tune a classification model. Third, integrate with the ticketing system. Fourth, build an agent-facing dashboard. Fifth, monitoring and retraining. A neat chart showed the stages ending in a launch party.

Four weeks in, they were still on phase one. Labelling was slow, the labels disagreed with each other, and every week someone found a new category worth adding. The team lead, a practical person we will call Dana, noticed the mood in standups. People were tired, defensive and vague about progress. Nobody had spoken to an agent in a month.

The conversation that changed things

Dana asked one of the agents a simple question over coffee: “If you could have one thing right now to help you pick the next ticket, what would it be?” The answer was almost embarrassing in its modesty. “Tell me which ones mention a payment problem. Those are always urgent.”

That was it. Not a trained model, not a dashboard. A flag for tickets that mention payment trouble. Dana wrote the sentence on the whiteboard and drew a line through the plan’s phases. The goal for the next nine days: flag payment-related tickets, show the flag in the agent’s queue, and ask agents what they thought.

Days one to three: building the thinnest path

The team chose a deliberately plain approach: a set of keyword rules reviewed by two agents, wired into the queue as a small tag. No machine learning at all. A teammate named Ravi objected that this felt like cheating, a step backwards from the real project. Dana’s answer was practical: “If the rules do not help, a model will not either. If they do, we have found out what matters.”

They kept the usual engineering standards. The tag service had tests, ran behind a switch that could turn it off in seconds, and wrote a log line for every flag it raised. Small scope, ordinary care.

Days four to six: a quiet launch

They enabled the tag for a single shift of agents, about a dozen people, and posted a short note in the team channel: what the flag did, what it did not do, and how to report mistakes. They did not call it a launch. They called it a trial of a first version. The wording mattered; agents felt invited, not tested.

On the first afternoon, agents reported a false alarm: a customer had written “no payment problem, thanks.” The rules treated it as urgent. By evening the team had added a small exception. Over the following days the reports kept arriving, and each fix took under an hour. For readers weighing how to keep such a tool healthy once it is running, an article on monitoring basics for newcomers to machine learning operations gives helpful grounding on what to watch.

Days seven to nine: reading the signals

The team reviewed three things: the log of flags, the agents’ comments, and a quick sample of tickets the tag had missed. A pattern emerged that nobody had predicted. Agents rarely cared about the payment flag in isolation. What they really wanted was to know when the same customer had written twice. Repeat contact, not keywords, was their true signal of urgency.

That discovery rewrote the roadmap. Half of the original six-month plan, including a large labelling effort around dozens of categories, turned out to be unnecessary. The next version added a “repeat contact” indicator. It took three days.

What the team wrote down

Dana insisted that the team record decisions in plain language: the attempt, the outcome and the intended follow-up. Those notes were useful internally and, with a few details removed, became a public write-up that other teams asked about. Teams who want an easy home for notes like these can use a free blogging platform. BlogAion positions itself as free to join and issues each writer a personal subdomain, so a first version of an engineering note can go live promptly and be edited when a better understanding arrives.

Eventually, the model

The story does not end with rules forever. Several weeks later, with real tickets flowing and agents’ judgments recorded, the team had something they never had in the original plan: a trustworthy, small set of labels, produced by real use rather than a labelling marathon. A simple model built on that data had a clear baseline to beat, the keyword and repeat-contact rules, and had to earn its place against them.

For the architecture of that stage, they leaned on general material such as a piece on building dependable and scalable AI products, which fitted well once there was something real to scale.

What the team would tell others

When Dana presented the story to colleagues, she kept it to a few points:

  • Talk to a user in the first week. One conversation rewrote the project.
  • Prefer a rule you can explain to a model you cannot. At least at first.
  • Keep standards, shrink scope. Tests, logs and an off switch took hours, and saved days.
  • Label the release honestly. “First version” invited help instead of judgment.
  • Plan the review before launch. The review is where the roadmap changed.

The numbers that mattered, and the ones that did not

The team did track how many tickets were flagged and how many flags agents rejected. But their most useful measure was qualitative: whether agents kept the tag switched on after the first week. They did, and several asked for more. That quiet continuation told the team more than a chart.

Afterword

Ravi, the teammate who had called the rules a step backwards, ended up presenting the repeat-contact discovery at a company meeting. He described it as the most useful thing the team shipped that year, and then added the observation that stuck with everyone: “We would never have found it if we had waited until the model was ready.”

That sentence sums up the case. Waiting for the perfect version would have delivered a polished answer to the wrong question. A nine-day first version delivered a small answer to the right one, and then time to improve it.