{"id":1202,"date":"2026-11-06T19:30:00","date_gmt":"2026-11-06T19:30:00","guid":{"rendered":"https:\/\/cotocus.org\/blog\/?p=1202"},"modified":"2026-11-06T19:30:00","modified_gmt":"2026-11-06T19:30:00","slug":"hands-on-mlops-practice-diagnosis","status":"publish","type":"post","link":"https:\/\/cotocus.org\/blog\/hands-on-mlops-practice-diagnosis\/","title":{"rendered":"Why Our MLOps Team Kept Failing, and How Hands-On Work Fixed It"},"content":{"rendered":"<p>Hands-on MLOps practice was the one thing Arjun&#8217;s team had never had, and it took them a painful quarter to realise it. On paper they were well prepared. Every member of the small machine learning group had completed at least one online specialisation. Their shared drive held neatly organised notes on feature stores, model registries and drift detection. Their slides about deployment strategy were genuinely good. And yet, for months, almost nothing they built reached production in a stable state. This is the story of how they diagnosed the problem, and what fixed it.<\/p>\n<p>The team is fictional, assembled from situations that come up again and again in consulting and training work. The pattern is real.<\/p>\n<h2>The symptoms: plenty of knowledge, very little working software<\/h2>\n<p>The trouble showed up in small ways at first. A model that performed well in a notebook produced strange predictions once wrapped in a service. A scheduled retraining job failed silently for a week before anyone noticed. A dependency upgrade broke the inference container on a Friday afternoon, and nobody on the team felt confident enough to roll it back without calling the platform engineers.<\/p>\n<p>Each incident had its own technical cause, and each was fixed. But they kept coming. Arjun, who led the group, began to suspect the issue was not any single bug. Something deeper was missing.<\/p>\n<h2>The first wrong diagnosis<\/h2>\n<p>Arjun&#8217;s initial instinct was to buy more training. Perhaps the team needed an advanced course on deployment, or a deeper module on monitoring. He drafted a request for another round of online learning.<\/p>\n<p>Before sending it, he asked each team member a simple question: &#8220;When did you last deploy a model yourself, from start to finish, without following a tutorial?&#8221; The answers were uncomfortable. Two had never done it. One had done it once, at a previous job, with heavy help. The others had deployed only within guided course environments, where everything was preconfigured and failures were scripted.<\/p>\n<p>Priya, the newest data scientist, put it bluntly in the meeting that followed. &#8220;I can draw the whole architecture on a whiteboard,&#8221; she said. &#8220;I just have no idea what it feels like when it breaks.&#8221; Nobody disagreed. Several people admitted they had been quietly avoiding deployment tickets, passing them to the platform team because they feared making a mess in front of colleagues. The avoidance had become self-reinforcing: the less they deployed, the less confident they felt, and the less confident they felt, the more they avoided it.<\/p>\n<p>The team did not lack information. It lacked contact with the real thing.<\/p>\n<h2>The real diagnosis: recognition without production<\/h2>\n<p>There is a well-known gap between recognising a correct solution and producing one independently. Online courses, even very good ones, mostly build recognition. You watch a pipeline being assembled, you answer questions about it and you complete exercises with guard rails. When you later face a blank terminal and a failing container, that recognition helps you name the problem but not necessarily solve it.<\/p>\n<p>Arjun&#8217;s team had built a large store of recognition and almost no production experience. Their notes were excellent maps of territory they had never walked. Our article on <a href=\"https:\/\/cotocus.org\/blog\/best-practices-for-learning-mlops-from-scratch-a-step-by-step-guide\/\">learning MLOps from the ground up<\/a> makes the point that practical exposure has to begin early; this team had postponed it indefinitely, always intending to practise once they understood enough.<\/p>\n<h2>The solution, part one: a hands-on sprint in one room<\/h2>\n<p>Instead of another course, Arjun booked a meeting room for two full days and invited a platform engineer who had supported several model deployments. The rules were simple. Laptops open, chat closed. One real model, chosen from the team&#8217;s backlog. By the end of the second day, it would be running in a staging environment with monitoring in place, and every team member would have personally performed at least one stage of the process.<\/p>\n<p>The first morning was slow and frustrating. Environment variables were wrong. The container image was far larger than expected. A data path that worked locally did not exist on the staging cluster. Each of these problems would have taken hours to solve alone. In the room, they took minutes, because the platform engineer could look over a shoulder and say, &#8220;Check the mount,&#8221; or because a colleague had hit the same snag ten minutes earlier.<\/p>\n<p>By the afternoon, something shifted. People started anticipating problems rather than reacting to them. They asked better questions. Someone suggested adding a health check before anyone prompted them to.<\/p>\n<h2>The solution, part two: monitoring they had built themselves<\/h2>\n<p>On the second day, the group set up monitoring for the deployed model. They had studied drift detection extensively, but implementing it was different. Choosing which features to watch, deciding what counted as a meaningful change and wiring alerts to the right people forced concrete decisions that the course material had left abstract. The team used our guide on <a href=\"https:\/\/cotocus.org\/blog\/model-monitoring-basics-every-mlops-learner-should-know\/\">the essentials of watching models in production<\/a> as a reference, but the understanding came from doing it with their own data.<\/p>\n<p>When they deliberately fed the model shifted inputs to test the alerts, the dashboards lit up exactly as planned. The small cheer that went round the room was not about the technology. It was relief at finally having done the thing they had been reading about for months.<\/p>\n<h2>The solution, part three: keeping the contact going<\/h2>\n<p>Arjun knew a single sprint would fade without follow-up. The team adopted three habits.<\/p>\n<ol>\n<li><strong>Rotate deployment duty.<\/strong> Each sprint, a different person owns the full deployment of at least one change, with a colleague available but not driving.<\/li>\n<li><strong>Pair with platform engineers monthly.<\/strong> One afternoon a month, a data scientist sits with the platform team while they handle real operational work.<\/li>\n<li><strong>Attend external hands-on events.<\/strong> Team members are encouraged to join community workshops and developer meetups, where they see how other organisations handle similar problems in practice.<\/li>\n<\/ol>\n<p>For the third habit, discovery turned out to be easier than expected. The team was based in Bengaluru, where <a href=\"https:\/\/bangaloreorbit.com\/\">Bangalore Orbit lists the city&#8217;s tech and developer events, workshops and meetups<\/a> alongside its other live listings, so finding a relevant session most months took little effort.<\/p>\n<h2>What changed after a quarter<\/h2>\n<p>Three months on, the difference was clear without any formal measurement. Incidents still happened, but the team resolved most of them without escalating. Deployments that once stretched across weeks now took days. Newer members, onboarded straight into the rotation, became productive faster than their predecessors had. Perhaps most tellingly, the team&#8217;s notes changed character. They became shorter, more specific and full of hard-won warnings, the kind of documentation written by people who have actually run the system.<\/p>\n<p>Several members also found the experience good for their careers. Our piece on <a href=\"https:\/\/cotocus.org\/blog\/mastering-mlops-your-key-to-accelerating-ai-career-growth\/\">how MLOps skills accelerate a career in AI<\/a> argues that practitioners who can take a model all the way to production are in strong demand. Arjun&#8217;s team had moved from describing that journey to completing it.<\/p>\n<h2>Lessons for other teams<\/h2>\n<ul>\n<li><strong>Ask the direct question.<\/strong> When did each person last do the core task alone? The answer often reveals the real gap.<\/li>\n<li><strong>Do not default to more courses.<\/strong> If the team already knows the concepts, additional information rarely helps.<\/li>\n<li><strong>Put people in the same room with real work.<\/strong> Shared, hands-on sessions with an experienced practitioner compress weeks of solo struggle into hours.<\/li>\n<li><strong>Make contact routine.<\/strong> Rotations, pairing and external workshops keep skills from decaying.<\/li>\n<\/ul>\n<p>The fix for Arjun&#8217;s team was not glamorous. It was two days in a room, a real model and someone willing to look over their shoulders. That simple dose of first-hand experience did what months of study could not.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A well-trained machine learning team kept stumbling at deployment. The diagnosis was not missing knowledge but missing practice, and a room fixed it.<\/p>\n","protected":false},"author":4,"featured_media":1203,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[129],"tags":[],"class_list":["post-1202","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops"],"_links":{"self":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1202","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/comments?post=1202"}],"version-history":[{"count":1,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1202\/revisions"}],"predecessor-version":[{"id":1209,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/posts\/1202\/revisions\/1209"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media\/1203"}],"wp:attachment":[{"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/media?parent=1202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/categories?post=1202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cotocus.org\/blog\/wp-json\/wp\/v2\/tags?post=1202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}