A head of ML at a 120-person company told us recently that his team had spent nine months trying to stand up a “proper MLOps platform.” They had evaluated three orchestration tools, designed a feature store architecture, and written a design doc for a custom model registry. In that same nine months, they had shipped exactly zero models to production. The data scientists had quietly given up and started deploying their models by hand through a shared Jupyter notebook on a GPU VM.

This is the MLOps problem most mid-market teams actually face. The dominant discourse around MLOps is written by platform engineers at companies large enough to have a platform engineering org. A company with 5-20 engineers building ML does not have that team, and pretending otherwise is how you end up with a nine-month platform project and nothing in production.

We see this pattern repeatedly. A 60-person fintech hired two talented data scientists, gave them a GPU instance, and expected models to flow. The data scientists spent four months evaluating orchestration frameworks before writing a single line of training code. By the time they had a working pipeline, the business question they were hired to answer had changed twice. The platform they built was technically sound, but it was a platform for a team of twenty serving a company of five hundred — not for two people serving a company of sixty. The gap between the MLOps literature and the mid-market reality is not a knowledge gap. It is a scale mismatch, and bridging it requires a different set of defaults.

This guide is for the people caught in that gap — the ML lead, the senior data scientist who has accidentally become responsible for infrastructure, the engineering manager trying to get models shipped without burning the team out. It covers the four areas that actually move the needle at this scale: model versioning, CI/CD for ML, experiment tracking, and deployment patterns. The goal is not to replicate the enterprise platform. It is to ship reliable models with the team you have.

Why Mid-Market MLOps Is a Different Problem

Enterprise MLOps is built on a premise of leverage. You invest in a platform team because the platform will be reused across dozens of model teams, and the math works at scale. Feature stores, custom orchestrators, in-house model registries — these pay off when twenty squads are going to use them.

At 5-20 engineers, that leverage does not exist. You probably have one or two model teams, a handful of production models, and a backlog of experiments. The platform investment amortizes across too few users to justify itself, and it competes directly with model work for the same scarce engineers. Every week spent building internal tooling is a week not spent shipping the models that generate revenue.

The MLOps systems that work at this scale share three traits. They use managed services or mature open-source tools instead of custom infrastructure. They impose a small number of non-negotiable disciplines instead of comprehensive pipelines. And they are boring on purpose, because boring is what survives staff turnover and a busy quarter. The four pillars below follow that logic.

Pillar 1: Model Versioning — Treat Models as Code, Not as Files

The first failure mode we see in mid-market ML teams is a shared drive full of model files with names like model_final_v3_really_final.pt. No one knows which model is in production. When a model starts degrading, the first hour of incident response is spent figuring out what is actually running.

Model versioning is the discipline that prevents this, and it does not require a custom registry. It requires treating every model artifact the way you already treat code: versioned, traceable, and tied to the inputs that produced it.

What to Version, and Where

A model artifact on its own is nearly useless for debugging. The useful unit is the bundle: the model weights, the code that trained them, the dataset version (or a reference to it), and the configuration used. If any of those four change, you have a new model version, full stop.

For storage, use an existing system rather than building one. Cloud provider registries — SageMaker Model Registry, Vertex AI Model Registry, Azure ML — handle versioning, metadata, and stage labels without infrastructure work. If you want to stay vendor-neutral, MLflow’s model registry is mature, runs on a single small VM, and integrates with most training frameworks. What you should not do is build a custom registry. We have never seen a 15-person team ship a custom registry that was better than MLflow, and we have seen several that wasted quarters trying.

The Discipline That Makes It Work

The discipline that makes versioning actually work is a single rule: nothing reaches production unless it has a version tag and a training run it traces back to. This rule has to be enforced at the deployment step, not hoped for at the training step, because data scientists will always optimize for the next experiment over the previous one’s hygiene. If the deployment pipeline refuses to ship an unversioned model, the versioning will happen. If it is a suggestion, it will not.

Pillar 2: CI/CD for ML — Extend What You Already Have

The second failure mode is manual deployment. A data scientist trains a model, copies the artifact to a server, edits a config file, and tells no one. This works exactly once. The second time it produces a broken model in production with no rollback path and no record of what changed.

The fix is not to build an ML-specific CI/CD system. The fix is to extend the CI/CD system your software engineers already use. If you have GitHub Actions, GitLab CI, or equivalent, you have 80% of what you need for ML deployment.

What the Pipeline Should Actually Do

A practical ML deployment pipeline at this scale has four stages, each a job in your existing CI system, and together they take less than a week to build on top of what your software engineers already use.

The first stage is testing. Run unit tests and a small smoke test on a held-out sample. This catches obvious breakages — shape mismatches, dependency version conflicts, a tokenizer that no longer matches the model. Do not try to automate full evaluation here; that belongs in experiment tracking. The test stage is a gate, not a benchmark. It exists to prevent broken models from reaching the next stage, not to decide whether the model is good enough for production. Keep it fast, keep it focused, and resist the temptation to expand it into a full evaluation suite.

The second stage is packaging. Build the model into a deployable artifact — a container image, a serialized bundle, whatever your serving layer expects. The output is a versioned artifact with a digest, pushed to your registry. The key discipline here is immutability: once the artifact is built, it does not change. The same digest that passes the test stage is the one that gets deployed. No re-building at deploy time, no “just tweaking a config” between stages. The artifact is the contract between the training pipeline and the serving environment, and breaking that contract is how production drifts from what was tested.

The third stage is deployment. Roll the artifact to the serving environment, ideally behind a flag or a canary rather than a hard cutover. The deploy step should be identical regardless of who triggered it, so that a 2 AM rollback is the same operation as a Tuesday afternoon release. This is the stage where the investment in the first two stages pays off. If the artifact is immutable and the deploy step is deterministic, then a rollback is just redeploying the previous digest. There is no “roll back the config changes” step, no “figure out which version of the model was running before.” The previous digest is in the audit log, and redeploying it is one command.

The fourth stage is recording. Write the deployment to an audit log — model version, environment, who deployed, when. This is the artifact you will reach for the first time a model behaves oddly and someone asks what changed. We worked with a team that had a churn-prediction model start producing wildly different outputs after a routine update. Because they had a deployment audit log, they could see in seconds that a new model version had been deployed two hours earlier. Without the log, that same investigation would have taken a day of digging through Slack messages and SSH histories.

The Branching Rule

The rule that makes ML CI/CD work at this scale: model deployment changes go through the same pull request workflow as code changes. A new model version is a PR that updates a version reference in the repo. The PR triggers the pipeline. A human reviews and merges. This feels slow to data scientists used to notebooks, but it is the single biggest lever for reducing production incidents, because it makes every model change visible, reviewable, and reversible.

Pillar 3: Experiment Tracking — Make Research Reproducible

The third failure mode is the experiment graveyard. A data scientist runs 200 experiments over three months, finds one that works, and cannot reproduce it. They did not record the hyperparameters consistently, they overwrote the training script, and the dataset they used has since been updated. The winning model becomes an artifact that no one can retrain, which means no one can improve it either.

Experiment tracking solves this, and it is the one piece of MLOps that genuinely pays for itself at any team size. It is also the cheapest to adopt, because the mature tools are free or near-free and require minimal infrastructure.

Choose One Tool and Make It Default

Pick one experiment tracking tool and make it the default for every training run. The realistic options at this scale are MLflow Tracking, Weights & Biases, or Comet. All three have free tiers that cover a small team. The choice matters less than the consistency — a team where half the experiments are in W&B and the other half are in scattered spreadsheets has no experiment tracking at all.

The integration cost is low. For most frameworks it is a few lines of boilerplate at the start of a training script that logs parameters, metrics, and artifacts. Once it is in the shared training template, every new experiment is tracked by default.

What Actually Needs to Be Logged

The trap is logging everything and looking at nothing. A useful setup logs four things on every run: the code commit (or a diff reference), the hyperparameters, the evaluation metrics that matter to the business, and a pointer to the model artifact. That is enough to reproduce a run and enough to compare runs meaningfully.

The other discipline that matters: make the experiment tracker the single source of truth for “which model is best.” When the team decides which model to promote, that decision should be made by looking at the tracked metrics for all candidates, not by someone’s recollection of a run from three weeks ago. If the metrics are not in the tracker, the run did not happen.

A concrete example makes this tangible. A healthcare analytics team we worked with had been tracking experiments in a shared spreadsheet that three people maintained voluntarily. When their lead data scientist left, the spreadsheet had 400 rows, half of which referenced model files that no longer existed on the shared drive. The new hire spent three weeks reconstructing which model was in production and how it had been trained, because none of that information was in a system — it was in the departing employee’s head. After we helped them adopt MLflow, the same information was available in seconds. The cost of adoption was one afternoon of integration work. The cost of not adopting it was three weeks of a senior engineer’s time, repeated every time someone left the team.

Pillar 4: Deployment Patterns — Match the Pattern to the Risk

The fourth failure mode is over-engineering deployment. Teams read about Kubernetes, KServe, and service meshes, and conclude that they need all of it to serve a single model. They do not. The deployment pattern should match the risk and the traffic, not the blog post that impressed the team.

At 5-20 engineers, three patterns cover almost every real workload.

Pattern 1: Batch and Scheduled Inference

For a large class of mid-market ML — scoring jobs, recommendation regeneration, report generation — you do not need a serving endpoint at all. You need a scheduled job that loads a model, processes a batch, and writes the results somewhere. This is the simplest, cheapest, and most reliable pattern, and it is dramatically underused. If your latency budget is measured in hours rather than milliseconds, batch is almost always the right answer. It runs on a single VM or a scheduled container job, has no uptime requirements, and fails loudly instead of silently.

Pattern 2: Managed Real-Time Endpoints

When you need real-time inference, use a managed endpoint rather than rolling your own serving stack. SageMaker Endpoints, Vertex AI Endpoints, Azure ML online endpoints, or serverless options like Modal and Baseten handle the hard parts — autoscaling, versioning, canary rollouts, GPU scheduling — that you do not want to own. The premium you pay is almost always less than the engineering time required to build and maintain an equivalent self-hosted setup at this scale. Reserve self-hosted serving for cases where you have a specific reason: unusual hardware, strict data residency, or cost at a scale the managed pricing model penalizes.

Pattern 3: Embedded and API-Based Models

For many mid-market use cases, the model is not something you serve at all. It is a call to an API — an LLM provider, a managed classification service, or an embedded model running inside an existing application. These still need versioning and monitoring, but the deployment pattern is just a versioned API reference in your code. If you are calling a model over HTTPS from a vendor, your MLOps for that model is mostly prompt versioning, evaluation, and cost monitoring — not Kubernetes.

Common Pitfalls We See

Three patterns show up repeatedly in mid-market MLOps work, and each one quietly undermines the team’s ability to ship.

Building the Platform Before Shipping the Model

The nine-month platform project is the most expensive failure mode. It feels productive because the team is building real software, but it produces zero business value until the first model ships on it. We watched a retail analytics team spend six months building a custom feature store before they had a single model that needed features. By the time the feature store was ready, the data science team had pivoted to a completely different model architecture that did not use the feature store’s schema at all. The platform was technically excellent and practically useless. The teams that succeed at this scale ship their first model on managed services and the simplest possible pipeline, then iterate toward better tooling only when the pain becomes specific and measurable. Every piece of infrastructure should be a response to a problem you have actually hit, not a prediction of a problem you might hit someday.

Optimizing for Scale You Do Not Have

A team serving 10,000 inferences a day does not need the same architecture as a team serving 10 million. We frequently see teams adopt Kafka, distributed training, and multi-region serving long before their traffic justifies it, and the operational burden eats the engineering capacity that should be going into model quality. Match your architecture to your actual numbers, with a comfortable margin, and revisit it when you hit the margin.

Letting the Notebook Be the Source of Truth

The notebook is where research happens, and that is fine. The problem is when the notebook becomes the production system. We worked with a logistics company whose demand forecasting model ran inside a Jupyter notebook on a shared VM. The data scientist who wrote it had left the company, and the notebook had been running on a cron job that someone had set up by adding a line to the crontab and never documenting. When the VM restarted after a security patch, the notebook state was lost, the model started producing zeros, and the operations team did not notice for three days because the dashboard that displayed the forecasts had also been built in the same notebook. Notebooks are not versioned reliably, they are not testable, and they hide state in ways that make bugs impossible to reproduce. The promotion path from notebook to production should be short and well-lit: parameterize the notebook, move the logic into a versioned script, and run it through the same CI/CD pipeline as everything else. If your production model lives inside a notebook, you do not have a production system — you have an accident that has not happened yet.

What Good Looks Like

A mid-market team with workable MLOps is not one with the most sophisticated platform. It is one where, if you ask about any production model, someone can answer four questions within a few minutes: what version is running, what data trained it, how it is being served, and what happens if it needs to be rolled back.

If those four questions have crisp answers for every model your team runs, you have MLOps. Everything else — the orchestration, the feature stores, the custom registries — exists to make those answers reliable and repeatable at larger scale. Build toward the answers, not toward the apparatus. That is how mid-market teams ship ML that actually lasts, without waiting for a platform engineering org they are unlikely to ever hire.