Lately, there's been more focus on creating specialized Small Language Models (SLMs) for high-throughput, real-time applications. But we seem to be at an impasse: we excel at fine-tuning these models, but we're not that great at maintaining them.

While deploying one LLM is like managing an API dependency, deploying multiple domain-specific SLMs – say, one for PII removal, one for intent detection, and yet another for structure-based data extraction – is a different beast altogether.

In this article, you'll learn how to design an architecture that'll help you avoid “model rot” for your whole fleet of SLMs. In particular, we'll focus on how to:

  1. Set up a Model Registry that keeps track of lineage and performance of models

  2. Implement a Gateway Pattern that can perform version-controlled routing

  3. Develop a Manifest-based Delivery System that allows you to deploy to edges

What We'll Cover:

Prerequisites:

To follow along most effectively with this tutorial, you should know Python well and have some experience working with ML models and training them. Experience with MLflow and any other Experiment Tracker would also be great.

The Problem: Model Rot at Scale

The "Black Box" S3 Bucket Problem

Storing model weights in an S3 bucket without an abstraction layer creates an informational void. When a model is just a binary file sitting in storage, you lose the "story" behind it.

The problem is that you have no way of knowing which training dataset, hyperparameters, or evaluation metrics produced that specific file. If a model starts performing poorly in production, you can't perform a Root Cause Analysis (RCA) because the "recipe" used to create the artifact is decoupled from the artifact itself.

This is risky, because if a team member leaves or a repository is archived, those weights become "zombie code." They're running in production, but no one knows how to retrain, reproduce, or safely replace them. This creates a state fear where no one wants to touch the model for fear of breaking the system.

The "final_model_v2_fixed" Naming Problem

Human-readable, "clever" names like final_model_v2_fixed are the antithesis of reproducible engineering.

These names are subjective and lose their meaning within weeks. Does "fixed" mean you addressed a PII leakage bug? Or did you just increase the context window? Did "v2" involve a new dataset, or was it just a change in the learning rate?

This naming convention forces you to manually track external spreadsheets to understand what's actually being deployed. It makes automated rollbacks and A/B testing impossible because the deployment pipeline can't programmatically infer the status, quality, or compatibility of a file named "final_model_v2_fixed."

Why This Leads to "Model Rot"

"Model rot" occurs when your models degrade in performance because they're not being actively monitored, versioned, or updated in response to real-world data drift.

Using black boxes and poor naming results in various problems. First, you can't observe easily. Without metadata, you don't know when a model's performance has fallen below an acceptable threshold.

Also, you can't recover when there's an issue. If a new model version fails, you have no automated path back to the "last known good" state.

Finally, you can't scale. As you add your 6th, 10th, or 20th SLM, the lack of a standardized registry makes the system cognitively impossible for a human to manage.

Here's how you should approach making an AI application ready for production.

How to Build a Model Registry

A Model Registry is the "Single Source of Truth" for your AI infrastructure. It's more than just a storage folder. It's centralized metadata system that binds your model’s lifecycle together. Think of it as a version control system (like Git) specifically designed for ML artifacts.

What is a Model Registry?

At its core, a registry acts as a database and storage interface that binds three distinct layers together:

  1. The artifact: The actual binary weights (for example, .safetensors, .onnx, or .bin files).

  2. The provenance: The "DNA" of the model-including the specific training code version, the hash of the training dataset, hyperparameter configurations, and environment dependencies.

  3. The performance metrics: Benchmarks, validation results, and production monitoring data attached to that specific version.

Why is This Essential for SLM Fleets?

When you are managing multiple domain-specific SLMs (for example, PII removal, intent detection, and data extraction), you encounter three critical operational challenges that only a registry can solve:

  1. Eliminating configuration drift: A registry enforces canonical versioning. It ensures that every node in your fleet pulls the exact, tested artifact that was registered, preventing the "it works on my machine" issues where edge devices run slightly modified versions of your models.

  2. Enabling reproducibility and auditability: If a model underperforms, the registry allows you to instantly pull up the exact dataset, code, and environment used to train it. This allows you to perform Root Cause Analysis in minutes, rather than spending hours trying to reverse-engineer a black-box file.

  3. Managing complex dependencies: As your fleet grows, you will inevitably have models that rely on others (for example, a data extraction model tuned to the output of an intent classifier). A registry allows you to manage these model-to-model dependencies, ensuring that downstream models remain compatible whenever a core model is updated.

To implement this effectively, you should do away with ad-hoc folders immediately. Whether you use MLflow, DVC, or Weights & Biases, the goal is the same: treat your model weights as managed software assets. This turns "deployment" from a risky manual copy-paste operation into a robust, automated pipeline.

Step-by-Step: How to Implement Your Model Registry

Building a registry is less about picking a specific tool and more about enforcing a version-controlled workflow. Here's how to operationalize it in your environment:

Step 1: Standardize Your Artifact Packaging

Stop saving raw files. Instead, package your model weights alongside a model_card.yaml file. This file must contain the model’s unique ID, the hash of the training dataset, the Git commit hash of the training script, and the hyperparameter config.

Step 2: Initialize the Central Registry

Using a tool like MLflow or Weights & Biases, create a project space. When you train a model, your script should automatically "log" the artifact to this registry via an API call rather than manual S3 uploads.

  • Example: registry.log_model(model_weights, metadata=params, tags={"stage": "staging"})

Step 3: Enforce a "Promotion" Lifecycle

Treat your registry like a CI/CD environment. A model shouldn't be "Production" by default. Implement a workflow where a model starts in Development, passes automated benchmark tests in Staging, and is only promoted to Production once it meets your predefined performance thresholds.

Step 4: Automate Metadata Tracking

Every time you run a training job, the experiment tracker should automatically capture the training duration, compute resources used, and evaluation metrics. By linking these to the model ID, you ensure that anyone on your team can click on a model version and immediately see the "proof" that it is ready for production.

The result: You're no longer "managing files." You are querying a database of production-ready assets. When your gateway requests a model, it queries the registry for the Production tag, ensuring it always pulls the right version without you having to touch a single line of config manually.

An MLOps architecture diagram showing the end-to-end automated pipeline process from data analysis and model development to deployment, serving, and continuous monitoring.

This diagram visualizes the closed-loop MLOps lifecycle, mapping the transition from initial data exploration to production deployment.

It distinguishes between the experimental "DS Development" phase-where models are built and tested-and the "Automated Pipelines" stage, which ensures that models are systematically engineered, registered, and continuously monitored.

By highlighting the interconnectedness of the Feature Store, ML Metadata Store, and the Model Registry, the image illustrates how a production-ready system avoids "model rot" by creating a traceable, automated path from raw data to a reliable prediction service.

How to Implement the Gateway Pattern for Version Control

When we treat model weights as code, we stop thinking of them as static files and start treating them as versioned dependencies.

In a production fleet, "using code for weights" means that your deployment system no longer points to a hardcoded file path. Instead, it references a logical version identifier that maps to a specific, immutable set of weights.

Why Semantic Versioning is Required

When you treat models like code, you must adopt Semantic Versioning (SemVer) (for example, v1.2.0) to prevent system instability.

  • Major versions (v2.0.0) indicate breaking changes, such as a change in model architecture or input feature requirements that would break downstream application code.

  • Minor versions (v2.1.0) indicate backward-compatible improvements, like a retrained model with better performance on the same dataset.

  • Patch versions (v2.1.1) indicate hotfixes, such as a model patched for PII leakage or security vulnerabilities.

Without SemVer, your application can't programmatically determine if a new model version is "safe" to load. By using versions, you allow your automated pipelines to reject breaking updates before they ever reach production.

What is the Gateway Pattern?

The Gateway Pattern acts as a dynamic routing layer between your application logic and your model artifacts. Instead of hardcoding a path (for example, model_path = "s3://bucket/intent_v2.safetensors"), your application queries a Gateway. The Gateway manages the mapping between a semantic version and the actual storage location of the weights.

This allows you to perform Hot-Swapping: you can update your configuration to point to a new model version, and the Gateway swaps the weights in memory with zero downtime.

Implementation Example

Here's how you can implement a simple Python gateway to abstract away your model artifacts:

class ModelGateway:
    def __init__(self):
        # The gateway acts as a lookup table for model versions
        self.routes = {
            "intent-classifier": {
                "v1.0.0": "models/intent_v1.safetensors",
                "v2.0.0": "models/intent_v2.safetensors"
            },
            "active_version": "v1.0.0"
        }

    def predict(self, input_text):
        # Application logic calls the 'active_version' dynamically
        model_path = self.routes["intent-classifier"][self.routes["active_version"]]
        return self.load_and_run(model_path, input_text)

    def switch_version(self, version):
        # Hot-swap the version without redeploying the app
        if version in self.routes["intent-classifier"]:
            self.routes["active_version"] = version
            print(f"Traffic successfully routed to {version}")
        else:
            raise ValueError("Version not found in registry.")

The key insight here is that switch_version lets you swap from v1 to v2 in milliseconds. There's no downtime and no pipeline rerun. You update the config-perhaps via a remote central file or environment variable-and the gateway handles the rest.

How to Handle Edge Deployment with a Manifest System

The final challenge is synchronization. When your models are running on edge devices (like laptops or local inference servers), you can't risk large, redundant downloads every time a model is updated. Instead, you should use a Manifest-based Delivery System. This ensures your fleet remains synchronized without forcing users to download gigabytes of weights for minor updates.

The Process: How Manifest-Based Deployment Works

Think of your manifest as a "version control lockfile" for your model weights. The workflow follows these four steps:

  1. Registry update and hash generation: When you deploy a new model version (for example, a fine-tuned LoRA adapter) to your Central Registry, the system calculates a unique hash for the new weight files.

  2. Manifest broadcasting: Your edge application periodically checks a tiny, lightweight manifest.json file hosted on your server. This file acts as the "source of truth," containing the canonical versions and the associated hashes of all required models.

  3. Delta synchronization: Your edge client compares the local manifest with the remote manifest.json. If the hashes don’t match, the client identifies exactly which weight files have changed. It then triggers a targeted download, downloading only the specific delta or the new weight file rather than the entire model architecture.

  4. Atomic swap: Once the new weights are downloaded, the client updates the local reference and triggers the Gateway to hot-swap to the new version in memory.

Example: The Manifest Structure

A manifest file is simple, human-readable, and machine-parsable. It typically looks like this:

{
  "project": "intent-classifier-fleet",
  "models": {
    "intent-v2": {
      "version": "2.1.0",
      "hash": "a1b2c3d4e5f6...",
      "path": "s3://models/intent_v2.safetensors",
      "dependencies": ["base-tokenizer-v1"]
    }
  },
  "last_updated": "2026-06-15T14:30:00Z"
}

Why this scales

This approach mirrors how package managers like npm or pip function:

  • Efficiency: You never download extra, unnecessary files.

  • Reliability: You are always certain of exactly which weight version is active on every single device in your fleet.

  • Resilience: If a download is interrupted, the client verifies the hash of the partially downloaded file, ensuring that corrupt weights never make it into your inference pipeline.

Why This Matters

The end of the “one-size-fits-all” model era is here. Once you’ve established a rigid registry and routing design, you can transition your attention from repairing failed models to maximizing their efficiency.

Here's what you've designed throughout this article:

  1. An Artifact Registry that records lineage, hashing of datasets, and benchmarking for every artifact

  2. The Gateway Design Pattern that allows separating the version of a model from its source code and hotswapping it without any downtime

  3. An Edge Delivery system based on Manifests and syncing the delta of weights between nodes

Treating model weights like code is a good practice and gives you more control. Now that you know what exactly runs, why it runs, and how it can be swapped within milliseconds, you become more than just an AI developer: you can be a AI systems engineer.