The riskiest change in an AI feature is usually the one that looks smallest: swap the model, or edit the system prompt. A new model can be cheaper and sharper on your evals and still regress on the one prompt shape your users actually send. The usual way to make that change, an environment variable and a redeploy, gives you exactly one setting for the whole fleet and one way to undo it: ship again. There is no way to send the new model to 10 percent of users, keep each of them on it consistently, and pull it back in seconds when it misbehaves.
A feature flag holding the model configuration gives you all three. This is the hands-on companion to using feature flags in an AI or LLM app: where that page covers the shape of the idea, this one is the walkthrough. Put the model, prompt, and settings in one JSON flag, read it on the request path, ramp a new model from 10 to 100 percent with per-user sticky bucketing, and make the fallback config a kill switch. The examples use the Python SDK; the same flag works from any language Featureflip supports.
Key Takeaways
- Hold the model name, system prompt, temperature, and token limit in one JSON feature flag, so changing the model or tuning the prompt is a dashboard edit, not a deploy.
- Read it on the request path with
client.variation("llm-model-config", ctx, default=SAFE_CONFIG). The default is a known-good config the SDK serves during an outage, so the safe model doubles as the fail-safe fallback.- Migrate a new model with a percentage rollout: 10, then 50, then 100. Deterministic sticky bucketing keeps each user on one model as you ramp, so a single conversation never switches models between turns.
- The flag is a kill switch: flip it or ramp back to zero and every SDK reverts within a second or two over streaming, fleet-wide, with no redeploy. Reads come from cached config, so the switch survives a control-plane blip.
- The flag owns the config and the routing decision. Whether the new model is better, and what it costs, stays with your eval and observability stack.
1. Why a model swap needs a flag, not an env var
Model choice is configuration that changes often and can go wrong quietly, and an environment variable is the wrong container for it in three specific ways.
It is all-or-nothing. A MODEL=gpt-4o variable is one value for every process. You cannot give it to a tenth of your users while you watch the numbers, because there is no per-user dimension to an env var. The only rollout it supports is “everyone, on the next deploy.”
It is not sticky. Push a config change or restart a pod and every in-flight user flips at once. If a customer is three turns into a conversation, their next reply comes from a different model with a different temperature, and the thread reads like two assistants stitched together.
It is not fail-safe. When the new model starts timing out or returning garbage, the old value is gone from the running process. Recovering means editing config and shipping again, on the worst possible day to be running a deploy.
Holding the model config in a flag fixes all three. A percentage rollout gives you the per-user ramp. Deterministic sticky bucketing keeps each user on one model across the ramp. And the default you pass at evaluation is a fallback the SDK serves when it cannot reach the control plane, so a bad model is one dashboard flip away from gone rather than one deploy away. In short, holding a model name and prompt in a flag is a form of remote config, with a rollout and a kill switch attached.
2. Put the model config in one JSON flag
Create a JSON flag, llm-model-config, whose value carries the whole configuration for the feature: the model name, the system prompt, and the generation settings. One flag holds it all, so the pieces change together.
In your code, build the client once and read that flag with a known-good default. In the Python SDK a single variation call covers every type: pass a dict default and you get a dict back, so a JSON flag needs no special method.
import osfrom featureflip import FeatureflipClient
client = FeatureflipClient(sdk_key=os.environ["FEATUREFLIP_SDK_KEY"])
# The default is a model config you trust. The SDK serves it if it# cannot reach Featureflip, so an outage falls back to a known-good# model rather than an experimental one.SAFE_CONFIG = { "model": "gpt-4o-mini", "system_prompt": "You are a helpful support assistant.", "temperature": 0.3, "max_tokens": 1024,}
def model_config_for(user): ctx = {"user_id": user.id, "tenant_id": user.tenant_id, "plan": user.plan} return client.variation("llm-model-config", ctx, default=SAFE_CONFIG)The default argument is doing double duty. It is the value returned if the flag is missing or the SDK is briefly cut off from Featureflip, and it is your statement of what “safe” means for this feature. Make it a model and prompt you would be comfortable serving to everyone, because during an outage that is exactly what happens. The Python quickstart has the minimal client setup; the rest of this guide is what you build on top of it.
3. Read the config on the request path
Reading the flag happens right before you build the model request. Evaluation is an in-memory lookup against a ruleset the SDK already holds, so it adds no network hop ahead of the model call, where latency is felt most.
def generate_reply(user, user_message): config = model_config_for(user) return llm.chat( model=config["model"], temperature=config["temperature"], max_tokens=config["max_tokens"], messages=[ {"role": "system", "content": config["system_prompt"]}, {"role": "user", "content": user_message}, ], )The application code never names a model. It asks the flag which configuration applies to this user and builds the request from the answer. That indirection is the whole point: the model, the prompt, and the settings are now values you change from a dashboard, and the code that calls the provider stays fixed.
4. Roll a new model out 10, then 50, then 100
To migrate from the current model to a new one, give the flag two variations, a current-model config and a new-model config, and put a percentage rollout on it. Start the new variation at 10 percent, watch your evals and cost, then raise it to 50 and 100 as the signal holds. Rolling back is the same move in reverse and takes effect in seconds.
The property that makes this safe for a conversational feature is stickiness. Bucketing is deterministic: the SDK hashes the context key you bucket on, so the same user_id lands in the same bucket every time, in every worker and every replica. A user who is in the new-model group stays there as you raise the percentage, so a single conversation does not switch models between turns, and you never need sticky sessions or a shared cache to hold the assignment.
Because a JSON flag serves a whole config object, this is a multivariate rollout, not just on versus off: each variation is a full model configuration, and the rollout decides which one a given user gets. You raise and lower the percentage from the dashboard, and every connected SDK reflects the new split on its next evaluation. The mechanics are the same rollout strategies a UI feature uses, pointed at model choice.
5. Make the fallback a kill switch
A gradual rollout limits the blast radius of a bad model. A kill switch removes it. Both come out of the same flag.
The manual switch is the flag itself. When a model starts hallucinating, slowing down, or running up cost, ramp its rollout back to zero or flip the flag to the safe variation. Every SDK picks up the change over a Server-Sent Events stream within a second or two, fleet-wide, with no redeploy. Because reads are served from a local cache of the config, the switch keeps working even if the control plane is briefly unreachable, and the default you passed is the ultimate backstop.
The automatic switch is in your code. Wrap the model call so a failure retries with the safe config rather than surfacing an error to the user:
def generate_reply(user, user_message): config = model_config_for(user) try: return call_model(config, user_message) except (ModelError, TimeoutError): # The chosen model failed. Fall back to the known-good config. # Flip 'llm-model-config' in the dashboard to move the whole # fleet off the bad model in seconds, with no redeploy. return call_model(SAFE_CONFIG, user_message)The one rule that keeps this honest: the default and the fallback are the model you trust, never the experimental one. If the new model is the fallback, a control-plane blip or a caught error routes traffic to the unproven model, which is backwards. Default to known-good, and let a rollout or a rule opt users into the new one. The same pattern powers a general feature flag kill switch for any code path, not just model calls.
6. Route a model or prompt per tenant
Some customers need a specific model regardless of where the migration stands: an enterprise account pinned to a particular model, a cohort on a different prompt, or a tenant held on the old model while everyone else moves. A targeting rule handles it.
Add a rule on an attribute you already pass in the context, such as tenant_id or plan, and serve that segment a specific variation. Rules run top to bottom and the first match wins, so an override for a named tenant sits above the broad rollout and takes precedence over it. Pinning a customer to a model, or releasing them back into the migration, is a dashboard change that every SDK reflects on its next evaluation. The attributes and operators available are covered in the targeting docs.
7. What the flag does not do
A flag delivers and routes the model configuration inside your application. It is deliberately not an AI platform, and keeping the boundary clear keeps you from asking it to do a job it cannot.
It does not tell you the new model is better. Rolling a model to 100 percent because it did not throw is not the same as knowing it improved your responses. Scoring output quality and catching regressions is the work of an eval or observability tool. The flag ships the model; those tools grade it. If your goal is a statistically valid comparison rather than a safe migration, that is an experiment, and the delivery half looks like A/B testing with feature flags with your analytics calling the winner.
It does not track token cost or latency. Moving traffic between models changes both, but the numbers live in your metrics and tracing stack. The flag can shift the traffic; observability tells you what the shift cost.
It is not a prompt store or a secrets manager. The flag holds the prompt that is live right now and switches it. Authoring, versioning, and diffing every prompt you have written belongs in your repo or a prompt tool. And provider API keys stay in your secrets manager, never in the flag value, especially since client SDKs evaluate config in the browser.
Those boundaries are the same honest split the AI use-case page draws: the flag owns config and routing; your eval, observability, prompt, and secrets tools own the rest. Many AI features need the flag from day one and add the other tooling later, when the migration should be driven by data rather than by whether the model errored.
The shorter version
Swapping an LLM or editing a system prompt is a config change that an environment variable handles badly: all-or-nothing, not sticky, not fail-safe. Put the model name, system prompt, temperature, and token limit in one JSON feature flag instead, and read it on the request path with client.variation("llm-model-config", ctx, default=SAFE_CONFIG), where the default is a known-good config the SDK serves during an outage. Migrate a new model with a percentage rollout, 10 then 50 then 100, and deterministic sticky bucketing keeps each user on one model so a conversation never switches mid-thread. The flag doubles as a kill switch: ramp to zero or flip to the safe variation and every SDK reverts over streaming in seconds, no deploy, and the switch survives a control-plane blip because reads come from cached config. Add a targeting rule to pin a tenant to a specific model. What the flag does not do, judge whether the model is better or what it costs, stays with your eval and observability stack.
Frequently asked questions
How do I roll out a new LLM model gradually?
Put the model configuration in a JSON feature flag with two variations, the current model and the new one, and add a percentage rollout. Start the new variation at 10 percent, raise it to 50 and 100 as your evals and cost hold, and roll back in seconds by lowering the percentage. Deterministic sticky bucketing means the same user stays in the same group as you ramp, so the users already on the new model are not reshuffled when you widen its share. You change the percentage in the dashboard, and every SDK picks up the new split over its stream with no redeploy.
How do I keep a user on the same model during a conversation?
Bucket the rollout on a stable key such as the user id or a conversation id. Featureflip hashes that key deterministically, so it maps to the same bucket in every worker and every replica, and a user assigned to the new model at 10 percent is still on it at 50 and 100. That stickiness is what stops a single conversation from switching models between turns, and it holds without sticky sessions or a shared assignment cache because every SDK computes the same bucket from the same key.
Can I use a feature flag as an LLM kill switch?
Yes. Hold the model config behind a flag and give the evaluation a known-good fallback. When a model starts erroring, hallucinating, or running slow or expensive, flip the flag to the safe variation or ramp its rollout to zero, and every connected SDK reverts within a second or two over streaming, fleet-wide, with no redeploy. Because reads are served from a local cache of the config, the switch keeps working even if the control plane is briefly unreachable, and the default you pass at evaluation is the ultimate backstop.
Should I store the system prompt and temperature in a feature flag?
Store the live selection there. One JSON flag can hold the model name, system prompt, temperature, and max_tokens as a single value, so the whole configuration changes together and ships to every instance at once. Keep the authoring and version history of your prompts in your repo or a dedicated prompt tool, and keep provider API keys in your secrets manager rather than the flag. The flag is where you choose which configuration is live, not where you archive every version or store secrets.
Do feature flags add latency to each model call?
No. With a server SDK, flag evaluation is an in-memory read against a ruleset the SDK already holds, so reading the model config is a local lookup with no network round trip. That read runs in the time of a dictionary lookup and finishes long before the model call, which is where the real latency is. The SDK keeps the ruleset current over a streaming connection in the background, so a config change reaches your process without adding any per-request work.
Featureflip is a focused, flat-priced feature flag platform, and it puts your AI feature’s model, prompt, and settings behind one flag: swap models without a deploy, migrate gradually with sticky per-user bucketing, and kill a bad model in seconds with a fail-safe fallback. Because it uses flat pricing with unlimited evaluations, a high-throughput AI service never sees its flag bill scale with request volume. And when an AI assistant is the one writing the code, it can manage the flags too: the MCP server lets Claude Code and Cursor create, ramp, and retire flags from the editor. Read the SDK overview to pick your language, see the pricing page for the numbers, and start on the free Solo plan without a credit card.