Automated Feature Flag Removal: What a Bot Must Refuse

Every feature flag platform can tell you which flags are dead. Detection is a database query against evaluation traffic, and it has been a solved problem for years. Deleting the flag from your source code is the harder half, and the handful of tools that attempt it disagree about how it should work. That gap is wider than it looks.

A stale-flag report is a list. Lists do not get worked. The flag stays in the code because removing it means opening the file, reading the branch, deciding which arm survives, deleting the other one, and convincing yourself you did not just change behavior in production. That is twenty minutes of careful work for a change that ships nothing, and it loses to every other item in the sprint, every sprint, forever.

The useful question is whether a tool can be trusted to remove a flag without a human reading every diff. Most of the answer lives in what such a tool declines to do.

Key Takeaways

• Detection and removal are different problems. Detection reads telemetry, removal rewrites your source, and only one of them can break production.
• Automated removal is no longer rare. What varies is whether the same input always produces the same diff, and whether the tool can name the shapes it will refuse before it runs.
• A flag removal that produces valid, compiling code can still be wrong. Mock stubs are the clearest case, and the compiler has nothing to say about them.
• Correctness has to be enforced outside the rewrite rules, by something that parses the whole file afterward and throws the work away if it looks wrong.
• The parser you choose decides whether the tool works at all. A syntax check with a high false-positive rate produces no diffs forever and reads as “nothing to clean up.”
• The list of shapes a cleanup tool refuses to touch is the specification. A short refusal list usually means the hard cases have not been found yet.

1. The half that got solved, and the half that did not

Flag detection works because the platform already has the data. Every evaluation is a request, requests get counted, and a flag whose count has been zero for ninety days is a strong candidate for deletion. Featureflip exposes this through find_stale_flags, and so does most of the market in some form.

Then the report lands, and nothing happens.

This is why flag debt is so durable. Removing a flag is not one edit. You delete the read, then fold the branch it guarded, then remove the variable it was assigned to, then delete the import if nothing else uses it, then check whether the value was exported to some other module, then run the tests, then convince a reviewer none of it changed behavior. Each step is easy. The sequence is tedious, it carries real risk, and the payoff is a smaller file.

Automating the report was the easy part, and the market stopped there for years.

That is changing, and the question has moved with it. Automated removal is no longer rare. What differs now is how the removal happens: a language model reading your code and proposing an edit, or a transform that parses the syntax tree and applies rules to it. Both open a pull request. The second kind produces the same diff every time, and can tell you in advance which shapes it will refuse to touch.

That second kind is what we built. The Featureflip flag cleanup Action runs in your CI, reads dead flags from the public API, and opens one pull request per flag across thirteen languages. Merging that pull request can archive the flag it removed, so the code and the flag list stop drifting apart.

The rest of this post is about what a tool like that has to refuse, because the refusals are what make it safe to leave switched on.

2. What “safe” has to mean before you let a bot commit

The obvious bar for an automated rewrite is that the result compiles. It is also nowhere near enough.

Consider a test that stubs a flag read with Mockito:

when(client.boolVariation("checkout-v2", ctx, false)).thenReturn(true);

The flag is dead and serves true, so the naive rewrite folds the read to its value:

when(true).thenReturn(true);

That compiles. Javac has no objection, the test suite runs, and the assertion that used to depend on a stubbed flag now depends on nothing at all. The stub silently stopped stubbing. Worse, the flag key is gone from the file, so no future run of the tool will ever look at this line again. The failure is permanent and invisible.

Mocking libraries do this because they stub by performing the call and capturing the invocation it registers. The text of the read is the payload. Fold it and you have deleted the payload while leaving the wrapper standing.

Go makes the same mistake impossible to miss, by accident:

m.EXPECT().BoolVariation("checkout-v2", ctx, false).Return(true)

Folding that read leaves true.Return(true). Go’s bool is a builtin with no method set, so a selector on a boolean literal is broken code by construction, and the build fails immediately. Java hides the same error and Go cannot. One language got a compiler diagnostic for free and the other needs the tool to know the names of the stubbing entry points.

The honest version of this story is that the Go case shipped before it was refused, and a customer’s build was what reported it. Most of the guards in a tool like this were added the same way, after something got through.

3. Two gates that sit outside the rewrite rules

The tempting design is to make each rewrite rule guard itself: match only the shapes you are sure about, and the output is safe by construction. This does not hold up. A tree-sitter query written as a list of safe forms will, twice, let a form outside the list through, because the list was written by someone who did not think of it.

So the guarantees belong somewhere else, in code that parses the entire file after the rewrite and knows nothing about which rule produced it.

The first gate re-parses every rewritten file and discards the transform if it introduced a syntax error, put a reserved word where a name should be, stranded a keyword where an expression has to be read, or let two surviving statements fuse together under JavaScript’s automatic semicolon insertion. The discard is all-or-nothing across every file in the change, because a partially applied rewrite is worse than none.

All four checks run differentially against the input, which is the part that makes them usable. A file the grammar already disliked stays eligible, and only newly introduced breakage is rejected. Without that, any repository with one unusual file would see the whole run refuse itself. Measured across 463 real TypeScript and TSX files and 833 simulated deletions, in both semicolon-terminated and semicolon-free style, that gate produced zero false positives.

The second gate runs before the rewrite and asks a narrower question: is the name bound to this flag read bound anywhere else in the same file? If it is, the rules that inline the constant are withheld for that file, whatever the other binding looks like. It keys off the grammar’s binding fields, which are a small closed set, rather than off initializer node types, which are an open-ended set that TypeScript keeps growing. That is what makes it sound without doing real scope analysis.

Both gates are conservative on purpose. A same-named variable in an unrelated function is enough to withhold the cleanup, and the cost of that caution is one leftover const on = true; for someone to delete by hand.

Neither gate is a correctness proof, and saying otherwise would be a lie. Three of the first gate’s four checks ask whether the result still parses as intended, and a wrong rewrite can parse perfectly. They close specific, known failure classes. A new failure class needs a new check, and until someone writes it, the diff is the only thing standing in the way.

4. The parser decides whether the tool works at all

One implementation detail decides more about this tool than any of the rewrite rules do.

The first gate needs something that can tell a broken file from a working one. For JavaScript and TypeScript the obvious candidate is node --check, which is free, already installed, and purpose-built. It was evaluated and rejected, because it parses JavaScript rather than TypeScript. Run against 431 real TypeScript files from a production codebase, it reported 286 of them as syntax errors.

Think about what that failure looks like from the outside. The gate is doing its job, every candidate file is judged broken, every transform is discarded, and the tool cheerfully reports zero changes on every run. Nobody files a bug, because the output is indistinguishable from a clean codebase. A tool that is wrong in this direction is far more dangerous to trust than one that crashes.

5. The refusal list is the specification

Once the gates are in place, the remaining design work is deciding what the tool will not touch. This list is long, and its length is the point.

Your own wrapper is the common case, and it is left alone. Most real codebases do not call the SDK at the read site. They call useFlag("checkout-v2") or isOn("checkout-v2"), some thin function that adds logging or a default. Matching is by method name plus key string, so a wrapper is invisible, and the tool reports that it has nothing to change for a flag your code reads in forty places. That is the correct answer, and calling it a gap gets the risk backwards. Rewriting an unknown function because its first argument happened to be a flag key is how you delete a real call. It is also the single most common reason a cleanup run reports no changes for a flag you can grep for everywhere.

An exported binding stays. Given export const on = client.boolVariation("checkout-v2", ctx, false), the read is replaced by its value and the declaration is kept. The modules that import that binding do not contain the flag key, so they are never in the candidate set, and the tool has never parsed them. Removing the export would break files it cannot see.

A Python read bound at module or class level refuses the whole flag. Same reasoning, different outcome. Inside a function the binding is removed and its references replaced. At module level the name is public API, and leaving use_legacy = True standing would be permanent, because the key is gone and no later run will revisit it. So the tool declines the flag entirely and says which file stopped it.

A read inside a C# expression tree is untouchable, because the lambda describes the call instead of performing it, and a mocking library reads the text at run time. The tell is not the method name, it is that the read’s receiver is the lambda’s own parameter, which covers any library taking an Expression<> while still rewriting a genuine Func<> lambda that reads off a captured client.

There are more. Optional calls, non-boolean accessors, un-awaited OpenFeature reads that hand back a promise instead of a value, keys held in reassignable bindings, keys hoisted to a constant in a different file. Each row exists because folding it would change behavior, and each one was decided in advance instead of being discovered by a customer.

6. What running it actually looks like

The Featureflip flag cleanup Action runs inside your CI, in your repository, holding your token. It reads removal candidates from the public API and opens one pull request per flag. It supports TypeScript, TSX, JavaScript, PHP, Ruby, ERB, Dart, Java, Go, Python, Kotlin, C# and Swift.

name: Featureflip flag cleanup
on:
  schedule:
    - cron: '0 9 * * 1'
  workflow_dispatch: {}

permissions:
  contents: write
  pull-requests: write

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: canopy-labs/featureflip-flag-cleanup-action@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          api-token: ${{ secrets.FEATUREFLIP_API_TOKEN }}
          org: my-org
          project: my-project
          staleness: dead
          dry-run: true

Three properties decide whether a team leaves this switched on.

It reads from the flag platform and writes nothing. A Viewer token scoped to one project is enough for this job. Nothing archives, toggles, or deletes a flag. Removing the code is proposed as a pull request, and what happens to the flag afterward stays a human decision. Section 7 covers the opt-in workflow that changes this, and why it deliberately runs somewhere else.

Start with dry-run: true. It computes and prints every diff without contacting GitHub at all. Every refusal a real run can produce is reachable in a dry run and reported identically, so the preview cannot approve something the next real run turns down. The default is false, which means omitting it opens real pull requests on the first run. That is deliberate and it is documented, but read it twice before you copy the snippet without the flag.

One run is capped. max-prs defaults to ten. A project with two hundred dead flags would otherwise get two hundred branches on the first morning, and if that run was mistaken, all two hundred were. Nothing is dropped permanently, because flags that already have a branch are not proposed again and the next run continues where the last one stopped.

7. The flag itself, after the code is gone

Merging the pull request deletes the code. The flag stays in Featureflip, live, until somebody archives it, and that is one more thing to remember at the exact moment the work feels finished. It is a small step, which is why it gets skipped, and skipping it leaves a flag list that slowly stops matching the codebase.

mode: archive-on-merge handles it, as a second workflow on a different trigger:

name: Featureflip archive on merge
on:
  pull_request:
    types: [closed]

jobs:
  archive:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - uses: canopy-labs/featureflip-flag-cleanup-action@v1
        with:
          mode: archive-on-merge
          api-token: ${{ secrets.FEATUREFLIP_ARCHIVE_TOKEN }}
          org: my-org
          project: my-project

Keeping it in a separate workflow is deliberate. Archiving needs a token that can write to your flags, while reading removal candidates needs one that cannot, and separate workflows can carry separate secrets. The weekly job that scans your repository keeps a token whose worst case is disclosing flag names.

It works out which flag from the branch name, which the Action encodes reversibly when it opens the pull request. Nothing is read from the title or the body. Edit them freely, and squash the merge if you like.

Close a pull request without merging it and nothing is archived. Declining a removal is a decision, and it has to leave the flag exactly as it was.

Archiving can also refuse, and the message says which case you are in. A flag that another live flag lists as a prerequisite cannot be archived until that prerequisite is gone. Neither can one with a scheduled change still pointing at it. Both arrive after the code has already merged, which makes them a note about leftover work.

8. Where this leaves flag debt

The four-step cleanup loop does not change. You still detect, triage, remove, and prevent, and the cleanup playbook still describes the mechanics of doing it by hand, which you will need for the flags a tool refuses to touch.

What changes is which step is expensive. Removal was the step that consumed a careful engineer’s afternoon and therefore never happened. When it arrives as a reviewable pull request with one flag in it, the cost drops to reading a diff, and reading a diff is something teams already do all day. Retiring the flag behind it costs nothing at all once you have switched that on.

That leaves triage as the expensive step, which is a better place for human attention anyway. Whether a flag is a finished release, an abandoned experiment, or a permanent operational switch is a judgment about intent that no static analysis can make. Deleting the code once you have decided is mechanical.

The shorter version

• Detecting dead flags is a solved problem. Removing them safely is not, and the gap is why flag debt survives.
• Automated removal is no longer rare. Whether a run is repeatable, and whether the tool declines the shapes it cannot handle, is what separates one from another.
• A rewrite that compiles can still be wrong. Mock stubs fold into code the compiler accepts and the test suite no longer tests.
• Correctness lives outside the rewrite rules: a differential re-parse of every changed file, plus a file-scoped check on shadowed bindings.
• The syntax gate’s parser choice is load-bearing. A high false-positive rate produces silence, which reads as a clean codebase.
• The list of shapes the tool refuses is the specification. Wrappers, exported bindings, module-level Python, and expression trees are all deliberate declines.
• Run it in dry mode first, cap the pull requests, and give the scanning job a read-only token.
• Archiving the flag once its removal merges is opt-in, runs as its own workflow, and carries its own write-scoped secret.

Frequently asked questions

Can feature flag removal be fully automated?

The code removal can be, for the shapes a tool can rewrite with confidence. The decision cannot. A tool can tell that a flag has served one value for ninety days and can produce a correct diff that deletes the dead branch, but whether that flag was a finished release or a kill switch you intend to keep is a judgment about intent. That is why removal arrives as a pull request instead of a commit.

What happens if the tool cannot safely remove a flag?

It leaves the code alone and says so. Depending on the shape, that means either no diff for that flag at all, or a refusal that names the file and exits with a non-zero code. A wrapper function around the SDK produces no change, and a mock stub holding the flag read refuses the whole flag. Neither case produces a partial rewrite.

Does automated cleanup touch my feature flags themselves?

Only if you switch that on. By default the Action reads removal candidates over the public API using a token you supply, and a read-only Viewer token scoped to a single project is enough. It archives, toggles, and deletes nothing, and retiring the flag after the pull request merges stays your call. A separate opt-in workflow, mode: archive-on-merge, archives a flag once its removal pull request is merged. That one needs a token with write access, so it runs on its own trigger and carries its own secret. The read-only token stays where it is.

Which languages does automated flag removal support?

TypeScript, TSX, JavaScript, PHP, Ruby, ERB, Dart, Java, Go, Python, Kotlin, C# and Swift. Coverage is per flag-read shape, not per language, so a file in a supported language can still contain a read the rules decline to rewrite.

How is this different from a stale flag report?

A report tells you which flags are dead and leaves the work with you. Most flag platforms produce one. The difference is whether anything acts on it: a list of forty stale flags competes with feature work and loses, while forty small pull requests compete with nothing and get reviewed.

Featureflip includes stale flag detection and automated removal on every plan, including the free one, because flag debt is not a problem that only large teams have. Start with a free account and point the Action at your repository in dry-run mode to see what it would propose.