When does AI subject keying beat a colour key — and when does it fail?
It beats a colour key in exactly one case: when the key is eating your subject. The two results are merged with a union — a pixel stays opaque if the colour key or the person model believes in it — so the AI can only add, never subtract. On our test scene a green shirt next to the screen colour went from 81.45% of the subject wrongly erased to 0.00%. It does nothing for the thing people expect it to fix: on a beige room background, background residue measured 100.00% before the mask and 100.00% after.
The one line that decides everything
The tool runs the colour key first, then runs a person model over the same frame, then merges the two with a single expression: Math.max(chromaA, aiA). That is the entire relationship between the two methods. There is no weight, no blend mode and no threshold you can tune.
Two consequences follow from a union, and neither is negotiable. It can reduce how much of your subject gets erased, and it can never reduce how much background survives. Every measurement below is one of those two facts showing up in a different scene.
The checkbox ships checked, so if you have never touched it you have been running both methods at once and taking the more generous answer from each.
Where it genuinely wins
A colour key keys colour, and anything on your subject that sits near the screen colour is collateral damage. We rendered the standard 640×360 test scene with the subject wearing #00A030 against a #00B140 screen: at the default sliders the keyer erased 81.45% of the subject pixels.
With the person mask merged in, wrongly erased subject pixels fell to 0.00%. A mask that under-claims by two pixels — the model missing a hand or a wisp of hair — still left only 1.43%, and one under-claiming by four pixels left 5.01%. The model does not know what colour the shirt is, so it puts the shirt back.
Background residue did not move at all: 22.95% before, 22.95% after. Everything the mode buys you, it buys on the subject side of the edge.
Where it does nothing at all
The natural expectation is that a person model handles a background that is not a flat colour. We tested that: a beige wall with a dark shelf and a grey floor, no green anywhere. The colour key removed nothing — 100.00% of background pixels stayed opaque — and after merging a perfect person mask, 100.00% still stayed opaque.
That is not a tuning problem, it is the union doing what a union does. The colour key says "not the key colour, keep it" and votes 1; the model says "not a person, drop it" and votes 0; the maximum is 1. Nothing the model is certain about can lower an alpha the colour key raised.
If you are cutting against a beige wall, do not reach for AI mode — reach for the eyedropper. The keyer will key any colour you give it, and a flat wall is a good key target. The person mask is then worth keeping only as a safety net for whatever else on your subject happened to match the wall.
What it costs when the model over-claims
The other half of the union is the bill. Because the higher alpha always wins, every pixel the model claims that is not really your subject comes back — and it comes back as the original screen colour, not as transparency.
Measured against the true silhouette: a mask over-claiming by one pixel added 1,090 half-transparent pixels around the edge and raised the mean edge error from 38.12% to 40.81%, while solid background residue stayed put. Over-claim by three pixels and 1,244 background pixels came back fully solid — residue rose from 22.95% to 23.59% and edge error to 49.29%. At five pixels, 2,623 background pixels returned and residue reached 24.31%.
The damage arrives as soft haze first and as a green rim second. One pixel of over-claim is close to invisible; three is a visible fringe on a subject that was clean before you switched the mode on.
Why the AI edge is soft before you do anything
The mask is drawn onto a canvas the size of your source frame before it is read, so one mask pixel covers several source pixels and the boundary between them is interpolated. We reproduced that by rendering our own mask at a fraction of frame width and stretching it back: the half-transparent band measured 4 px wide at full resolution, 6 px at half, 10 px at a quarter and 19 px at an eighth.
A model whose mask is much smaller than your frame therefore cannot hand you a crisp hair edge at 1080p, however good its classification is. That is often acceptable — a slightly soft cut-out beats a jagged one — but it is the wrong tool when the job needs a razor edge on fine detail.
When the model has no opinion
Before merging, the code counts pixels above 127 and divides by the frame. Under 1% and the whole mask is discarded, so you silently get the pure colour key. Two situations trigger it: the model returns empty masks while it warms up, and it returns an empty mask when it finds no person.
The second one is a real size limit. In our synthetic standing figure, filling the frame measured 13.38% coverage; at 60% of frame height, 4.92%; at 30%, 1.23%; at 25%, 0.85% — under the gate, mask thrown away. In pixels the gate is 9,216 at 720p, 20,736 at 1080p and 82,944 at 4K.
The model shipped here is a people model: this build loads the landscape variant of a selfie segmentation model. A product, a vehicle or a pair of hands is not what it looks for, and if it reports nothing the gate quietly hands the frame back to the colour key.
On video, the clock is part of the deal
Frames are pushed through the model one at a time on a serial queue, and each call carries a five-second guard. If no result comes back in time the call resolves as nothing, and that frame keeps the colour-key result alone.
The failure is not a bad frame, it is an inconsistent one: its neighbours got the union and it did not, so the edge changes shape for a single frame. If your footage keys cleanly, switch the mode off for the render — same result, faster, without that risk.
So: on or off?
Leave it on when the colour key is harming the subject — clothing near the screen colour, an unevenly lit screen that is pulling parts of your subject into the key window, or hair the key is eating. It is the only thing here that can put the subject back.
Turn it off when the problem is on the background side: leftover screen, a backdrop that is not a flat colour, or a halo you cannot get rid of. The union cannot subtract, so fix those with the key colour, the sliders or the lighting.
A fast way to tell which side you are on: toggle the checkbox and watch the edge of the subject rather than the background. If the subject fills in, keep it. If a haze of screen colour appears, lose it.
Related questions
Does AI subject mode work if I have no green screen?
Not as implemented. On our beige room scene, 100.00% of background pixels stayed opaque before the person mask and 100.00% after, because the two results are merged with a union that can only raise alpha. Pick the background colour with the eyedropper and key that instead.
Why did a green rim come back when I turned AI mode on?
The model claimed pixels that are not really your subject, and the union keeps whichever method votes higher. In our runs an over-claim of three pixels brought 1,244 background pixels back to fully solid and lifted residue from 22.95% to 23.59%.
Does it work on products, or only people?
The shipped build loads a selfie segmentation model, so it looks for people. On a non-person subject it may report nothing at all, and under 1% mask coverage the whole mask is discarded — 20,736 pixels at 1080p — leaving you with the plain colour key.
Should I leave AI mode on for a video render?
Only if the colour key is damaging the subject. Frames go through the model one at a time with a five-second guard per frame, and a frame that times out gets the colour key alone, which makes its edge differ from the frames around it.