Can you remove a background without ever using a green screen?
Yes, but only if the background you happen to have is uniform enough and far enough away in colour from everything in the subject. A key can only thread the gap between how much the background varies and how close its colour comes to something in the subject. Measured with the same subject in front of five ordinary backgrounds, blue sky cleared at tolerance 27 with 0.0% of the subject lost, essentially matching a green screen; a beige wall needed 66 and took 18.3% of the subject with it, a grey wall took 73.0%, and a real room could not be done at all. This page's AI mode cannot stand in for it either: the code runs the colour key first and merges by taking the greater of the two alphas, so it can only put pixels back, never take them away.
What going without a screen actually changes
A green screen is not magic and it is not the only colour that works. What it gives you is control: you picked the colour, you know it appears nowhere on the subject, and you lit it deliberately so that every part of it reads the same. Take the screen away and you lose all three at once.
Now the background is whatever was behind the subject. You did not choose its colour, you cannot guarantee the subject is not wearing something similar, and you almost certainly did not light it evenly. Those are three separate problems, and a colour key has exactly one mechanism for dealing with all of them: a sphere of one radius around one colour.
The one condition that decides it
That mechanism reduces the whole question to a comparison of two distances. The first is how far the background strays from its own average across the frame — its spread. The second is how close the nearest colour in the subject sits to that same average. A tolerance has to be bigger than the first to catch the background and smaller than the second to spare the subject, so it works only when spread is less than the distance to the subject.
Written as a ratio of separation to spread, anything above one is separable and anything below is not. That is the whole test, and it is why the answer to this question is a qualified yes rather than a flat one.
Five ordinary backgrounds, one subject
We put the same figure — skin, dark hair and a dark blue-grey shirt — in front of five backgrounds, gave each the sort of uneven lighting it would really have, and scanned the tolerance slider for the best result each could manage. Lighting was part of the variable: the screen was lit flat at 0.30 stops of falloff, the walls were lit by a window at 0.90, and the sky was nearly even at 0.25.
The green screen, lit flat, varied by 0.067 from its own average while the closest subject colour sat 0.450 away — a ratio of 6.68, and it cleaned up at tolerance 22 with 0.6% of the background left and nothing lost from the subject. Blue sky came out at 0.090 against 0.464, a ratio of 5.14, and cleared at tolerance 27 with 0.5% left and, again, 0.0% of the subject lost.
The walls failed. A beige living-room wall varied by 0.289 while skin sat only 0.229 away, a ratio of 0.79, and the tolerance that finally cleared it took 18.3% of the subject with it. A grey office wall was worse: 0.196 of variation against 0.100 of separation, ratio 0.51, and clearing it destroyed 73.0% of the subject, because grey sits right on top of the shirt.
Why the sky works and the wall does not
Nothing about green is doing the work. Sky wins for two ordinary reasons: a large area of open sky is genuinely even, and its blue is a long way from skin, hair and the shirt we gave the subject. It passes the same test the screen passes, by a similar margin.
The wall fails the other side of the comparison. Beige is close to skin by definition, since both are warm mid tones, and a window-lit wall is far less even than a studio-lit screen — 0.289 of variation against the screen's 0.067. Even where the colour is theoretically distinguishable, the wall's own unevenness eats the gap before the subject's similarity does.
One honest caveat on the sky result: it measures colour separability only. Footage shot against open sky usually has the sun behind the subject, which brings its own problems that have nothing to do with whether the matte can be pulled.
The room fails on structure, not colour
A real room has a wall, a floor, furniture and a window, and no single colour describes it. Measured the same way, its spread was 0.517 — more than seven times the green screen's — while the nearest subject colour was 0.190 away, a ratio of 0.37.
This is the case where raising tolerance is actively harmful. Pushing the slider to its maximum of 100 still left 5.7% of the background standing and removed 91.3% of the subject. There is no radius that contains a bookcase, a floor and a lamp without also containing the person in front of them.
That is the real ceiling on removing a background you did not plan for. Past a certain amount of structure, the problem stops being about colour at all.
Why AI mode cannot rescue it here
It is worth being precise about what the AI checkbox on this page does, because it is not a replacement for the key. Reading the render path, the colour key runs first and always; the segmentation mask is applied afterwards by taking, for every pixel, the greater of the chroma alpha and the AI alpha.
Taking the greater of two values can only ever add opacity. Measured on the beige wall, the colour key at its default left 80.5% of the background behind, and applying the union left it at exactly 80.5% — unchanged, because the model had nothing to add there. The AI can put a lost shoulder back; it can never take a wall away.
There is also no way to run it on its own here. Set tolerance to 0 and the key leaves 99.0% of the frame fully opaque, and since the union can only raise alpha, merging any mask on top of that still leaves 99.0% opaque. The key is always in the chain.
What to do with footage you already have
Run the comparison before you try anything. Sample the background in several places and see how much it varies, then find the closest thing in the subject to that colour. If the gap is comfortably larger than the variation, a key will work, whatever the colour happens to be — sky, a painted wall, a flat backdrop.
If the gap is small, no amount of slider work will fix it, and this is where hand masking earns its time. If the background has real structure in it, colour is the wrong tool entirely and you need something that understands shape rather than hue.
And if you can reshoot, the cheapest fix is still the one the ratio rewards: put something flat behind the subject, in a colour nobody is wearing, and light it evenly. That is all a green screen ever was.
Related questions
Is a green screen actually necessary?
No. Blue sky measured a separation-to-spread ratio of 5.14 against the green screen's 6.68, and cleared at tolerance 27 with 0.0% of the subject lost. What matters is that the background is even and far in colour from the subject, not that it is green.
Why does a plain wall fail if it is one colour?
Because it is not one colour once a window lights it, and because its colour is close to things you cannot change. The beige wall varied by 0.289 from its own average while skin sat only 0.229 away, so a tolerance wide enough to catch the wall was already inside the subject.
Can AI subject mode remove a background on its own?
No. The colour key runs first and the mask is merged by taking the greater of the two alphas, which only adds opacity. On the beige wall the 80.5% of background left behind was unchanged by the union, and at tolerance 0 the frame stays 99.0% opaque no matter what the model returns.
How can I tell quickly whether my footage will key?
Compare two numbers: how much the background varies across the frame, and how close the nearest subject colour is to the background's average. If the second is larger, it will key at some tolerance; if not, no setting will separate them and you should plan on masking by hand.