When is hand masking better than a chroma key?

Q: Is hand masking better than chroma keying?

Sometimes. It wins when the subject's colour is within about 0.2 of key green (the keyer cannot make the body opaque at all there: 52 dE against the hand's 17), when the edge is defocused past roughly a 14 px ramp, when the subject is semi-transparent and you author per-pixel alpha, and on stills. On an ordinary hard edge with a subject far from green, a tuned keyer wins: 10.7 dE against 17.6 for a 0.25 px hand.

Q: Why does my keyed still look worse than a hand-drawn mask?

Usually because the subject is near the key colour or the edge is soft. The keyer's alpha is a fixed-slope ramp only 0.098 wide, so it crushes anti-aliased edges and truncates defocused ones. Skin sits at 0.896 from key green, so its edge saturates to full opacity by 29% coverage.

Q: Does hand masking remove green spill?

No. Spill is green light physically landing on the subject and a mask does not change pixel colour. With a perfect alpha and no despill, 10% contamination still costs 3.17 dE and 30% costs 9.23 — the same for both methods.

Q: How accurate does a hand-drawn mask have to be to beat a keyer?

About 0.2 px of wander on a hard edge with a subject at 0.6-0.8 from key green. At 0.25 px the error is 17.6-18.0 dE against the tuned keyer's 10.7-15.1; at 0.5 px it is 28.6 and at 1 px it is 37.5.

Q: Should I hand mask video frame by frame?

Almost never. The keyer costs nothing per extra frame; a hand mask costs M minutes per frame. Break-even is the tuning time divided by M, so at 20 minutes of tuning and 5 minutes per mask the keyer wins from frame 4. A minute of 25 fps video is 1,500 frames.

Less often than the advice suggests, and for a different reason than people give. We ran both methods on scenes with a known ground-truth alpha and composited them identically. On an ordinary hard edge, a keyer with its gain set properly leaves about 10.7 dE of edge error, while a hand drawn to 0.25 px accuracy leaves about 17.6 — you would need a steadier hand than anyone has. Hand masking wins in four specific places instead. When the subject's colour is within about 0.2 of key green, the keyer cannot make the body opaque at all: at a distance of 0.1 the error is 52 dE against the hand's 17. When the edge is defocused or motion-blurred, the keyer stays at roughly 10 to 11 dE no matter how wide the ramp gets, while a hand mask with a matched feather improves from 18 dE on a 6 px ramp to 4.4 dE on a 54 px one. When the subject is semi-transparent, only authored per-pixel alpha is right — a binary roto mask is the worst answer of the three. And on a single still, the arithmetic almost always favours the hand. What hand masking does not do is fix spill: with a perfect alpha, 10% green contamination still costs 3.2 dE and 30% costs 9.2.

Why a keyer's alpha is a fixed-slope guess

The comparison is only meaningful if you know what the right answer is, so we built scenes where the true coverage of every pixel is known, rendered them over key green, and measured the composited result against ground truth. Both methods were treated identically: the alpha is applied by unpremultiplying against key green and re-compositing over a new background, so a perfect alpha reproduces ground truth exactly and the only thing on trial is the alpha.

That setup exposes the mechanism. A pixel that is partly covered is a linear mix, so its distance from key green is exactly coverage times the subject's colour distance. Call the coverage a and the colour distance D, and the keyer computes a fixed-slope ramp: (a x D - LO) / W, clamped to 0..1, where W is only 0.098 at default settings. The two only agree when D happens to be about 0.098.

No real subject is that close. Taking real colours:

skin #D8A98A — D = 0.896, alpha saturates at 1 once coverage passes 29%
white shirt #F2F2F2 — D = 1.205, saturates by 22% coverage
dark suit #2A2A2A — D = 0.561, still 0 at 25% coverage, then jumps to 1
blue shirt #2E5C8A — D = 0.477, 0 at 25% coverage, 0.79 at 50%

Everything between 0 and 29% coverage is being rounded to either nothing or everything. That is the whole defect: the anti-aliased edge is crushed into a hard one. Whether a hand can beat that depends entirely on how accurately a hand can be drawn.

Hand masking wins: the subject is close to the key colour

Here the keyer does not degrade, it fails outright. The window starts at LO = 0.161, so a subject whose colour distance is below that can never reach an alpha of 1 at default similarity — the body itself turns translucent, and no gain setting fixes a negative number.

Mean edge error on a hard anti-aliased edge, by colour distance D:

D = 0.10 — keyer 52.0 dE (cannot reach opacity), hand at 0.25 px: 17.3
D = 0.15 — keyer 50.5 dE, hand: 17.4
D = 0.20 — keyer 47.1 dE default / 41.8 tuned, hand: 17.5
D = 0.30 — keyer 25.8 / 29.5, hand: 17.7
D = 0.45 — keyer 18.6 / 20.6, hand: 18.0
D = 0.60 — keyer 26.5 / 15.1, hand: 18.0
D = 0.80 — keyer 32.6 / 10.7, hand: 17.6

Read the two columns against each other. The hand's number barely moves — 17.3 to 18.0 across the entire range — because drawing accuracy has nothing to do with what colour you are tracing. The keyer's number swings from 52 to 10.7 depending on the subject. So the crossover is not a property of the tool, it is a property of the shot: below about D = 0.45 hand masking wins, and below 0.2 it wins by a mile.

This is the same boundary our conditions piece found from the other direction — a subject must not contain colours within 0.259 of the key colour at all. If yours does, you are not tuning a keyer, you are drawing a mask.

Hand masking wins: the edge is soft

Defocus and motion blur turn the 1 px anti-aliased edge into a wide ramp, and this is where the keyer's narrow window hurts most. Measured at D = 0.60, with the ramp width reported as the span where true alpha runs from 5% to 95%:

ramp 6 px — keyer default 18.8 dE, tuned 9.8, hand with matched feather 18.2
ramp 14 px — default 15.7, tuned 10.6, hand 11.3
ramp 26 px — default 17.7, tuned 10.9, hand 6.7
ramp 54 px — default 18.3, tuned 11.2, hand 4.4

The keyer's error is flat: it does not care whether the ramp is 6 px or 54 px, because it only has a 0.098-wide window to work with and it truncates every ramp the same way. The hand mask gets better as the ramp widens — a 1 px wander is a large fraction of a 6 px ramp and a negligible fraction of a 54 px one.

The crossover is around a 14 px ramp. Past roughly 26 px the hand wins clearly. If you shoot wide open with a shallow depth of field, or the subject moves during the exposure, hand masking is the better tool — provided you feather to match what you can see. A hand mask left at a 1 px feather stays at about 17-21 dE at every ramp width, which is worse than the tuned keyer everywhere.

Hand masking wins: nothing is opaque, or nothing is moving

Semi-transparency is the one case where both automatic answers are wrong and the hand is right only if it does more work than tracing. With a glass-like subject of transmittance T over a new background:

T = 0.35 (true alpha 0.65) — keyer returns alpha 1.00, error 36.6 dE; hand traced as opaque, 36.6 dE — identical, both fully wrong
T = 0.65 (true alpha 0.35) — keyer 0.50, 17.6 dE; hand traced as opaque, 67.9 dE
T = 0.80 (true alpha 0.20) — keyer 0.00, 17.3 dE; hand traced as opaque, 83.7 dE
any T, hand mask with per-pixel alpha filled in by eye — 0.0 dE

So the honest statement is narrower than 'hand masking is better for glass'. A traced binary mask is the worst option in the table. What wins is authored alpha, which is a different and much slower kind of hand work.

The other clean win is arithmetic, and it is not close. The keyer has a one-off tuning cost K and then costs nothing per frame; a hand mask costs M minutes every frame. The break-even frame count is K / M. If tuning takes you 20 minutes and a mask takes 5, the keyer wins from frame 4; at 10 minutes per mask, from frame 2. For a still, N = 1, and the hand wins whenever M < K — which for a single frame is nearly always true. Video is never a hand-mask job: a minute of 25 fps is 1,500 frames.

Where the keyer wins, and it is most of a normal shot

It is worth saying plainly, because the folk advice goes the other way: on a normal hard edge with a subject well away from green, a tuned keyer beats a hand you can actually draw.

At D = 0.60 the realistically tuned keyer lands at 15.1 dE and at D = 0.80 it reaches 10.7. A hand held to 0.25 px — which is a very careful roto artist on an easy shape — sits at 18.0 and 17.6. To beat the tuned keyer you would need to draw to roughly 0.2 px, and the model says the error scales directly with that wander: 0.5 px gives 28.6, 1 px gives 37.5.

Fine structure goes the same way. On a grating of thin bars at D = 0.60, the mean alpha error was:

1.5 px bars — keyer default 12.7%, tuned 12.0%; hand at 0.25 px: 13.1%, at 1 px: 33.0%
3 px bars — keyer 4.4% / 3.0%; hand 10.2% / 30.1%
12 px bars — keyer 1.1% / 0.8%; hand 8.6% / 28.9%

A hand cannot trace a 3 px mesh. Neither can a camera resolve a 1 px one, incidentally: a grating with a 2 px period lands exactly half-covered on every pixel, so it is flat 50% grey in the source before any keyer touches it. That is Nyquist, not a failure of either method.

What the keyer's best case actually costs

The numbers above labelled 'tuned' are not the keyer's floor. If you let it search every similarity, smoothness and gain combination against the known answer, its error drops to between 0.0 and 1.0 dE in every experiment — near perfect.

You cannot do that on a real shot, because there is no reference to tune against. But the search also shows what those settings do to the background, which is the real objection. Surviving background alpha when the lighting is uneven:

similarity 18 — background 10% darker is left at alpha 0.335; 20% darker is left fully opaque
similarity 22 — 10% darker: 0.131; 20% darker: 0.884
similarity 34 — 10%: 0; 20%: 0.272; 30%: fully opaque
similarity 42 (default) — clean to 20%, then 0.617 at 30%
similarity 60 and above — clean to 30%

So the slider that sharpens your edge is the same slider that lets the background survive. Every improvement you buy on the subject is paid for on the backdrop, and on a real shoot you are paying it blind.

The thing hand masking does not fix

The most common reason people reach for a hand mask is a green fringe, and that is the one thing it cannot deliver. Spill is green light physically landing on the subject. A mask decides which pixels to keep; it does not change their colour.

With a perfect alpha — the best case for either method — composite error from spill alone, with no despill applied:

2% spill — 0.64 dE
5% — 1.60 dE
10% — 3.17 dE
20% — 6.26 dE
30% — 9.23 dE

Those numbers belong to the hand mask and the keyer equally. Drawing a better path does not touch them. If you are hand masking to escape a fringe, you still need despill afterwards — and you will be applying it to a much larger area than a keyer's edge band.

Where these numbers stop

These are synthetic scenes with analytically known alpha, not real footage. That is what makes the error measurable and it is also the limit: no real camera noise, no real fabric, no real hair.

The keyer model is the one we measured on the live tool — normalised RGB distance, a similarity/smoothness window and a gain — and other keyers use different colour spaces and often despill and edge-processing stages. A keyer with real spill suppression will beat these figures on the spill rows specifically.

The hand mask is modelled as a path traced at the visual midpoint with low-frequency wander of 0.25, 0.5 or 1 px and a feather. That is a reasonable description of a careful artist and a bad description of a careless one; it also gives the human no credit for looking at the result and correcting it, which real artists do.

Errors are CIE76 dE measured on the edge band against one replacement background, a dark blue. A background closer to the subject's colour produces smaller numbers for every method, though the ordering did not change in any configuration we ran.

Finally, no despill is applied anywhere. Every figure above is the alpha's error alone.