Why do blocky compression artifacts get masked out along with the background?
Because compression moves pixels around in colour, and a colour keyer reads nothing but colour. We ran a real encode to see how much it costs: libx264 at 1280x720, CRF 12 through 40, keyed with the same settings this site uses. Two findings. The first is that bitrate is almost irrelevant — dropping the bitrate by a factor of 49, from 132.8 KB to 2.7 KB per frame, takes background residue from 0.74% to 1.40%, less than double. Switching from 4:2:0 to 4:4:4 at the same CRF takes it from 0.87% to 0.22%, a 75% cut. Chroma subsampling is the culprit, not bitrate. The second is that the artifacts are not actually blocky any more: across every setting we tested, zero 8x8 blocks came out fully mis-keyed. Modern encoders run a deblocking filter, so what survives is scattered pixels rather than tidy squares — with deblocking off and CRF 40, still only 2.1% of affected blocks came out whole.
What was claimed, and what we did instead of guessing
The claim is that cameras compress, so the blocky artifacts come out with the background — the keyer treats compression noise as if it were part of the screen. The direction is right and the magnitude has never been attached to it, so we measured it.
We built a 1280x720 frame — key green #00B140 background, a denim body, a light-skin head, and fine hair strands — then encoded it with ffmpeg 9.0.1 and libx264, decoded it back, and ran the result through the same keyer this site uses, comparing pixel by pixel against the uncompressed original. Every figure below comes from that run, and the intermediate files are kept so the run can be repeated.
Two versions of the frame, because the difference turned out to matter:
- IDEAL — flat, noise-free background and 2 px hair strands. What a synthetic test image looks like.
- REAL — sensor noise on the background, 1 px hair strands, and 25% corner falloff. What a camera actually hands you.
The artifacts are not blocky any more
This is the part that contradicts the common description, so it goes first.
If compression artifacts were blocky, then mis-keyed pixels should line up with the encoder's 8x8 grid — whole blocks coming out wrong at once. We sorted every residual background pixel into the 8x8 grid and counted how many blocks were affected and how many came out entirely mis-keyed:
- CRF 12 through 34, both footage types — 0 fully mis-keyed blocks
- CRF 40, REAL footage — 3 of 529 affected blocks, 0.6%
- CRF 40 with deblocking disabled — 10 of 484, 2.1%
Essentially none. Turning off the encoder's deblocking filter, which is the one thing that could have restored the block structure, barely moved it.
The reason is that the artifact people remember is from an era of encoders that left block edges alone. H.264 runs a deblocking filter by default and it does what it says: it smooths the block boundaries. What survives into your key is not a grid of squares, it is scattered pixels whose colour moved slightly.
That matters for how you diagnose it. If you are looking for squares, you will not find them, and you will conclude compression is not your problem.
Chroma subsampling is the culprit, not bitrate
Here is the measurement that should change what you spend money on.
Background residue — how much of the screen that should have been cleanly removed survives — on the REAL footage:
- CRF 12 — 0.74% (132.8 KB per frame)
- CRF 23 — 0.75% (5.9 KB)
- CRF 34 — 0.87% (3.4 KB)
- CRF 40 — 1.40% (2.7 KB)
Read the ends together: a 49x reduction in bitrate, and the damage goes up by less than a factor of two. Now the same CRF with chroma resolution restored:
- CRF 34, 4:2:0 — 0.87% residue
- CRF 34, 4:4:4 — 0.22%, a 75% cut
- CRF 23, 4:2:0 — 0.75%
- CRF 23, 4:4:4 — 0.21%, a 72% cut
Shooting 4:2:2 or 4:4:4, or recording to a codec that keeps more chroma, does more for your key than any amount of extra bitrate. And it is worth being precise about why: a colour keyer works on chroma. Halving the chroma resolution halves the resolution of the exact information the keyer reads, and no bitrate setting puts it back.
On the IDEAL footage the same comparison holds, just smaller — 0.24% against 0.16% at CRF 34. The effect is there even when there is nothing wrong with the footage.
Dirty footage pays four times over
The two synthetic frames differ only in noise, hair thickness and falloff, and the gap between them is larger than the gap between the best and worst bitrate.
- IDEAL — residue 0.18% at CRF 12 rising to 0.33% at CRF 40
- REAL — residue 0.74% at CRF 12 rising to 1.40% at CRF 40
Roughly four times the residue, from the same encoder at the same settings. And the damage to the subject diverges much harder. Pixels inside the subject that should have been solid but were keyed away:
- IDEAL — 0.01% at CRF 12, 0.10% at CRF 40
- REAL — 0.01% at CRF 12, 0.31% at CRF 40
That is a 31-fold change on the noisy footage against a 10-fold change on the clean one. Compression pushes some subject pixels toward the key colour, and the keyer takes them. Whether that happens a lot depends mostly on how much noise was in the frame to begin with.
One finding that cuts against expectation: the fine hair did not get eaten. Mean alpha in the hair band was 0.7161 uncompressed and between 0.7666 and 0.8689 after encoding — slightly higher, because the encoder smoothed the noise that had been pulling those pixels toward the key colour. We expected compression to destroy fine strands and in this run it did not.
Noise is what eats your bitrate
One number from the run that is not about keying at all, but explains a lot of frustrating shooting days.
At CRF 12, the IDEAL frame encodes to 5.9 KB and the REAL frame to 132.8 KB — 22 times larger, for the same picture content and the same quality target. At CRF 23 the gap is still 5.9 KB against 3.7 KB in the other direction once the noise is gone.
Compression spends its budget on the thing it cannot predict, and noise is exactly that. A noisier green screen is not merely harder to key, it is more expensive to record, and the bitrate you spend on the noise is bitrate you did not spend on the subject.
Which closes the loop: light the screen properly and you get less noise, less noise encodes cheaper, and cheaper footage keys better. The three problems are one problem.
What to do with this
In the order it pays off.
Shoot more chroma, not more bitrate. 4:2:2 at a moderate bitrate beats 4:2:0 at a high one for keying, and the measurement above says the gap is large. If your camera only offers 4:2:0 internally, an external recorder often does not.
Light the screen until the noise goes away. Sensor noise is what the encoder spends its budget on and what pushes pixels across the key window. More light on the screen is cheaper than more bitrate, and it fixes the residue and the file size at once.
Do not expect raising the bitrate to rescue a bad key. Forty-nine times the compression moved the damage by less than double. If your key is falling apart, the bitrate is almost never the lever.
Stop looking for squares. The deblocking filter means the artifacts are scattered now. If you are inspecting a key and hunting for 8x8 blocks, you are hunting for something the encoder was designed not to leave behind.
Where these numbers stop
This run is specific, and here is exactly what it is specific to.
The footage is synthetic: flat colour regions, Gaussian noise, drawn shapes. A real camera adds rolling shutter, real lens blur, real sensor patterns and demosaicing artifacts, all of which move pixels in ways this test does not include. The relationships should hold; the absolute percentages will not.
It is one frame, not a sequence. Temporal compression — P-frames predicting from earlier frames — drifts in ways a single frame cannot show, and that drift is usually where long-GOP footage gets into trouble.
Everything here is libx264 at default settings apart from CRF, pixel format and the deblocking test. H.265, AV1 and camera hardware encoders will distribute the damage differently. The 4:2:0 versus 4:4:4 finding should transfer, because it is about chroma resolution rather than about a particular codec.
And the noise is synthetic Gaussian noise at one level. Camera noise is signal-dependent and often worse in the shadows, which is where an underlit screen lives.
Related questions
Do compression artifacts really get keyed out with the background?
Yes, in both directions. Some background pixels move far enough from the key colour to survive as residue — 0.74% to 1.40% depending on bitrate — and some subject pixels move toward the key colour and get removed, up to 0.31% of the subject's interior at CRF 40 on noisy footage.
Are the artifacts actually blocky?
Not with a modern encoder. Across every setting we tested, zero 8x8 blocks came out fully mis-keyed. With the deblocking filter disabled at CRF 40, 2.1% of affected blocks came out whole. H.264's deblocking filter smooths block edges, so what reaches your key is scattered pixels.
Should I raise the bitrate to improve my key?
Almost certainly not. Dropping the bitrate 49x took background residue from 0.74% to 1.40%, less than double. Recording 4:4:4 instead of 4:2:0 at the same CRF cut it by 75%. Chroma resolution is the lever, because a colour keyer reads chroma.
Why does my noisy green screen footage encode so large?
Noise is unpredictable and compression spends its budget on what it cannot predict. In our run the noisy version encoded to 132.8 KB per frame at CRF 12 against 5.9 KB for the clean one — 22 times larger for the same content.
Does compression destroy fine hair detail?
It did not in this test. Mean alpha in the hair band was 0.7161 uncompressed and rose slightly after encoding, to between 0.7666 and 0.8689, because the encoder smoothed the noise that had been pulling those pixels toward the key colour.