Can you key against something that isn't flat, like dyed water?
Yes, but only if the surface is flat. A dead-flat, still water screen reads (37,168,52) in our model and keys with the stock preset untouched, leaving 0.00% of it behind, so the dye itself is not the problem. What breaks it is that a water surface is also a 2.8% mirror: every ripple tilts part of that mirror towards a light, and the reflected patch is pale and outside the key window, so at 1 mm of ripple 8.4% of the background survives as opaque holes, at 2 mm 21.0% does, and over half a second of video 95.4% of the background changes its opacity at least once.
What a key actually asks a background for
A colour key does not ask a background to be green, or cloth, or opaque. It asks the background to sit inside one small ball in normalised RGB, so that a single distance threshold can separate it from everything else in the frame. The material is irrelevant; the spread is everything. That is the whole test, and it is why a mirror can in principle be a perfectly good screen.
We built a back-lit tank 12 cm deep with a green food dye in it, a camera at 45 degrees above the surface, and the same keyer this page runs. With the surface dead flat and still, the water reads (37,168,52). Its distance to the stock preset #00B140 is 0.1566, and the keyer starts softening at 0.161, so it keys with the preset untouched: 0.00% of the screen survives. The margin is 0.004 of distance. Flatness, not colour, is what bought that.
The surface is a 2.8% mirror sitting on top of the dye
Water reflects 2.8% of what is above it at 45 degrees of incidence. That number sounds too small to matter, and it would be, if the dye were not doing the rest of the work. At 12 cm the dye passes red at 0.030 in linear light. A softbox two stops brighter than the panel behind the tank, reflected at 2.8%, contributes 0.112. The reflection is nearly four times what the dye lets through in that one channel.
So the facets that happen to point at the light read (89,181,96) while the rest of the surface reads (38,168,53): pale, desaturated, 0.275 away from the key colour and outside the 0.259 outer edge of the window. Those pixels come out at alpha 0.902. They are opaque holes punched in a background that otherwise disappeared completely, and they are the wrong colour to be mistaken for anything else in the frame.
How still the water has to be
Ripple amplitude, 20 cm wavelength, camera at 45 degrees, default Tolerance 42 and Smoothness 14: 0.5 mm leaves 0.00% of the background, 0.75 mm leaves 1.61%, 1 mm leaves 8.36%, 1.5 mm leaves 16.88% and 2 mm leaves 20.96%. In slope terms those are 1.08, 1.62, 2.16, 3.23 and 4.31 degrees. The comfortable margin is about half a millimetre, which is not a state any open container of water holds for long.
Past 2 mm the instantaneous number falls back: 4 mm leaves 17.53% and 8 mm leaves 6.31%. The reason is that steep facets sweep through the source instead of sitting in it, so glint coverage goes 22.2% at 2 mm and 6.9% at 8 mm. That is not a recovery. The damage becomes intermittent rather than steady, which is worse for a shot you have to cut.
It is the surface, not the dye
We separated the two effects on 8 mm waves. Keep the wave slopes and freeze the depth and the residual is 6.26%, essentially all of the 6.31% the full model gives. Keep the depth changes and freeze the surface normal and it is 0.00%, with a 95th percentile distance of 0.057. Remove the reflection entirely and it is 0.00% again. How much dye the light passes through on its way out is not what breaks the key.
The one exception is a bottom that is not flat. Still water over a slope running from 6 cm to 20 cm across the frame leaves 27.99% of the background at the default, and needs Tolerance 77 at the 95th percentile, 89 at the worst pixel. Depth only becomes expensive when it varies by tens of centimetres, not when waves vary it by millimetres.
One frame is never the whole story
Twelve frames at 1/25 s with 4 mm waves: the residual is steady at 17.35% to 18.30%, which looks like a stable problem you could dial out. It is not. Across those twelve frames 95.44% of the background pixels change their alpha by more than 0.5 at some point, and the mean frame-to-frame change is 0.262. Glint coverage stays near 20% the whole time. It is a different 20% every frame.
A single photograph can be rescued, because one frame has one arrangement of holes and you can tune to it. A clip cannot, because any Tolerance you choose is negotiating with a surface that keeps re-deciding which of its pixels fall inside the window.
What the controls can and cannot do
Tolerance, 8 mm water: 30 leaves 7.52% of the background, 42 leaves 6.31%, 60 leaves 2.14% and 80 leaves 0.00%. The screen is finally clean at 80, but the anti-aliased rim of the subject has dropped from alpha 0.441 to 0.158 and the shirt has started to go at 0.965. At 100 the shirt is at 0.000 and the hair at 0.485. You are buying a clean background with the edge.
Spill suppression does nothing to the holes, and the reason is in the order of operations: the mask is computed from the distance first, and spill runs afterwards, so alpha is byte-identical at 0, 35, 70 and 100. All it does is change the colour of the surviving patch, from (89,181,96) to (89,96,96) at the maximum, which is a grey-green hole instead of a pale-green one.
What actually gets rid of it
Kill the reflection. Water's Brewster angle is 53.1 degrees from the surface normal, which means a camera 36.9 degrees above the surface; a polariser there takes the 8 mm case from 7.92% to 0.10%, once you open a stop to pay for the filter. It is not a general fix: at 45 degrees elevation it only halves the damage, 6.31% to 2.12%, and at 60 degrees elevation it is slightly worse, 4.40% to 5.73%.
Two other levers measured clean. Shoot steeper: at 20 degrees elevation above the surface the residual is 25.76% and needs Tolerance 137, which is past the end of the control, while 70 degrees leaves 3.52% and needs 44. Dilute the dye: 0.6 times the optical depth drops the 8 mm residual to 0.83%. Or skip it. A flat surface keys at 0.00%, and cloth is flat before you do anything to it.
Related questions
Is the problem that dyed water is not the right green?
No. Flat, still water reads (37,168,52) in our model, which is 0.1566 from the preset #00B140 against a softening threshold of 0.161, and it keys leaving 0.00% of the screen behind. If the surface is flat you can use the preset as it ships. The failures all come from variation across the surface, not from the hue of the dye.
Does a polariser fix it at any camera angle?
Only near Brewster's angle. With the camera 36.9 degrees above the surface, so 53.1 degrees of incidence, the 8 mm case goes from 7.92% of the background surviving to 0.10%. At 45 degrees elevation it only halves, 6.31% to 2.12%, and at 60 degrees elevation it is slightly worse, 4.40% to 5.73%, because a polariser also passes less of the light you want.
Can I just raise Tolerance until the holes close?
On 8 mm waves, 80 clears the background completely, and 60 already brings it to 2.14%. The cost is the subject: the anti-aliased rim falls from alpha 0.441 to 0.158, the shirt starts dropping at 0.965, and at 100 the shirt is at 0.000 with the hair at 0.485. You are trading the edge of your subject for the holes in the background.
Why doesn't spill suppression remove the shiny patches?
Because the mask is computed before spill runs. The keyer measures the distance to the key colour, decides the alpha, and only then touches the green channel, so alpha is identical at spill 0, 35, 70 and 100. What changes is the colour of the surviving patch, (89,181,96) becoming (89,96,96) at the maximum, which is still fully opaque.