Thursday, April 16, 2020

Oil Painting Color Calibration

A simple, effective method and algorithm for low-cost DIY color calibration of oil painting photos. Uses readily made neutrals of pure black, any number of grays, and pure white to automatically and efficiently set white balance and levels.

Working Principle

Color arises from the interaction of light, surface reflectance, and an observer - see here for more on how this
works. The amount and quality of light illuminating an object affects its perceived color. This is easy to overlook, however, because the human brain is extremely proficient at mentally normalizing for lighting so that perceived colors are consistent across different types of lighting. Overcoming this remarkable ability is a central challenge of becoming a good painter! For example, you probably see a banana as simply "yellow", regardless of the lighting conditions that can actually cast it as beige, brown, or orange.

Bananas - yellow, or not?

Cameras, on the hand, have no such ability. They record only the literal information of a scene, and in doing so make guesses about the nature and amount of light present. These are often inaccurate, leading to white balance and levels adjustments being very common operations in photo retouching.

White balance describes the quality of the light in the scene. Cameras often have predefined white balance settings for outdoors, indoors, shade, etc. If the setting is not specified, the camera is forced to guess the "color cast" that the lighting imparts on the scene, and adjust for it accordingly. A neutral light, one without a color cast, imparts no color of its own on the scene. Non-neutral light can have any color cast, but in everyday settings, the most common are yellowish/warm as with incandescent bulbs, and bluish/cool as with fluorescent bulbs.

Opinions differ on best methods for displaying paintings, and as a matter of taste, there is no right answer. I believe that a neutral light is preferable, for two key reasons:
  1. Neutral light imparts no color cast, resulting in maximum color differentiability.
  2. The most natural, familiar, and abundant of light sources, daylight, is approximately neutral.

It follows that a neutral light incident on a neutral paint produces a neutral color. The algorithm outlined here enforces this constraint for any number of known neutrals, thereby correcting unwanted color cast. This is commonly called a white balance correction.

Since black and white are neutral, it's possible to not only use them for white balance, but also to use them for levels - in other words, defining which colors correspond to black and white, by remapping. By capturing black, white, and several neutrals in a single operation, you can elegantly set calibrated white balance and levels at the same time.

Neutrals Colors

A neutral is any color with zero saturation: black, white, and all shades of gray in between. 

In Digital Color

Neutrals satisfy the condition R=G=B in RGB color space. Since the space is 8-bit, there are 2^8 = 256 true neutrals possible; R=G=B=0 is pure black, and R=G=B=255 is pure white. A much larger number of near-neutrals also exist.

In Oil Paint

Neutrals can be achieved similarly by mixing between pure white and pure black. Unlike in digital color, an infinite number of neutral grays are possible between pure black and white.

Neutral Calibration Swatches

The key to this method is to have known neutrals in the same frame, under the same lighting, as the painting being photographed. Conventionally, photographers purchase pre-made certified-neutral swatches, but out of pandemic-related necessity I opted to make my own. However, I later realized that there's actually substantial benefit in a DIY approach, because the calibration is then tailored to your specific palette; plus, mixing a value range is good general practice anyway, and produces a helpful chart you can refer back to later.

In theory, this method can work with as few as two neutrals (black and white), but having a number of intermediate neutrals between ensures that the white balance is correct over the full value range, not just at the limits.

Using what I had on hand, I arbitrarily mixed six neutrals, 1" each on a 6" panel, starting with pure Reeves ivory black and brightening progressively up to pure Winsor & Newton titanium white. Ideally the neutrals are equally spaced in terms of their value (lightness), but this is not strictly necessary. Similarly, the paints and mixtures thereof should be truly neutral, but exact neutrality in the scientific sense is not necessary. Working by eye is sufficient for both aspects.

Mixing six neutrals on my palette

First attempt - too little spacing on dark side

Second attempt - better but still room for improvement

Photographing the Painting

The next step is photographing the work with the neutral calibration swatches alongside it, at the same angle and under the same lighting. This topic is covered in great detail in other sources, so I won't go into much depth here. In short, the keys to a good photo include bright, diffuse light, the lens axis being centered and perpendicular on the work to minimize perspective distortion, the ISO and shutter speed set as low as possible, and no saturation in the area of interest. My setup - shooting in my sunroom in indirect mid-afternoon light - is hardly ideal, but it works decently, and well enough to demonstrate this concept.

Two-tripod sunroom photography setup


The photo is then passed into the algorithm.

Collect Neutrals

The algorithm requests that the user left-click any number of neutrals in their provided image, necessarily including black and white. After each selection, the algorithm sparsely samples a 1,000-pixel circular neighborhood around the selection, takes the midtone color as the mode from each channel, and overlays it on the image for qualitative verification. When complete, the user right-clicks to proceed.

All neutrals identified, sampled, and labeled

The solid color inside the circle is the RGB mode of the pixels inside its boundary

Note that the color extracted by taking the mathematical mode represents the midtone; shadows and highlights are ignored. Taking an average would mix these values together, which results in non-physical and worse results.

Show Neutrals on Histogram

The algorithm shows a per-channel histogram for the original image,
and for each neutral, shows where those R, G, B values occur on their respective histogram. This is helpful to get a qualitative feel for the nature and magnitude of the white balance correction needed. Here, we can see that there is a reddish cast on the image, since red RGB values always exceed the neutral grayscale value.

Extract Painting Frame

The algorithm requests that the user left-click the interior corners of the painting, so that just the painting can be exported. The user can right-click to skip this step. The selection is shown as four dashed lines.

Full set of inputs provided by the user: six neutrals and two corners - eight clicks total

Generate White Balance Correction Curves

For each provided neutral, the algorithm calculates the corresponding grayscale value using the standard luminosity weighting of 21% R, 72% green, 7% B. It then calculates the required gains for those RGB values to match the grayscale value. For example, RGB [123 130 145] has a grayscale value of dot([0.21 0.72 0.07], [123 130 145]) = 130, so the gains at R=123, G=130, and B=145 are [1.06 1.00 0.90], respectively. Repeating this for each neutral, the result is a set of white balance correction curves. Outside the extreme values provided, the closest adjacent value is held as an assumed constant, so that the values are defined over the full domain of 0 to 255.

Apply White Balance Correction Curves

Using linear interpolation, the algorithm multiplies the entire image by the correction curves. For example, an input G value of 120 corresponds to a gain of 1.025, so the output G value is round(120 * 1.025) = 123, and so on for all pixels and color channels. For an ~18 megapixel image, the calculation time is a few seconds on my 2011 laptop.

Verify White Balance

The white balance-corrected image is then displayed. The user can independently verify that the selected neutrals have been balanced correctly.

Image after white balance correction

For example, neutrality can be quickly verified in GIMP, by checking that R=G=B within a reasonable tolerance of about ± 1%.

GIMP Color Picker showing nearly zero neutrality error between R, G, and B

The neighborhoods of the input neutral indices are then re-sampled, and overlaid on the updated image histogram. The line segments connecting the R, G, and B values are now nearly vertical and coincident with the corresponding target grayscale values, indicating correct balance. Note that due to discretization and rounding, zeros begin appearing in the histogram; for visual clarity, I set these zeros to NaN so that they appear as gaps rather than as spikes or divots.

Crop and Adjust Levels

Last, the image is cropped to the input painting frame, and the levels are adjusted. Here's it's necessary to make a judgement call on how "black" your black paint should appear to be, and so on for the white. I use 2.5% and 97.5% as my RGB value limits, but tastes will vary.

The darkest and lightest neutrals are taken as representing black and white, and their 1,000-pixel neighborhoods are again sampled, but this time for the minimum and maximum in each channel, rather than the mode. For this to work correctly, the input image must not saturate the 0% or 100% value limits.

The darkest color within the black neutral is mapped to 2.5%, and the lightest shade within the white neutral is mapped to 97.5%. All intermediate values between are mapped linearly. I use MATLAB's imadjust function for convenience, but it could also be done in HSV or LAB color space using the V or L channels, respectively.

The histogram is shown one last time for the cropped and level-adjusted image.

Note that due to the levels adjustment, the gaps open significantly. This is not desirable and leads to graininess, but in my view is a much better problem than inaccurate colors or levels. If the correction magnitude was less, these gaps would also be smaller. This is an area of future work for me. I expect that incorporating a remote shutter trigger, reducing the ISO and shutter speed, and shooting in RAW will improve the initial image quality and reduce the correction magnitude, improving the overall result.

Demo Gallery

Below are some input/output examples shot under a variety of lighting conditions. These paintings are all unvarnished.

No comments:

Post a Comment