Oil Painting Color Calibration

A simple, effective method and algorithm for low-cost DIY color calibration of oil painting photos. Uses readily made neutrals of pure black, any number of grays, and pure white to automatically and efficiently set white balance and levels.

Working Principle

Color arises from the interaction of light, surface reflectance, and an observer - see here for more on how this works. The amount and quality of light illuminating an object affects its perceived color. This is easy to overlook, however, because the human brain is extremely proficient at mentally normalizing for lighting so that perceived colors are consistent across different types of lighting. Overcoming this remarkable ability is a central challenge of becoming a good painter! For example, you probably see a banana as simply "yellow", regardless of the lighting conditions that can actually cast it as beige, brown, or orange.

Bananas - yellow, or not?

Cameras, on the hand, have no such ability. They record only the literal information of a scene, and in doing so make guesses about the nature and amount of light present. These are often inaccurate, leading to white balance and levels adjustments being very common operations in photo retouching.

White balance describes the quality of the light in the scene. Cameras often have predefined white balance settings for outdoors, indoors, shade, etc. If the setting is not specified, the camera is forced to guess the "color cast" that the lighting imparts on the scene, and adjust for it accordingly. A neutral light, one without a color cast, imparts no color of its own on the scene. Non-neutral light can have any color cast, but in everyday settings, the most common are yellowish/warm as with incandescent bulbs, and bluish/cool as with fluorescent bulbs.

Opinions differ on best methods for displaying paintings, and as a matter of taste, there is no right answer. I believe that a neutral light is preferable, for two key reasons:

1. Neutral light imparts no color cast, resulting in maximum color differentiability.
2. The most natural, familiar, and abundant of light sources, daylight, is approximately neutral.

It follows that a neutral light incident on a neutral paint produces a neutral color. The algorithm outlined here enforces this constraint for any number of known neutrals, thereby correcting unwanted color cast. This is commonly called a white balance correction.

Since black and white are neutral, it's possible to not only use them for white balance, but also to use them for levels - in other words, defining which colors correspond to black and white, by remapping. By capturing black, white, and several neutrals in a single operation, you can elegantly set calibrated white balance and levels at the same time.

Neutrals Colors

A neutral is any color with zero saturation: black, white, and all shades of gray in between. 

In Digital Color

Neutrals satisfy the condition R=G=B in RGB color space. Since the space is 8-bit, there are 2^8 = 256 true neutrals possible; R=G=B=0 is pure black, and R=G=B=255 is pure white. A much larger number of near-neutrals also exist.

In Oil Paint

Neutrals can be achieved similarly by mixing between pure white and pure black. Unlike in digital color, an infinite number of neutral grays are possible between pure black and white.

Neutral Calibration Swatches

The key to this method is to have known neutrals in the same frame, under the same lighting, as the painting being photographed. Conventionally, photographers purchase pre-made certified-neutral swatches, but out of pandemic-related necessity I opted to make my own. However, I later realized that there's actually substantial benefit in a DIY approach, because the calibration is then tailored to your specific palette; plus, mixing a value range is good general practice anyway, and produces a helpful chart you can refer back to later.

In theory, this method can work with as few as two neutrals (black and white), but having a number of intermediate neutrals between ensures that the white balance is correct over the full value range, not just at the limits.

Using what I had on hand, I arbitrarily mixed six neutrals, 1" each on a 6" panel, starting with pure Reeves ivory black and brightening progressively up to pure Winsor & Newton titanium white. Ideally the neutrals are equally spaced in terms of their value (lightness), but this is not strictly necessary. Similarly, the paints and mixtures thereof should be truly neutral, but exact neutrality in the scientific sense is not necessary. Working by eye is sufficient for both aspects.

Mixing six neutrals on my palette

First attempt - too little spacing on dark side

Second attempt - better but still room for improvement

Photographing the Painting

The next step is photographing the work with the neutral calibration swatches alongside it, at the same angle and under the same lighting. This topic is covered in great detail in other sources, so I won't go into much depth here. In short, the keys to a good photo include bright, diffuse light, the lens axis being centered and perpendicular on the work to minimize perspective distortion, the ISO and shutter speed set as low as possible, and no saturation in the area of interest. My setup - shooting in my sunroom in indirect mid-afternoon light - is hardly ideal, but it works decently, and well enough to demonstrate this concept.

Two-tripod sunroom photography setup

Algorithm

The photo is then passed into the algorithm.

Collect Neutrals

The algorithm requests that the user left-click any number of neutrals in their provided image, necessarily including black and white. After each selection, the algorithm sparsely samples a 1,000-pixel circular neighborhood around the selection, takes the midtone color as the mode from each channel, and overlays it on the image for qualitative verification. When complete, the user right-clicks to proceed.

All neutrals identified, sampled, and labeled

The solid color inside the circle is the RGB mode of the pixels inside its boundary

Note that the color extracted by taking the mathematical mode represents the midtone; shadows and highlights are ignored. Taking an average would mix these values together, which results in non-physical and worse results.

Show Neutrals on Histogram

The algorithm shows a per-channel histogram for the original image, and for each neutral, shows where those R, G, B values occur on their respective histogram. This is helpful to get a qualitative feel for the nature and magnitude of the white balance correction needed. Here, we can see that there is a reddish cast on the image, since red RGB values always exceed the neutral grayscale value.

Extract Painting Frame

The algorithm requests that the user left-click the interior corners of the painting, so that just the painting can be exported. The user can right-click to skip this step. The selection is shown as four dashed lines.

Full set of inputs provided by the user: six neutrals and two corners - eight clicks total

Generate White Balance Correction Curves

For each provided neutral, the algorithm calculates the corresponding grayscale value using the standard luminosity weighting of 21% R, 72% green, 7% B. It then calculates the required gains for those RGB values to match the grayscale value. For example, RGB [123 130 145] has a grayscale value of dot([0.21 0.72 0.07], [123 130 145]) = 130, so the gains at R=123, G=130, and B=145 are [1.06 1.00 0.90], respectively. Repeating this for each neutral, the result is a set of white balance correction curves. Outside the extreme values provided, the closest adjacent value is held as an assumed constant, so that the values are defined over the full domain of 0 to 255.

Apply White Balance Correction Curves

Using linear interpolation, the algorithm multiplies the entire image by the correction curves. For example, an input G value of 120 corresponds to a gain of 1.025, so the output G value is round(120 * 1.025) = 123, and so on for all pixels and color channels. For an ~18 megapixel image, the calculation time is a few seconds on my 2011 laptop.

Verify White Balance

The white balance-corrected image is then displayed. The user can independently verify that the selected neutrals have been balanced correctly.

Image after white balance correction

For example, neutrality can be quickly verified in GIMP, by checking that R=G=B within a reasonable tolerance of about ± 1%.

GIMP Color Picker showing nearly zero neutrality error between R, G, and B

The neighborhoods of the input neutral indices are then re-sampled, and overlaid on the updated image histogram. The line segments connecting the R, G, and B values are now nearly vertical and coincident with the corresponding target grayscale values, indicating correct balance. Note that due to discretization and rounding, zeros begin appearing in the histogram; for visual clarity, I set these zeros to NaN so that they appear as gaps rather than as spikes or divots.

Crop and Adjust Levels

Last, the image is cropped to the input painting frame, and the levels are adjusted. Here's it's necessary to make a judgement call on how "black" your black paint should appear to be, and so on for the white. I use 2.5% and 97.5% as my RGB value limits, but tastes will vary.

The darkest and lightest neutrals are taken as representing black and white, and their 1,000-pixel neighborhoods are again sampled, but this time for the minimum and maximum in each channel, rather than the mode. For this to work correctly, the input image must not saturate the 0% or 100% value limits.

The darkest color within the black neutral is mapped to 2.5%, and the lightest shade within the white neutral is mapped to 97.5%. All intermediate values between are mapped linearly. I use MATLAB's imadjust function for convenience, but it could also be done in HSV or LAB color space using the V or L channels, respectively.

The histogram is shown one last time for the cropped and level-adjusted image.

Note that due to the levels adjustment, the gaps open significantly. This is not desirable and leads to graininess, but in my view is a much better problem than inaccurate colors or levels. If the correction magnitude was less, these gaps would also be smaller. This is an area of future work for me. I expect that incorporating a remote shutter trigger, reducing the ISO and shutter speed, and shooting in RAW will improve the initial image quality and reduce the correction magnitude, improving the overall result.

Demo Gallery

Below are some input/output examples shot under a variety of lighting conditions. These paintings are all unvarnished.

Prioritizing Color Over Value

A discussion on value-priority vs. color-priority in color theory and painting, and a quantitative method for shifting between them in digital photos.

Motivation

As a painter, I work from both life and from photos. Both approaches have key advantages and drawbacks. Painting from life is the best method for accurately capturing a scene's colors, but is somewhat limited to static scenes. Painting from photos can capture dynamic scenes, but has limited color accuracy. As an engineer, I'm interested in unifying the approaches in a way that preserves the best aspects of each.

Let me start by defining my terminology:

• Painting photo: a photo of a painting that accurately matches the color of the physical painting
• Camera photo: a photo straight from a camera without modification
• Reference photo: a camera photo, augmented digitally for better use as a painting reference

Because all these photo types reside within the same digital color gamut, as I'll show, nothing prevents a reference photo from having the same desirable characteristics as a painting photo. However, camera photos make for poor references photos due to their inaccurate and uninspiring portrayal of color. Therefore, I'm interested in how to turn camera photos into reference photos using image processing. You can think of this approach as an alternate path that parallels the traditional/academic process, both leading to the same end result.

I've previously written about two image processing techniques toward this goal - Histogram Diffusion for color differentiability, and 3D Focal Point Blur for focal realism. In this article, I'll discuss another approach, inspired by artistic color theory.

Background

In his 2009 book Landscape Painting, Mitchell Albala describes the concept of "color identity":

"Extremes of value profoundly affect a color's ability to be read as color...A color's chromatic identity is most visible when its value is neither too dark nor too light but is in the middle-value range."

Albala illustrates this concept of "color identity" with a chart, reproduced below, showing various colors transitioning from very light (near-white) values to very dark (near-black) values. Colors that are very light or very dark have nearly no perceptible "color". In other words, they read almost as pure values, or grayscale. Another way to think of this is how "nameable" a color is. This broad definition is what Albala means by "color identity".

To clarify the terminology:

• Hue: color, independent of lightness and intensity
• Saturation: how vibrant, intense, or colorful a color is
• Value: how light or dark a color is

Albala illustrates this concept by comparing two schools of landscape painting:

1: 17th century Dutch "Golden Age" landscapes are characterized by "value-priority", maximizing value contrast (sometimes called chiaroscuro) at the cost of low color identity.

Meindert Hobbema, Entrance to a Village, c. 1665

I've copied the Levels histogram from GIMP (an open source Photoshop equivalent), a visualization for how the photo utilizes the value gamut. This painting shows a strong bimodal (two-peak) distribution which creates a value-priority effect.

2: Impressionist landscapes are characterized by "color-priority", maximizing color identity at the cost of low value contrast.

Claude Monet, Antibes Seen from Las Salis, 1888

The Levels histogram shows a clear difference: just one peak, so colors are much closer in terms of value, differentiated primarily by color, creating a color-priority effect.

To simplify, you can think of these two schools as favoring differing regions of the color chart to get their ideas across:

Which is more realistic? More aesthetically pleasing? More correct? These are fundamentally subjective questions, best left to personal taste. As Albala notes, "[both] successfully depict natural light", "[although] the Impressionist approach to color is perhaps more in tune with contemporary color sensibilities".

My take is that neither image is particularly realistic, but I personally find the Impressionist approach much more painterly and compelling, and closer to "correct".

Albala has since expanded on this topic at his blog with a couple excellent articles:

• How Value is Used to Heighten the Effects of Color in Landscape Painting
• The Affect of Value on Color Identity in Impressionist Painting

Quantifying Color Identity

Digital color provides a quantitative framework to examine and enact Albala's view of color identity. But it first requires some careful consideration of color space nuance.


RGB Color Space

Let's start at square one with RGB color, the standard for storing and displaying digital images. RGB color has three channels corresponding to red, green, and blue light, and within a channel, a pixel can take any of 256 values, 0-255. Therefore, there are 256^3 or ~16,700,000 colors available.

RGB's core purpose is telling monitors how to display color, and it's very effective for this. However, its utility in addressing color identity is very limited, because the intuitive properties of hue, saturation, value - the way we think about color - are all interdependent with respect to the axes of R, G, B.

HSV Color Space

Invented in the 1970s, the HSV (hue, saturation, value) color space was designed to provide more intuitive framework for working with digital color. Whereas RGB is a cube, HSV is a cylinder; hue describes the angle, saturation the radius, and value the height. It's computationally easy to transform between RGB and HSV, which was initially a significant advantage, but largely moot in the current era. HSV's key shortcoming is that saturation and value are not actually independent - just close to it. This can be observed from a constant-hue chart, which corresponds to half of a cross section of the HSV cylinder. Here is H=130°, arbitrarily:

Columns theoretically have constant saturation, and rows theoretically have constant value. To test the latter, we can convert the chart to grayscale, either by explicitly switching the image mode or desaturating it completely. Both provide the same result:

Examine the grayscale chart and you'll see upon close inspection a couple key shortcomings:

1. As you move horizontally within a row, value decreases (darkens) slightly. The actual disparity between min and max across rows is about 5%.
2. As you move vertically within a column, the rate at which value changes is not uniform. Note for example the steep gradient from 10-15% value.

While very subtle in this example, these issues significantly impede HSV's ability to address color identity.

Lab Color Space

The Lab color space, invented in 1976, solves these problems with full independence of hue, saturation, and value, as well as perceptual uniformity. Like RGB, Lab is a 3D orthogonal color space:

• L corresponds to a color's value, or lightness/darkness
• a corresponds to a theoretical green-magenta/red axis
• b corresponds to a theoretical blue-yellow axis

Whereas RGB is discretized 0-255, Lab is continuous and unlimited. Therefore, Lab can theoretically describe "imaginary" colors that cannot be displayed in RGB, or describe infinite colors between RGB discretizations. It is perceptually uniform, unlike RGB and HSV, meaning that Euclidean (straight-line) distance corresponds linearly to perceived color distance/difference in the human eye, which is a significantly non-linear sensor.

Every color in RGB color space can be converted to Lab, and vice versa. Because the equations that perform this conversion are non-linear, the neat RGB cube becomes non-trivially deformed in Lab color space:

In this visualization, a uniform, dense point cloud is created inside the RGB cube, then converted to Lab, and its boundary is approximated using a concave hull algorithm. The mesh for each boundary is overlaid on the volume. The non-linearity of human vision causes the initially-uniform point cloud to be significantly non-uniform in Lab color space, which is reflected in its boundary mesh.

These visualizations are very colorful - in fact, they could not be any more colorful, because their outer surfaces contain their most highly saturated colors. However, keep in mind that this visualization shows only the outer surfaces, which contain a small fraction of the complete color gamut. To view these less-colorful colors, you have to look inside these volumes. This can be achieved with a cross-section:

Here, I section the RGB cube along its R axis arbitrarily, and the Lab volume along its L axis. On the Lab gamut, I overlay the color space's "neutral axis", which occurs where a=0 and b=0. In other words, all colors on this line have zero hue because they are pure shades of gray. Note that the neutral axis intersects pure black and pure white. Hue and saturation are defined simply as:

• Hue is a color's angle from the neutral axis, atan2d(b,a)
• Saturation is a color's distance from the neutral axis, sqrt(a^2+b^2)

With this visualization, you can show that Lab has true value independence by converting any frame to grayscale:

Note that in Lab, there is only a single, constant shade of gray within the cross-section boundary - not so with RGB which has no such constraint.

This framework provides a basis for corroborating Albala's notion of color identity. Since the colors with the strongest color identity occur on the outer surface of the gamut volume, we can deduce that saturation is a proxy for color identity. Further, we can take the section area as a proxy for color identity potential at that value:

This is a general result for all hues; as you can see from the cross-section, different hues achieve maximum color identity at different L values. This can be compared with the color swatch chart from earlier:

Here I've superimposed the earlier color swatch chart with the color identity potential plot. The swatches don't necessarily occur at the correct L values, so the result is merely suggestive, but the qualitative fit between the two is clearly strong.

Why Photos Tend Toward Value-Priority and Drab Colors

Modern digital cameras in "automatic" mode are so adept in adjusting their settings under the hood that it's easy to think of their images as objective truth. However, much of this process is actually based on heuristics that are designed to provide decent results over a wide range of conditions. This is ideal for the typical consumer who simply wants to point and shoot with minimal fuss.

However, for those who paint from photos, the automatic settings often provide seriously sub-optimal results - namely, they tend to minimize color identity. This is both deliberate and reasonable: creating a value-prioritized image is algorithmically easy and will usually look decent, whereas creating a color-prioritized image is not straightforward and will be prone to pitfalls. Allow me to illustrate:

Suppose you are the camera's setting adjustment algorithm. You look out at the scene and see this distribution of light:

Note that a scale and units are not provided; a camera is not measurement tool. You simply see that there are some darker regions, some midtones, and some lighter regions. So that the image can be saved and displayed, your job is to fit this distribution into a fixed bounding box:

How do you do this? Approaches vary by manufacturer, but one certainty is that you don't want to saturate any part of the spectrum. This would mean "pushing" any part of the distribution outside of the bounding box, resulting in pure white or pure black. For example:

No good - the lower end is saturated.

No good - the upper end is saturated.

Maybe okay, but there's so much leftover space. What to do with it? You can shift the distribution left or right by as much as 25% without saturating. So which of these 50 different solutions is best? There's simply isn't a good, general-purpose, algorithmic way to make this decision. So what generally happens is that the algorithm simply scales and stretches the distribution so that it's just shy of saturation on both ends.

This requires just a couple hard-coded parameters to work correctly, and is simple, open-loop (requiring no feedback), and almost always produces decent results. Wide distributions are safer and easier to work with, because by design they fit neatly into the bounding box.

This distribution comes from a photo of mine from Yellowstone:

Does the distribution or photo look familiar? In the sense of having a wide, bimodal (two-peak) distribution with few midtones, it's actually very similar to the drab Dutch landscape from earlier. This is no coincidence - the pictures were, in effect, recorded with a very similar process.

So, digital photos have a tendency to be drab for the same reason that 17th century Dutch landscapes did. And by drab, I mean lacking in color identity. What can be done about this?

How to Shift From Value-Priority to Color-Priority

The Lab color space works well for shifting photos from value-priority to color-priority, or vice versa, for the reasons discussed earlier. To summarize the process in geometric terms:

1. Store a photo as an RGB point cloud
2. Convert the RGB point cloud to Lab color space
3. Manipulate the Lab point cloud using linear transforms
4. Convert the Lab point cloud back to RGB for display
5. Repeat from Step #3 until satisfied with result

The linear transforms in Step #3 are simple. For each of the three properties of hue, saturation, and value, the linear transform is defined by an offset and scale which is applied globally to the entire point cloud. Continuing with the geometric analogy, these transforms as a whole enable you to move, stretch, compress, twist, and expand the point cloud in an intuitive and freeform way. It's easy to push parts of the point cloud outside the limited RGB gamut, so the Lab-to-RGB conversion process accounts for this by saturating, or limiting its input before conversion.

Because these adjustments are best done by eye, they require a good deal of tweaking to get right. To facilitate this, I wrote a GUI to quickly get the settings right before exporting. Here's an overview:

A control panel lets you define the scale and offset for lightness (i.e. value), hue, and saturation on sliding scales. There's also a slider to control the quality of the preview, and buttons to restore defaults, compare before/after images, and to export the image at full resolution.

A histogram panel shows the result of the scaling operations, before and after. This is most helpful for shifting value to where the desired level of color identity can be achieved. The functionality for scaling hue is fun to play with, because it can produce wild and impossible color palettes, but for my purposes I leave it as-is.

Lastly, there's a preview panel which shows a downsampled copy of the image with the transforms applied. An alphanumeric string across the top conveys the transform parameters, which are also appended to the filename on export.

There's some skill and taste involved in manipulating photos with this technique. In general, I try to keep the peaks of the value histogram between 20 and 80%, outside of which there is minimal color identity. With the saturation scaling, the offset helps "push" neutrals outward for greater color identity, while the scale helps accentuate the already-vibrant regions of an image.

Demo Gallery

Here are some photos that I've attempted to shift from value-priority to color-priority using this technique, all taken with my 2003 Canon Rebel without any prior modifications. In each photo, the left half is the original and the right half is the result.

And to circle back to the original concept, here are a couple examples of shifting a painting's color gamut to align with a different school of painting:

Recasting a Dutch landscape as an Impressionist painting:

Meindert Hobbema, The Avenue at Middelharnis, 1689

Recasting an Impressionist painting as a Dutch landscape:

Claude Monet, Coquelicots [Poppy Field], 1873

Source Code

Freely available on my GitHub.

Acknowledgements

• Mitchell Albala provided corrections and suggestions for the Motivation and Background sections.