Sunday, March 8, 2020

Prioritizing Color Over Value

A discussion on value-priority vs. color-priority in color theory and painting, and a quantitative method for shifting between them in digital photos.



Motivation

As a painter, I work from both life and from photos. Both approaches have key advantages and drawbacks. Painting from life is the best method for accurately capturing a scene's colors, but is somewhat limited to static scenes. Painting from photos can capture dynamic scenes, but has limited color accuracy. As an engineer, I'm interested in unifying the approaches in a way that preserves the best aspects of each.

Let me start by defining my terminology:


  • Painting photo: a photo of a painting that accurately matches the color of the physical painting
  • Camera photo: a photo straight from a camera without modification
  • Reference photo: a camera photo, augmented digitally for better use as a painting reference

Because all these photo types reside within the same digital color gamut, as I'll show, nothing prevents a reference photo from having the same desirable characteristics as a painting photo. However, camera photos make for poor references photos due to their inaccurate and uninspiring portrayal of color. Therefore, I'm interested in how to turn camera photos into reference photos using image processing. You can think of this approach as an alternate path that parallels the traditional/academic process, both leading to the same end result.



I've previously written about two image processing techniques toward this goal - Histogram Diffusion for color differentiability, and 3D Focal Point Blur for focal realism. In this article, I'll discuss another approach, inspired by artistic color theory.

Background

In his 2009 book Landscape Painting, Mitchell Albala describes the concept of "color identity":
"Extremes of value profoundly affect a color's ability to be read as color...A color's chromatic identity is most visible when its value is neither too dark nor too light but is in the middle-value range."
Albala illustrates this concept of "color identity" with a chart, reproduced below, showing various colors transitioning from very light (near-white) values to very dark (near-black) values. Colors that are very light or very dark have nearly no perceptible "color". In other words, they read almost as pure values, or grayscale. Another way to think of this is how "nameable" a color is. This broad definition is what Albala means by "color identity".



To clarify the terminology:


  • Hue: color, independent of lightness and intensity
  • Saturation: how vibrant, intense, or colorful a color is
  • Value: how light or dark a color is

Albala illustrates this concept by comparing two schools of landscape painting:

1: 17th century Dutch "Golden Age" landscapes are characterized by "value-priority", maximizing value contrast (sometimes called chiaroscuro) at the cost of low color identity.


Meindert Hobbema, Entrance to a Village, c. 1665


I've copied the Levels histogram from GIMP (an open source Photoshop equivalent), a visualization for how the photo utilizes the value gamut. This painting shows a strong bimodal (two-peak) distribution which creates a value-priority effect.

2: Impressionist landscapes are characterized by "color-priority", maximizing color identity at the cost of low value contrast.

Claude Monet, Antibes Seen from Las Salis, 1888




The Levels histogram shows a clear difference: just one peak, so colors are much closer in terms of value, differentiated primarily by color, creating a color-priority effect.

To simplify, you can think of these two schools as favoring differing regions of the color chart to get their ideas across:


Which is more realistic? More aesthetically pleasing? More correct? These are fundamentally subjective questions, best left to personal taste. As Albala notes, "[both] successfully depict natural light", "[although] the Impressionist approach to color is perhaps more in tune with contemporary color sensibilities".

My take is that neither image is particularly realistic, but I personally find the Impressio
nist approach much more painterly and compelling, and closer to "correct".


Albala has since expanded on this topic at his blog with a couple excellent articles:


Quantifying Color Identity

Digital color provides a quantitative framework to examine and enact Albala's view of color identity. But it first requires some careful consideration of color space nuance.



RGB Color Space

Let's start at square one with RGB color, the standard for storing and displaying digital images. RGB color has three channels corresponding to red, green, and blue light, and within a channel, a pixel can take any of 256 values, 0-255. Therefore, there are 256^3 or ~16,700,000 colors available.

RGB's core purpose is telling monitors how to display color, and it's very effective for this. However, its utility in addressing color identity is very limited, because the intuitive properties of hue, saturation, value - the way we think about color - are all interdependent with respect to the axes of R, G, B.

HSV Color Space

Invented in the 1970s, the HSV (hue, saturation, value) color space was designed to provide more intuitive framework for working with digital color. Whereas RGB is a cube, HSV is a cylinder; hue describes the angle, saturation the radius, and value the height. It's computationally easy to transform between RGB and HSV, which was initially a significant advantage, but largely moot in the current era. HSV's key shortcoming is that saturation and value are not actually independent - just close to it. This can be observed from a constant-hue chart, which corresponds to half of a cross section of the HSV cylinder. Here is H=130°, arbitrarily:



Columns theoretically have constant saturation, and rows theoretically have constant value. To test the latter, we can convert the chart to grayscale, either by explicitly switching the image mode or desaturating it completely. Both provide the same result:



Examine the grayscale chart and you'll see upon close inspection a couple key shortcomings:


  1. As you move horizontally within a row, value decreases (darkens) slightly. The actual disparity between min and max across rows is about 5%.
  2. As you move vertically within a column, the rate at which value changes is not uniform. Note for example the steep gradient from 10-15% value.

While very subtle in this example, these issues significantly impede HSV's ability to address color identity.

Lab Color Space

The Lab color space, invented in 1976, solves these problems with full independence of hue, saturation, and value, as well as perceptual uniformity. Like RGB, Lab is a 3D orthogonal color space:


  • L corresponds to a color's value, or lightness/darkness
  • a corresponds to a theoretical green-magenta/red axis
  • b corresponds to a theoretical blue-yellow axis

Whereas RGB is discretized 0-255, Lab is continuous and unlimited. Therefore, Lab can theoretically describe "imaginary" colors that cannot be displayed in RGB, or describe infinite colors between RGB discretizations. It is perceptually uniform, unlike RGB and HSV, meaning that Euclidean (straight-line) distance corresponds linearly to perceived color distance/difference in the human eye, which is a significantly non-linear sensor.

Every color in RGB color space can be converted to Lab, and vice versa. Because the equations that perform this conversion are non-linear, the neat RGB cube becomes non-trivially deformed in Lab color space:



In this visualization, a uniform, dense point cloud is created inside the RGB cube, then converted to Lab, and its boundary is approximated using a concave hull algorithm. The mesh for each boundary is overlaid on the volume. The non-linearity of human vision causes the initially-uniform point cloud to be significantly non-uniform in Lab color space, which is reflected in its boundary mesh.

These visualizations are very colorful - in fact, they could not be any more colorful, because their outer surfaces contain their most highly saturated colors. However, keep in mind that this visualization shows only the outer surfaces, which contain a small fraction of the complete color gamut. To view these less-colorful colors, you have to look inside these volumes. This can be achieved with a cross-section:



Here, I section the RGB cube along its R axis arbitrarily, and the Lab volume along its L axis. On the Lab gamut, I overlay the color space's "neutral axis", which occurs where a=0 and b=0. In other words, all colors on this line have zero hue because they are pure shades of gray. Note that the neutral axis intersects pure black and pure white. Hue and saturation are defined simply as:


  • Hue is a color's angle from the neutral axis, atan2d(b,a)
  • Saturation is a color's distance from the neutral axis, sqrt(a^2+b^2)

With this visualization, you can show that Lab has true value independence by converting any frame to grayscale:



Note that in Lab, there is only a single, constant shade of gray within the cross-section boundary - not so with RGB which has no such constraint.

This framework provides a basis for corroborating Albala's notion of color identity. Since the colors with the strongest color identity occur on the outer surface of the gamut volume, we can deduce that saturation is a proxy for color identity. Further, we can take the section area as a proxy for color identity potential at that value:



This is a general result for all hues; as you can see from the cross-section, different hues achieve maximum color identity at different L values. This can be compared with the color swatch chart from earlier:




Here I've superimposed the earlier color swatch chart with the color identity potential plot. The swatches don't necessarily occur at the correct L values, so the result is merely suggestive, but the qualitative fit between the two is clearly strong.

Why Photos Tend Toward Value-Priority and Drab Colors

Modern digital cameras in "automatic" mode are so adept in adjusting their settings under the hood that it's easy to think of their images as objective truth. However, much of this process is actually based on heuristics that are designed to provide decent results over a wide range of conditions. This is ideal for the typical consumer who simply wants to point and shoot with minimal fuss.

However, for those who paint from photos, the automatic settings often provide seriously sub-optimal results - namely, they tend to minimize color identity. This is both deliberate and reasonable: creating a value-prioritized image is algorithmically easy and will usually look decent, whereas creating a color-prioritized image is not straightforward and will be prone to pitfalls. Allow me to illustrate:

Suppose you are the camera's setting adjustment algorithm. You look out at the scene and see this distribution of light:


Note that a scale and units are not provided; a camera is not measurement tool. You simply see that there are some darker regions, some midtones, and some lighter regions. So that the image can be saved and displayed, your job is to fit this distribution into a fixed bounding box:


How do you do this? Approaches vary by manufacturer, but one certainty is that you don't want to saturate any part of the spectrum. This would mean "pushing" any part of the distribution outside of the bounding box, resulting in pure white or pure black. For example:


No good - the lower end is saturated.


No good - the upper end is saturated.


Maybe okay, but there's so much leftover space. What to do with it? You can shift the distribution left or right by as much as 25% without saturating. So which of these 50 different solutions is best? There's simply isn't a good, general-purpose, algorithmic way to make this decision. So what generally happens is that the algorithm simply scales and stretches the distribution so that it's just shy of saturation on both ends.


This requires just a couple hard-coded parameters to work correctly, and is simple, open-loop (requiring no feedback), and almost always produces decent results. Wide distributions are safer and easier to work with, because by design they fit neatly into the bounding box.

This distribution comes from a photo of mine from Yellowstone:


Does the distribution or photo look familiar? In the sense of having a wide, bimodal (two-peak) distribution with few midtones, it's actually very similar to the drab Dutch landscape from earlier. This is no coincidence - the pictures were, in effect, recorded with a very similar process.

So, digital photos have a tendency to be drab for the same reason that 17th century Dutch landscapes did. And by drab, I mean lacking in color identity. What can be done about this?

How to Shift From Value-Priority to Color-Priority

The Lab color space works well for shifting photos from value-priority to color-priority, or vice versa, for the reasons discussed earlier. To summarize the process in geometric terms:

  1. Store a photo as an RGB point cloud
  2. Convert the RGB point cloud to Lab color space
  3. Manipulate the Lab point cloud using linear transforms
  4. Convert the Lab point cloud back to RGB for display
  5. Repeat from Step #3 until satisfied with result

The linear transforms in Step #3 are simple. For each of the three properties of hue, saturation, and value, the linear transform is defined by an offset and scale which is applied globally to the entire point cloud. Continuing with the geometric analogy, these transforms as a whole enable you to move, stretch, compress, twist, and expand the point cloud in an intuitive and freeform way. It's easy to push parts of the point cloud outside the limited RGB gamut, so the Lab-to-RGB conversion process accounts for this by saturating, or limiting its input before conversion.

Because these adjustments are best done by eye, they require a good deal of tweaking to get right. To facilitate this, I wrote a GUI to quickly get the settings right before exporting. Here's an overview:



A control panel lets you define the scale and offset for lightness (i.e. value), hue, and saturation on sliding scales. There's also a slider to control the quality of the preview, and buttons to restore defaults, compare before/after images, and to export the image at full resolution.




A histogram panel shows the result of the scaling operations, before and after. This is most helpful for shifting value to where the desired level of color identity can be achieved. The functionality for scaling hue is fun to play with, because it can produce wild and impossible color palettes, but for my purposes I leave it as-is.



Lastly, there's a preview panel which shows a downsampled copy of the image with the transforms applied. An alphanumeric string across the top conveys the transform parameters, which are also appended to the filename on export.

There's some skill and taste involved in manipulating photos with this technique. In general, I try to keep the peaks of the value histogram between 20 and 80%, outside of which there is minimal color identity. With the saturation scaling, the offset helps "push" neutrals outward for greater color identity, while the scale helps accentuate the already-vibrant regions of an image.

Demo Gallery

Here are some photos that I've attempted to shift from value-priority to color-priority using this technique, all taken with my 2003 Canon Rebel without any prior modifications. In each photo, the left half is the original and the right half is the result.























And to circle back to the original concept, here are a couple examples of shifting a painting's color gamut to align with a different school of painting:

Recasting a Dutch landscape
 as an Impressionist painting:


Meindert Hobbema, The Avenue at Middelharnis, 1689

Recasting an Impressionist painting as a Dutch landscape:


Claude Monet, Coquelicots [Poppy Field], 1873


Source Code

Freely available on my GitHub.

Acknowledgements


  • Mitchell Albala provided corrections and suggestions for the Motivation and Background sections.

No comments:

Post a Comment