Sunday, July 21, 2019

How Paints Mix

Summary

For augmented realism in artistic painting, this article summarizes mathematical methods to characterize any palette of primary paints and decompose any digital color into an analog paint mix recipe, using Kubelka-Munk theory, Schmid color charts, and visible-light spectroscopy.

Motivation

Ever eloquent, Lucian Freud best expressed the importance of color realism in painting:


"...[When] colors work, I think of them as the color of life, rather than colored."

Famously dogged and penetrating in style, Freud labored over ever
y stroke, working and reworking his canvases over long sessions even into old age. His eye was a perceptive color comparator, restless until the canvas felt as true as the scene. Similarly, Rembrandt wrote that his paintings sought to achieve "the greatest and most natural movement".

Venerated by Freud, Rembrandt, and all great painters, realism is the strongest unifying principle across painting's long and diverse history.

Rembrandt, 1628

Freud, 1985

When I started painting, I was confounded by the skill of color matching, the the act of mixing primary paints together to create colors that match those observed by the eye. It requires both seeing colors accurately and knowing how to mix them from primary paints. I've seen painters who can color match with astonishing ease and accuracy, but I can only stand apart and admire their skill. I still find it confounding, though with experience I've developed a decent working knowledge.

Godfrey Reggio said this of his 1982 film Koyaanisqatsi, one of my favorites:



"...[Our] language is in a state of vast humiliation. It no longer describes the world in which we live."

Try to describe
 the spectrum of real-world colors in words and you'll quickly see that Reggio's axiom neatly encapsulates what makes color, with its intangible, experiential, and subtle nature, so hard to describe. By extension, this also makes color matching very difficult. But, as the great masterpieces in oil painting's rich history show, it's not out of reach - it's just very hard.

Frustrated with my lack of talent at color matching, and with the clumsy old-fashioned models for color mixing, I began to wonder if 
there was a cleaner, mathematical way of describing paint and color. I reasoned that since every painting can be represented as a digital picture, and since every constituent pixel has a well-defined digital color, any painting can be represented mathematically. Put another way, every digital picture can be represented as a painting.

My focus then became developing the critical link between physical color (i.e. paint) and digital color (i.e. pictures). Less abstractly, I sought to develop an algorithm that could decompose any digital color into an analog mixing recipe. This algorithm, and path to developing it, is the subject of this article.

Early Missteps in Digital Color

Having some prior experience in digital image processing, I initially expected that the problem could be solved purely in a digital color space - simply quantify the colors of the primary paints, then mixtures thereof should be simple weighted averages. However, this approach doesn't work, as a quick counterexample shows:
  1. Blue and yellow mix to make green.
  2. Blue and yellow may be represented in RGB as [0 0 255] and [255 255 0].
  3. The average of [0 0 255] and [255 255 0] is [128 128 128].
  4. [128 128 128] is gray.
  5. Gray is not green.
Converting to alternate color spaces before averaging provides similarly erroneous results: Lab predicts a pale pink, and XYZ predicts periwinkle. I also tried dithering swatches of the target color using primary paints as the set of permitted colors, as if to simulate pigment particles with pixels. Initially encouraged by the fact that this method produced different results than a naive average that were also plausible and visually interesting, I eventually conceded that it didn't work after some simple experiments disproved its accuracy.

Interesting but unsuccessful experiment at color decomposition using dithering

These methods all fail due to their neglect of the microscopic optics at play in actual paint films, which cannot be captured by reductive digital imaging due to the irreversibility of the color perception causal chain.

Enter Kubelka-Munk

After some more research, I learned that this is actually a semi-solved problem, with the theoretical foundation having been laid in 1931 by German scientists/engineers Kubelka and Munk. Later researchers extended their framework by developing practical methods of using it, especially starting in the 1980s with the advent of personal computers. Kubelka-Munk theory (K-M) remains important to many industries, especially the manufacture of coatings, films, and paints. It applies to both translucent and opaque media, although as an alla prima oil painter, I'm only interested in the simpler opaque case.

K-M posits that diffuse light incident on a paint surface is either reflected (R) off the surface, absorbed (K) by the paint pigment itself, or scattered (S) through the surface to other pigment particles. Further, it posits that R, K, and S are all functions of light wavelength.


Diagram of reflectance, absorption, and scattering in a pigment-based paint film

K-M provides the following equation relating R, K, and S for an opaque film:



which may be rearranged to solve for K/S:


K-M also holds that paint mixtures can be described as weighted averages of the constituent K and S values:



where the subscript i denotes the paint index, and c denotes the dry pigment mass fraction in the mixture. This turns out to be the extent of math needed from K-M. More math is needed, though, to actually make use of these essential equations.

Light, Reflectance, and Color

Intuitively, paint has no observable color unless illuminated, and further, its perceived color changes depending on its illuminant - whether bright/dark, warm/cool, etc. Therefore, to begin modeling color, an illuminant is needed. Conveniently, several standard illuminants were defined starting in 1931 by the CIE (International Commission on Illumination), coincidentally the same year as K-M's publication. Due to its ubiquity, I chose CIE D65, representing "average [outdoor] midday light in Western/Northern Europe". Outdoor lighting is broad-spectrum, bright, and relatively temperature-neutral, often considered optimal for lighting and photographing paintings.



Next, paint reflectance spectra are needed. This is the same quantity as the R in K-M, and it is similarly defined as the normalized diffuse reflection as a function of wavelength.


Reflectance spectra for red, white, and a red/white mixture from literature

The interplay between illuminant and reflectance creates color. However, unlike illuminants and reflectance, color doesn't actually exist in an absolute, quantifiable sense. For example, the fact that 475 nm light looks "blue" or that 700 nm light looks "red" is a perceptual abstraction of the human brain, not an universal truth. Further, perception of color varies between individuals and species, and is best understood as an inherently personal characteristic. Nevertheless, CIE further provides a "standard observer" function which quantifies the perceptual sensitivity of the average human eye to light, again as a function of wavelength.



The CIE standard observer is comprised of three independent functions corresponding to the three types of cone cells in the human eye, which specialize in short-, medium-, and long-wavelength light. It's worth noting that human vision is significantly non-uniform. We are probably most sensitive to regions of color that were most evolutionarily advantageous to our ancient ancestors. Regardless, this is an important point for later discussion on color matching.

With these three pieces in place - illuminant, reflectance, and observer - color may be calculated essentially as their dot product. This provides color in the XYZ, or tristimulus format, representing the overall signal delivered to the eye. XYZ can then be converted to RGB for display on a digital screen, or to Lab for calculating color difference. For best results, I calculate all color differences in Lab space, which by design is perceptually uniform per the observer functions, unlike RGB space.

In this framework, color may be thought of as the checksum of the underlying spectral data, in that it provides an simplified summary of the underlying spectral data. This framework also shows how perception of color is inherently reductive, i.e. an infinite number of reflectance curves may produce the same perceived color, or put another way, a reflectance curve cannot be derived from a perceived color.

Characterizing Paints

The last and most difficult part of the color mixing problem is characterizing paints - specifically, the determination of their K and S spectra. A few aspects make this step challenging:
  1. K and S cannot be measured directly, only calculated indirectly through mix tests.
  2. K and S vary significantly with wavelength and are not well-approximated as constant.
  3. K and S cannot be determined without a spectrophotometer.
  4. K and S are easily overfitted if the number of mix tests is too small.
Each of these aspects was hard-learned through various failed attempts to devise new, simpler methods for cracking this sub-problem. As far as I can tell, there is no easy, elegant method for this step, despite my best efforts. It simply requires a lot of mixing and number crunching.

One approach (Haase, Meyer) is to assume an ideal white, i.e. K=0 and S=1, then tint (i.e. mix with white) one's primaries and measure the reflectance. Algebra can then be used to calculate K and S for the pure primary. I tried this method for 5 and 10% tints (i.e. 95 and 90% white), and it provided excellent results for primary/white mixes, but did not generalize well for primary/primary mixes. I concluded that K and S had been overfitted.

A much better approach (Walowit, McCarthy, and Berns) is to build up a large database of color mixes, including both recipes and corresponding reflectance spectra, then use the linear least-squares (LLS) method to find error-minimizing K and S values at each wavelength. The authors begin with K-M's proportional weighting of K and S in mixes:



which can be cross-multiplied and expressed as:



This can then be expressed in terms of the LLS formulation of:



which can be neatly solved for B, containing K and S values for every primary:



This method is very flexible and can accommodate any number of primaries, mixes, and recipe ingredients. A full derivation may be found in the Appendix.

However, the authors do not offer guidance on which recipes to mix to sample uniformly over a palette's color gamut. For this, I looked back to the art world and discovered Schmid color charts, which use a methodical approach to mix virtually all colors possible for an arbitrary set of primaries. Every primary is mixed with every other primary, in both a dominant and non-dominant capacity, at varying levels of tinting (whiteness/brightness). Schmid's convention is that every primary/primary pair be mixed at five levels of tinting. However, since I also needed to measure the exact mass breakdown for each recipe (while Schmid doesn't), I realized that I would need to reduce resolution to make the scope manageable. I eventually settled on three tinting levels per primary/primary pair.

I then carefully set my standard palette using my prior painting experience. These primaries cannot produce any arbitrary color, but they are naturalistic and cover the full gamut for the images that I paint. I chose my most essential and favorite paints, all from Winsor & Newton's "Winton" line:
  1. titanium white
  2. cadmium yellow
  3. cadmium red
  4. burnt sienna
  5. burnt umber
  6. sap green
  7. ultramarine blue
I set the dominant primary as 75%, and the non-dominant primary at 25%. I set my tinting levels at 0, 45, and 90% white. I set a mass target for each mixed paint volume at 3.00 g, with each ingredient subject to a +/- 0.02 g tolerance. This provided more than enough paint to thickly cover a 2 x 2" canvas board swatch without too much waste.

With this, it was simply a matter of sitting down and working through all the mixes required - 109 in total. Each sample took about 5 minutes to create.

Mixing 45% titanium white, 41% ultramarine blue, 14% cadmium red with a palette knife


Fully-mixed purple paint

With the amount of effort this ended up taking, about a full weekend, I decided that I ought to frame the results.

Schmid color chart for my selected primary paints


For measuring the reflectance of each swatch, I pursued a few different paths until ultimately finding and purchasing the Spectro 1, a fantastic new spectrophotometer aimed between the consumer and industrial levels. It overall offered the exact functionality I needed at a great price, and this project would not have been possible without it.


Spectro 1, a fantastic new spectrophotometer that is about the size of a pill bottle

With the swatches mixed, dried, and scanned, I was now able to use the LLS method, combined with my measured recipes, to calculate K and S spectra for all my paints. Implemented as described in the original paper, this method provided disappointing results when comparing the measured and simulated Schmid chart, both visually and numerically. To compare colors, I used CIE76, which is essentially Euclidean distance in Lab space, normalized to a "just noticeable difference" of 2.3 ΔE.


Disappointing results for a naive implementation of the LLS method; left=measured, right=simulated

A problem with this approach is that its results are not constrained by K-M, and consequently its fitted K and S values violate K-M and cause unintended color shifts. To resolve this, I applied an additional least-squares optimization:

Given K and S, find K* and S* such that:


  • error = c_1(K-K*)^2 + c_2(S-S*)^2 is minimized, and
  • K*/S* = (1-R)^2 / (2R), i.e. K-M is enforced

where c_1 and c_2 are fractional weights. This is solved numerically by specifying a range and resolution for S*, then deriving K* for each and calculating the corresponding error. This is performed at each wavelength for each paint. To optimize the values of c_1 and c_2, I defined bounds for c_1 as 0 to 1, and c_2 = 1-c_1. For each pair of weights c_1 and c_2, I optimized K* and S* using the above method, then calculated the average color prediction error against the Schmid chart.


Optimization result for full K* weight domain, 0-100%


Optimization result for K* weight domain near minimum, 2-4%

Repeating this for an array of c_1 values, it becomes clear that the optimal solution is to heavily weight S (in this case, c_1 = 0.027), nearly to the point of discarding K and defining it per K-M as a function of R and S. Incidentally, this method is much simpler and nearly as good, with an average color error that is only about 4% higher for my dataset.

With K*, S*, and c_1 optimized, the color predictions become very accurate - on average, 1.535 JND for the full dataset. In other words, the average color difference is just noticeable, and if you squint (a classic artist's trick) at the comparison chart, the seams between the colors practically disappear.


Final result with LLS and optimized enforcement of K-M; left=measured, right=simulated

Cross-Validation

Aware of the risk of overfitting K and S with too few samples, I cross-validated using the holdout method. To do this, I randomly and repeatedly partitioned the full dataset into "fit" and "test" subsets, calculated K and S using only the "fit" subset, then tested the resulting model against the "test" subset. For expedience, I kept c_1 as 0.027, which is sub-optimal and conservative from the perspective of cross-validation.


Holdout cross-validation showing stability of average color error with decreasing data

The relative flatness of the trial-average line, even up to 50% withheld, confirms that these methods and spectra are well-generalized and not overfitted. It also suggests that the number of swatches mixed could have been reduced with minimal penalty to the model's accuracy. A useful metric here is mixes per primary. One source (Baxter, Wendt, Lin) shows 71 mixes per 11 primaries, or 6.5 mixes/primary, with relatively poor color matching. My 109 mixes per 7 primaries, or 15.6 mixes/primary, has better results, though is much more time-consuming. More mix data, provided it is well-distributed over the color gamut, will only improve results, although with diminishing returns as K and S converge.

Decomposing Colors

Once K and S are known with confidence, decomposing colors into recipes is relatively trivial. A brute-force optimization function can simply simulate thousands of random recipes and select that which minimizes error to the target. The result can then be stated in the convenient and familiar format of "parts" for ease of use on a palette. To make the results useful, a few constraints are needed. These are baseline values I consider reasonable from painting experience:


  • Maximum number of ingredients: 3-4
  • Minimum fraction per ingredient: 5%
  • Fraction resolution: 5%

Before decomposing digital colors, it's necessary to first scale the image's value (i.e. lightness/darkness) range to match that of the palette. This ensures that all colors are producible by enforcing that the darkest image color corresponds to the darkest color in the Schmid chart, and so on for the lightest. This is analogous to a "Levels" adjustment in Photoshop or GIMP. This is necessary due to Illuminant D65's intense outdoor brightness which un-saturates even the darkest paint mixes that appear nearly black when indoors.



Value scaling comparison; left: original/indoor, right: re-scaled/outdoor


Demo: Decomposing Existing Paintings

Rembrandt's pigment choices and working methods are subjects of academic study in and of themselves, but for the sake of illustration let's suppose that Rembrandt used my well-characterized palette of seven primaries. The color of any pixel in the image may then be decomposed into an error-minimized recipe for mixing. An ingredient quantity limit of three is imposed. Representative samples are shown below. 


Color decomposition demo on a Rembrandt self-portrait

This painting is fairly monochromatic, and accordingly its recipes are dominated by burnt umber, titanium white, and cadmium yellow. Ultramarine blue, cadmium red, and sap green make only minor appearances. Burnt sienna is not used at all, suggesting that it is not needed for this particular painting.

Color decomposition demo on Monet's Impression, Sunrise

Here's another demo on Monet's Impression, Sunrise, which crystallized the Impressionism movement. I've increased the ingredient quantity limit from three to four, and so the recipes become more complex and accurate. Monet uses titanium white heavily to create the pastel hues associated with aerial perspective and marine fog. Every paint is represented, although cadmium yellow is nearly absent. Cadmium red is reserved for the intense morning sun, and burnt sienna is de-emphasized in favor of the cooler burnt umber. The darkest colors are still about 20% white, so Monet is remarkably restrained in his use of darks.

Demoing on photos of existing paintings provides an intuitive and instructive result, but this method works equally well in reverse with ordinary photographs. It could also be used to determine the best primaries for a given image, or estimate the primary paint volumes required to minimize waste.

Demo: Creating New Paintings


Working in reverse, any digital image may be translated into an oil painting. In these examples, all mixi
ng recipes were generated using brute-force color decomposition against a reference image. Unlike the Schmid color mixes which were precisely weighed, these recipe ingredients were measured by eye in the "parts" format, since weighing so many mixes would be impractical and un-painterly. Also, the error introduced from measuring recipes by eye opens the door to a degree of randomness and spontaneity that all great paintings possess.

Initial surveys of the reference images indicated that certain primaries were not necessary, so they were omitted from the palette. Since color decomposition is is done by random brute-force iteration, eliminating unnecessary primaries improves both the repeatability and quality of solutions. It also reduces paint waste and mixing complexity. This corroborates the traditional wisdom of the "limited palette" often advocated by art educators.


Still life; sap green and cadmium red omitted


Landscape; burnt sienna omitted

Conclusions

This article summarized some color science that is relevant to paintings, and methods for characterizing paint color mixing properties and decomposing colors into mixing recipes. All of this is carried out to the highest level of accuracy possible with a hobbyist's means. The resulting framework provides a new tool for creating accurate, engaging paintings, or decomposing existing paintings to better understand the artist's methods. Additional applications may include art restoration, forgery detection, and art education. My primary interest is to use this framework to develop color mixing intuition and support the creation of new paintings.

Acknowledgements

  • Orion Taylor suggested the K* and S* optimization which improved color prediction accuracy by 4%.
  • Drew Hmiel advised on spectroscopy aspects.
  • Bunny Harvey sparked my interest in painting and instilled the importance of color matching.
  • Serena Tolar clarified the terminology for color-measuring instruments.

Appendix


Derivation of LLS method for finding K and S values, per Walowit, McCarthy, and Berns

Sources

"Modeling Pigmented Materials for Realistic Image Synthesis", Chet S. Haase and Gary W. Meyer, 1992.
Provides a good introduction to K-M, although their method for finding K and S causes overfitting.

"Useful Color Equations", Bruce Lindbloom.
Provides all necessary equations for converting between XYZ, Lab, and RGB color spaces.

Colour & Vision Research Laboratory, Institute of Opthamology.
Provides observer and illuminant functions.

"IMPaSTo: A Realistic, Interactive Model for Paint", Baxter, Wendt, Lin, 2004.
Summarizes the linear least-squares method for finding K and S spectra.

"A Review of RGB Color Spaces", Danny Pascale, BabelColor, 2003.
Provides some quantities for working with RGB color spaces.

"An Algorithm for the Optimization of Kubelka-Munk Absorption and Scattering Coefficients", Walowit, McCarthy, and Berns, 1987.
Provides a detailed walkthrough of the LLS method as applied to K-M.


Alla Prima II”, Richard Schmid, 2013.
Describes the “Schmid Color Chart” method for use by painters.

8 comments:

  1. This is so far the most useful article I have found on Kubelka-Munk for color mixing, thanks! Did you publish anything on Github?

    ReplyDelete
  2. Thank you, Adrien! I have not published my code for this. However, I suggest you check out Paul Centore's fantastic "Kubelka-Munk Routines", which I now use. His methods for deriving K and S supersede WMB and significantly outperform all methods discussed here. For my dataset, the error is roughly cut in half. I've also learned that the reason a standard set of mixes is hard to find is that it's something of an industrial trade secret. A Schmid Chart works well as a simple "open-source" alternative! As you can see from the cross-validation, one can achieve nearly equal accuracy with far fewer mixes.

    ReplyDelete
  3. Thanks for the tip! Such a pity all these routines are written in Matlab and not C or Python! I will definitely try to "translate" this for my own use.

    ReplyDelete
  4. Dear Daniel,

    I hope you have some minutes to answer a few basic questions...
    I have downloaded Centaure's routines and give it a first try.

    When you are using Centore's algorithms on Octave (or are you using Matlab?), how do you proceed? Do you compute K and S of your entire database at once? Or do you compute K and S seperately for each individual pigment?

    For evaluation, I have made several mixtures of organic pigments and titanium white. Using a coating bar, a precision scale and a mechanical mixer to have reproductible results, I have made samples on opacity charts.
    I have measured the samples using an eye-one pro spectrophotometer and formated the data as specified by Centaure in: "Wavelengths", "Reflectances" and "Concentration" matrices.

    I am wondering, however, if it is better to calculate K and S individually, for each pigment, or if it is better to include all the available data, including mixtures that combine several pigments, into the data that is sent to the function "KandSfromMixtures"?

    For only one pigment + white, the "Concentrations" matrix is very small, with only two columns. However, if I include the entire data I get a large matrix with as many columns as there are pigments in the database. I am wondering if this is good practice.

    it seems to me that the computation of the results is extremely slow but maybe I have used a too high resolution of 3.333 nm for my samples, which increases a lot the computation time?

    Best,
    Adrien





    ReplyDelete
    Replies
    1. Hi Adrien - the inputs required for Centore's programs, which I run in MATLAB, are the same as described in the Walowit paper I cite. You will see in this paper that K and S for all paints and mixes are computed simultaneously, not individually. This is the standard practice. Your matrices should be large - for example, my concentrations matrix has dimensions of [109 mixes x 7 primaries]. As for your choice of mixes, I suggest you also include black in addition to white, or do something similar to what I've described, to have confidence in the generality and accuracy of your result. As for the wavelength resolution, 3.333 nm is very fine; you can safely coarsen to 5-10 nm without penalty. The overall calculation should not take more than a couple seconds. You can read more about black/white mixing in Kirchner's paper "Instrumental colour mixing to guide oil paint artists", which is very similar to my article. Good luck!

      Delete
  5. Thank you so much Daniel! for the method tip and the article, both are great. I didn't think everything had to be put together in the same matrix. Now it works really well!!!
    There's just a thing, I don't know why but Centaure's algorithms is extremely slow to execute on Octave and doesn't work on my version of Matlab (there are many detected errors in the code). Any idea why this would be so slow? I have implemented Walowit in numpy/python and it is extremely fast, however.

    ReplyDelete
  6. Glad to hear of your progress!

    Centore has developed a few different functions for deriving K and S. I tried several of them, and also had issues getting some of them to work at all. I did not investigate enough to determine if the issues were due to software or my specific dataset. Ultimately I settled on KandSfromMixturesCentore2013.m which may be obsolete from Centore's perspective, but nonetheless is still significantly better than the Walowit method, and practically about as fast. I think Walowit's method is fastest while Centore's are slower but provide better results. Walowit's method might still be useful in some scenarios where speed and simplicity are preferred over maximum accuracy.

    ReplyDelete
  7. Okay thanks! I'll have a look at it in more detail when I have time. Now I am familiarizing with the practical aspect of it using Walowit. Since I am using opacity charts, I may even try the traditionnal method that implies measuring the film's thickness and using the equation that contain log functions. In anycase thanks so much for your advice and feedback, that helped a lot.

    ReplyDelete