The Gimp is an extremely exciting image processing package. It has a very wide range of image manipulation functions, but a somewhat limited range of tools for "painting" original images. As such, it compares very favorably to Adobe PhotoShop, but is weaker competing against Fractal Design Painter on the latter's ground.
This document outlines a proposal to add "natural" brush types to the Gimp. The "core" natural brush functionality basically adds sensitivity to paper texture. Of course, any painting system today must support full sensitivity of Wacom pads and the like. Owen Taylor has been working extensively on Wacom support for the Gimp, so this document assumes that functionality is already present (although what I'm proposing here goes a bit beyond Owen's patches as of the time of this writing). This level of brush functionality gives satisfying pencil, pen, charcoal, chalk, crayon, pastel, and airbrush effects.
One consequence of these extensions is that painting has many more options and settings than in Gimp 1.0. It will be challenging to organize these features in a coherent, usable framework. Fractal Design Painter, in particular, has significant problems in this regard. I will propose a solution based on integration with a Web browser.
I'll also talk about fancier brush types, including watercolor (a high priority for me), liquid metal (graphic cliche of the month), and others.
These are very exciting times for free software. With the changes I propose, the Gimp will be the finest artistic painting program available, as well as the best general image processing program. This is an exciting goal. Let's do it!
When you select the paintbrush tool in the Gimp and start painting into a window, it seems pretty simple - it just paints. However, it's actually doing quite a lot of work behind the scenes.
The generic model for painting is to discretize the stroke into a sequence of dabs, each of which is processed separately. Here's what actually happens:
At the beginning of the brushstroke, a new temporary image buffer is created. It has an alpha channel (i.e. it's either GRAYA or RGBA), and is initially transparent.
Each dab starts by retrieving the current brush shape. This is done in paint_core.c by paint_core_get_brush_mask (). In the paintbrush mode, the brush_hardness is always SOFT, which means that the paint_core_subsample_mask () method is used. This method shifts the brush shape by quarter-pixel increments so that the dab is placed with extremely good accuracy. If dabs were placed with only single-pixel accuracy, visible stairsteps would be noticeable. Since there are only sixteen possible positionsm, the subsample
This brush mask is then composited with the current paint color into the temporary buffer. More technically, the brush mask is used as an alpha control for selecting between the current color (with 100% opacity) and the existing contents of the temporary buffer.
Finally, the temporary buffer is composited into the drawing layer, using the combination mode (e.g. normal, multiply, etc.) and opacity from the brush dialog.
Although these options are not available from the paintbrush UI, the Gimp internally supports a few variations on these modes. First, it doesn't have to to the initial compositing into a temporary buffer (the CONSTANT mode). Instead, it can paint directly into the layer (the INCREMENTAL mode). Also, the brush_hardness can be HARD, which causes a hard-edged brush mask to be generated (no subpixel sampling, each pixel is either fully opaque or fully transparent).
Owen's Wacom patches for the GIMP add a third brush_mode: PRESSURIZED. This mode starts by computing the subsampled brush, then making it more or less intense based on the pressure. This patch shows the way to more flexible brushes: the common element is to apply the brush effects in the steps between retrieving the original brush shape and returning a brush mask from paint_core_get_brush_mask ().
This section briefly describes the FDP "core" brush architecure. While I believe it is accurate, it is based only on my reading of their patent and experiments with their software.
The basic architecture of dividing the stroke into a sequence of dabs, computing each dab based on a brush mask and compositing, then compositing the result back into the drawing area is the same as the Gimp (in fact, it appears to be common to almost all paint programs). What makes FDP special is the way the brush mask is derived from the Wacom information.
The first step is to align the brush mask with a corresponding "paper height field." This field is generally a repeating tile (typically 256 pixels on a side) that represents the paper texture as a height value at each pixel. The theory is that many natural painting tools such as charcoal deposit a lot more pigment on the raised areas of the paper than the valleys.
Thus, the final brush mask is computed as a simple combination of the subsampled brush shape, the pressure from the Wacom, and the paper height field. I don't know this for sure, but I'd suspect it's a simple linear combination, clamped at 0 and 255 (if you trace very lightly with the pen, most of the drawing area is left untouched; similarly, if you press very hard, a lot of it is saturated with the pigment).
The "subsampled brush shape" mentioned above is actually a nontrivial transformation of the original brush shape. The patent talks about precomputing a subsampled brush shape for each combination of subpixel position and size (sizes can also vary dynamically based on Wacom pressure - a feature not present in Owen's patch). Then, when actually painting, FDP just picks a subsampled brush shape that's closest to the desired position and size. FDP 5 goes slightly further and includes rotations as well.
As you might imagine, then number of subsampled brush shapes can become quite large, I'd estimate typically in the low hundreds. FDP has an explicit "build brush" operation to precompute all of this stuff. For small brushes, it's not very time consuming, but for large ones, it can get in the way of flow. There is an "auto build" option, but it seems to be turned off by default.
FDP has a lot of painting options. The combine mode, sensitivity to paper, and a few more are rolled up into a small palette of "methods." The "cover" methods use a normal combine mode, while the "buildup" methods use multiply (actually, it's a little more complicated in reality). I'm not even going to start listing these options.
I find many of these options baroque and confusing. For example, it's not possible to use the same brush for painting in an opaque layer and on a transparent layer - there are two different brush modes. Plus, there are a lot of controls and options that probably aren't that interesting for serious artistic work (but maybe there are other people who use these on a day to day basis, who am I to say?).
FDP 5 also has a wide variety of plug-in brushes. I haven't really analyzed how they work. There are some serious usability problems, though. For example, for the liquid metal effect, you don't just select a liquid metal brush like you'd expect, you have to create a liquid metal layer and interact with the dialog rather than the regular painting process.
FDP also has some semi-interesting watercolor brush modes (dating all the way back to the first release). I'll mention these again in the section on watercolors.
Fractal Design Painter's painting techniques are covered by US Patent 5,347,620, granted to Mark Zimmer. The disclosure of this patent is quite detailed and well-written for a patent, and is well worth reading for more details on the techniques used.
However, the existence of this patent presents problems for other people trying to achieve similar results.
Here's my take on what the claims in this patent actually cover. There are three claim trees. The first (claims 1-3) covers the technique of adding color in a logarithmic color space. It can safely be ignored. The second (claims 4-7) covers the idea of precomputing a discrete set of brush masks of different sizes, then choosing the one closest to desired when actually painting. The third (claims 8-10) similarly covers a discrete set of subpixel offsets. This is probably the one we have to worry about most, especially because the subpixel brush mask in paint_core.c does cache brush masks that have been shifted by subpixel amounts.
Incidentally, Fractal Designs (now Metacreations) also appears to have a trademark on the phrase "natural media." Any ideas on what to call the Gimp's version of this idea?
With this background in place, I come to my proposal. Most of the "new" stuff has to do with the transformation from the original brush shape to the brush shape. As far as combining paper texture and pressure, I propose to do it essentially the same way as FDP.
All of the transforms done by FDP (subpixel placement, size, and rotation) are special instances of a general affine transformation. The core of my proposal is to store the original brush shape in somewhat enlarged form, then use a general affine transformation to generate the brush mask that is then combined with the paper texture and pressure controls. If the original brush shape is stored in enlarged form, then a simple nearest neighbor algorithm will give good results (i.e. small subpixel positioning errors) without slowing things down considerably. One significant optimization is limit transforms in most cases to five degrees of freedom rather than the six of a generalized affine transform (x pos, y pos, x scale, y scale, and x skew, omitting y skew). For rotationally symmetric brushes (a very common case), the five DOF case is just as general as the six.
This technique has a few advantages. The first is scaling and positioning without noticeable quantization errors. The second is the lack of a time-consuming "build brush" step. Third, it makes "tilting" effects much more accessible, which might be nice with the larger Wacom pads that support x and y tilt.
There are two costs. First, generating the brush mask using an affine transform will take some compute time, although I believe it will be be insignificant compared with the rest of the painting process. Second, to work well, the brush mask has to be either stored in fairly enlarged form or enlarged once when it's loaded. However, a single enlargement operation would seem to be a lot less problematic than precomputing a largish number of more specialized brush masks.
Thus, unlike Gimp 1.0 and FDP, I propose generating a brand new brush mask for every dab, rather than precomputing (or caching) all the brush masks you could ever need and selecting the one that's closest. This proposal steers wide of the Zimmer patent, provides more flexibility, and a very good performance tradeoff.
One thing that's been discussed on the Gimp IRC channel is the idea that simply using the paper texture in linear combination on a pixel-by-pixel basis is inadequate. Peter Mattis has proposed using the first or second derivative of the height field instead.
I agree that there is scope to do some cool things here. My idea is to let the paper texture vary depending on the brush size. If you're using a big piece of chalk, it's sensitive to all the peaks and valleys of the paper, while if you're drawing with a pointed tip, it rides the coarse features and remains sensitive to the fine ones. The simplest way to acheieve this would be to high-pass filter the paper texture based on the brush size. Doing the HPF on the fly would be quite expensive, so a reasonable compromise would be to store pre-filtered versions with different cutoff frequencies and interpolate between them.
I see this effect as being most useful with tilt-sensitive Wacom pads, simulating the effect of pastels used on-point and on the side. The two techniques give quite different textures, and I think it would be a cool feature to have (especially given that FDP lacks this level of expressivity). Only problem is, I don't have a tilt-sensitive pad. Anyone know where I can pick one up cheap?
Another important natural brush effect is watercolors. We have two models: FDP's watercolor brushes, and some watercolor simulation work done by Cassidy Curtis (University of Washington) and others (as a SIGGRAPH '97 paper). I find FDP's results disappointing compared with real watercolors, and while Curtis's results are impressive, the technique takes far too much computational horsepower to be useful as an interactive painting technique on today's hardware.
My proposal is somewhere between the two: to borrow some of the key ideas from the Curtis work, while keeping an imaging model that's much more like Gimp's existing layer modes than a detailed simulation.
As far as I can tell, FDP's watercolor mode is a basically a second RGB layer superimposed over the background layer in multiply mode, with one twist: the "wet fringe" effect. I think this is just a highpass filter effect to increase the amount of paint color near edges. It's computed dynamically from the stored RGB layer before compositing with the background layer. In this way, it's not unlike PhotoShop 4's "adjustment layers." Note, however, that FDP's watercolor brush was implemented before layers took over the world, so it's not really integrated with the rest of the program.
Here's a rough outline of what I propose. As in FDP, watercolors are done in their own layer - either RGBA (for normal compositing) or RGB (for multiply compositing, as the A channel is redundant in multiply mode, and painting into an RGBA multiply layer doesn't work too well in the Gimp anwyway). In addition to these channels, an additional channel is designated the "wet mask," more or less as in the Curtis paper. Actually, in their paper the wet mask is either off (i.e. dry) or on (i.e. wet) at each pixel. However, it seems obvious to extend this to a continuous value indicating how much water is on the paper.
Painting into a watercolor layer is similar to regular painting, with a few small differences. First, everywhere the brush touches becomes wet (i.e. sets the appropriate pixel in the wet layer). Second, when painting in an area that's already wet, the brush mask is modified to let the paint diffuse outward. This diffusion should be sensitive to the wet mask (i.e. diffusion stops at the wet/dry edge) and also to the paper grain (i.e. more diffusion in valleys than peaks). Other than that, though, it's basically regular painting.
In addition to painting, there is also a drying process. As explained in the Curtis paper, as watercolor paint dries, pigment is attracted to the wet/dry boundary, resulting in a sunburst effect. A slightly more subtle effect is for pigment particles to clump together, giving a grainy texture. This graininess resembles paper grain, but is subtly different because the grain patterns will differ from run to run, while paper grain patterns are deterministic. This makes a significant difference when painting one color, letting it dry, and overpainting another (or more than two layers) - you get a beautiful colored texture effect unique to watercolors.
I propose simulating the drying process using a fairly simple cellular automaton running as an idle process (i.e. it gets updated whenever the user is not painting). All wet pixels in the wet mask slowly tend toward becoming dry. The difference equations at each pixel cause color to flow towards the wet/dry edge, and also towards granulation centers. I'll probably cheat and synthesize the granulation centers directly rather than trying to derive them from a physically-based simulation.
There are a lot of different aspects of the Curtis work that I'm ignoring or abstracting. For example, in their model paints have nine different parameters (RGB reflection values, RGB scattering valuesq, and three parameters for the drying process). In my model, it's just RGB color and the choice between normal and multiply compositing mode. Some expressive power is definitely lost, but it makes the pigment model much easier to understand for someone who's already familiar with the Gimp. Looking at Figure 5 in the paper, it is clear that it's a reasonable model: pigments such as Quinacridone Rose are very accurately modelled by the multiply combine mode, while pigments such as Indian Red are fairly accurately modelled by the normal combine mode. Certain "exotic" pigments such as Hansa Yellow and Interference Lilac do not get accurately modelled, although the screen combine mode comes pretty close the latter case. It's possible that new combine modes may be of value, as well (one that interpolates between normal and multiply would seem to be especially interesting).
I'm also abstracting away the fact that the water can contain multiple pigments. There is only one "pigment," and it's directly represented by the RGB color in the layer.
One of the important watercolor techniques emphasized in the Curtis paper is "glazing." In this technique, different layers are painted, allowed to dry, and then painted over. Because of adsorption, the pigment that's already dried is not redissolved when more water is added to the page. They carefully simulate this adsorption process, but I think in the Gimp it would be easier to fake it using existing layer operations - i.e. to simulate adsorption, just merge the watercolor layer down and start a new watercolor layer (fully drying the watercolor layer first if necessary). If this turns out to be an important enough operation, we could add a menu entry to perform all these steps automatically.
I have little interest in modelling water diffusing the capillary layer (i.e. backrun effects). It adds considerably to the complexity of the model, and in any case it's not at all clear how to export this capability to the UI.