Color-space handling

[MrKepzie]

Standard version: 1.5 (Should be set as 1.5 milestone by @garyo)

Subcommittee
Dennis Adams, Phil Barrett (Their Github user account (if any) should be set to assignees by @garyo)

Standard Change Workflow

• Create proposal as issue (you're doing this now!)
• Tag this issue with standard change tag
• Identify subcommittee: at least one plug-in vendor, and at least one host
• Discuss the idea in this issue
• Write new or updated code and doc
• Publish updates as a pull request (ideally on a feature/PROPOSAL-NAME branch)
  • Make sure that PR references this issue number to keep them in sync
  • Discuss and review code in the PR
  • Meet all requirements below for accepting PR
• When subcommittee signs off and other members don't have any further review comments,
  maintainer merges PR to master which closes PR and issue

Requirements for accepting a standard change:

• Header files updated
• Documentation updated
• Release notes added
• Compatibility review completed
• Working code demonstrated with at least one host and one plugin
• At least two members sign off
• No further changes requested from membership

Overview

There is little to no information about the colour space (tone curve and RGB chromaticities) in which image sample values are to be interpreted.

Historically there has been some assumption that the tone curve is probably linear, with a nominal black point at 0.0 and nominal white point at 1.0, but this is not specified and is often not the case.

Depending on the host, unmodified images presented to plugins may be linear, or "video" (typically Rec.1886, or a simple gamma curve), or "log" (e.g. Cineon log, or a modern log curve such as ARRI LogC), or "HDR" (e.g. Rec.2020) or any number of other curves. The primaries may be Rec.1886, or a larger gamut such as Rec.2020 or ARRI WideGamut. Plugins have no way of knowing what the numbers in the image represent. Similarly hosts have no way of knowing in which colour space(s) (if any) a plugin would like its inputs, nor the colour space of its output image.

Some plugins won't care what space the images are in. Some however need to know in order to make sense of the data. A lens flare, for example, needs to behave very differently (a) when the input data is linear with highlights well above 1.0 (b) when the input data is log-like with highlights preserved but much lower in the range, (c) when the input data is in a display-referred colour space where highlights have been rolled off to fit under 1.0.

It may be that some plugins and/or hosts prefer to work with non-RGB primaries, e.g. X'Y'Z'. A mechanism to specify colour spaces could support this.

Solution

At first glance Pierre suggests:

Define strings to identify well-defined tone curves, and well-defined sets of primary chromaticities. This is necessarily vague since without defining tone curves using formulae we can't offer general support. Chromaticites could be identified precisely as three (x,y) coordinates but that might be too much?
Add kOfxImageClipPropToneCurve and kOfxImageClipPropPrimaries properties to clip property sets to identify the colour space of the clip, if known
Add kOfxImagePropToneCurve and kOfxImagePropPrimaries properties to image property sets to identify the colour space of the data, if known
Hosts set these properties where they know the colour space of input clips
Define a protocol so plugins can request that input clips are converted by the host into a specific colour space
Define a protocol so plugins can identify the colour space of their output image (during setup time, before rendering)

Denis Adams comments on 28 sept 2016:

Great proposal, well thought-out.

Long ago at an OFXA meeting someone (Gary @ GenArts?) brought up adding a color space property for images and my first comment was "no way it could work because there are so many and no host and plug-in could understand all of them" so the request got refined to "we'd at least like to know the primaries and if the tone curve is log-like (log or gamma with neutral gray near-ish to 0.5) or linear (neutral gray around 0.18)" which is really quite reasonable. Today I'd even expand "linear-like" to be SDR (mostly within 0.0 to 1.0) or HDR (can go much above 1.0 for HDR). You could get more specific (Gamma value, Cineon, S-Log, PQ, REDgamma4) using a secondary property for those host/plug-ins that really know. Note that hosts that let the user apply arbitrary LUTs really don't know what color space they are processing plug-ins in. That covers the tone curve side of things. Then add a “primaries” property, which could be “rec709”, “rec2020”, “dcip3”, “acesap1”, “sgamut3cine”, “DRAGONcolor”, etc. Again, perhaps more than most hosts and plug-ins could agree upon. Would they be better as CIE xy coordinates or something much simpler (a kind of S,M,L,XL range like “SD”, “HD”, “UHD”, “ACES”)? What exactly would plug-ins be doing with the primaries indication? We probably also need to think about color space for color-based parameters (which most hosts expose as color pickers) – traditionally they have been sRGB but maybe they need some kind of color space handling as well (or at least understood to be in the same color space as the images?).

In addition to a property indicating what color space images are in, we’d like a way for a plug-in to indicate that it can or can’t handle HDR (not sure if this should be opt-in or opt-out). Then the host can do range reduction (typically by converting to a log curve) to bring things into SDR range. For example, we have an HSL Adjust plug-in that hates HDR. Another example is a Lift/Gamma/Gain color corrector – they are meant to be run on images with a log-like tone curve. If all we had was an image property saying these were HDR then the plug-in could apply a log curve before and inversely after processing, but the host really knows more about what to do than a plug-in. A specific example: if the host is doing ACES it would convert to ACESproxy.

In any case, Sony is interested in this extension.

Pierre Jasmin comments on 28 Sep 2016

Some more notes- in general here we only care to have an approximation of gamma, for example linear, sRGB/gamma 2.2, log. So we would be care a bit but not that much.

The exception is our color matching tools, that would benefit a lot if the two clips could be automatically brought by host in same basis (example matching an h264 to a SLog)...

Color parameters would benefit from an additional hint so the values are in right space (thinking about host that display a color picker also).

Color space properties could perhaps be a parameter populated by host, a bit like in Spatial Format - the example of Nuke Spatial Format parameter. This way an host can populate that with what it supports instead of plugins understanding what it means.

I would think that maybe we might need a suite that has conversion methods abstracted to: e.g. Gamma 1.0 (linear) to ColorFormat (for example rec2020 selected in for example in a color space property menu). Like this effect can ask linear for inputs and pass result through host supplied conversion so everything stays in same colorspace on host side and plugin does not need to know about every possible Color Format, just call toColorFormat.

Pierre Jasmin comments on 31 Jan 2017

So for Color discussion item net before this meeting (and I stay out of this one as we have on this list FilmLight, Assimilate, BlackMagic and Sony here - and the last 2 even make cameras). Although as a reminder this started with Phil asking my host is natively LOG, in what color space do I give you images?

Also thanks to Alexandre, thinking about meta-data, the latest in the parallel world of OFX Meta-Data discussion is Clip Preferences Properties are basically clip associated (or indexed by clip) time-invariant meta-data. Clip Preferences Props happened to be the clip metadata that are defined in the API (given a name). So this becomes how to associate to a clip additional color meta-data with the particularity that this imply a way to set request to have clips converted to a specific color space. So a particular case (like we have for clips versus parameters) that does not fit the meta-data suite model, which I think means that it needs to be a suite where the first purpose of the Color suite is to pass back the ColorSpace wanted to affect the clipGetImage...

So if a ColorManaged image/clip could have an associated "ColorSpace MetaData collection" (so meta-data added to the base Clip Properties). And by collection I imply that it can be described by a set of values without callbacks. Once you see the world like that, there could be only one clip property ColorManaged or RAW (to mean here ColorManaged = false) in the Clip Preferences Prop. And the Clip Preferences action be a place to set the active ColorSpace MetaData collection (even if this is not a Clip Preferences Property per se). And unlike other types of meta-data I think only the host can populate supported "ColorSpace MetaData Collections" as it would probably be a bad idea for hosts to depend on an effect to convert images for another effect...

      Something Gary reminded us earlier is this has to work with clipLoadTexture not just clipGetImage. And Pierre (me) adds:   OfxStatus (* clipGetImagePlane) in multi-plane suite should also be following this as well on a sub-image basis (e.g reflection.r, reflection.g, reflection.b is of type colour so should I think follow same logic, while a depth image plane would not -or if you like would be tag as gamma 1).

In Phil's proposal (http://openeffects.org/standard_changes/colour-space-handling ) :

Define a protocol so plugins can request that input clips are converted by the host into a specific colour space
Define a protocol so plugins can identify the colour space of their output image (during setup time, before rendering)

      Now the word colorspace is ambiguous to start with (color space as LAB, YUV, XYZ, RGB,   color space as "color profile" - sRGB, Adobe RGB, rec 2020,   or color space as in Log versus Linear (and by extention linear with gamma - perhaps even with Levels handles for black and white points...). 

      So before we go too deep here the obvious:

        There is not much point in adding a CYMK colorspace.  By the absurb if I just said to host just give me all the colorspaces you support as option and I will let user decide - it's likely not to be very productive.
        Most of the time an effect will not care whether an image is in gamma 2.2, sRGB, rec 709,... you have to be a pretty specialized plugin to worry about these distinctions but one might so it's good to support such distinction (yes there can be effects that need to understand color gamut...). 
        To be in gamma 2.2 ballpark or linear(darker) or log(flat) however starts to matter to a lot more to effects. It matters more then to users in the sense that the user might be looking at the LOG file with an sRGB viewer so he does not even see the image as flat. The presence of white and black point only more matters when this as for consequence to remap e.g. 64 to 941  to 0-1023 in my back sort of things. Then it has conversion to LOG like effect.

Follows from that, we are not discussing here color correction  etc but a time-invariant property set that affects the clip pixels we will receive. Yes?
In that sense we sort of have variations of lift/gamma/gain (e.g. like CDL, or black point - white point, gamma)... we have typically device dependant Curves with strongly defined chromacities/white points and maybe LUT. I might have the breakdown wrong, but let me try - the goal in that list is to define what type of colorspace for what - not to say there is the equivalent of 4 supported ColorSpace MetaData Collection.

1. Do we need to start this discussion by Is Linear the reference space of OFX?  Is Linear gamma = 1.0 (what is ColorManaged=false) - with black point at 0.0 and white point at 1.0. Isn't it fair to say that most effects if they care, only care about gamma space. So the first collection to support maybe is gamma space and it has one value: gamma. Not lift/gamma/gain...  just GammaSpace. In this mode the effect is ignorant of black and white point (64 and 941 in 10 bit basis in example above). If I select this mode I don't expect host to stretch 64-941 to 0-1023.  Similarly if I say ColorManaged = false I mean it - that is if I draw a vertical greyscale ramp of 0 to 1023 in an image that is 1024 pixels high, the next effect will get 0-1023 (as opposed to get a color transform in my back because the host thinks it's in gamma 2.2 color space or something and assumed because I said ColorManaged=false that I must be linear and need to be converted.

2. This does not address 2 basic things:  Host native LOG (e.g. FilmLight) who would want to support that and to give another example, in Sony Vegas one can work in 32b Video levels or 0 to 1.0.  I am not sure if these two things can be combined in a Cineon/DPX like model...  (except LOG to LOG, LOG to Gamma Space, Gamma Space to LOG, Gamma Space to Gamma Space maybe?) - this adds black and white point support to the Color Image Meta Data. This could also have more descriptors (so be like lift/gamma/gain or CDL (same operators in different order - not commutative?)). Not my job to desnarl but obviously we are still somewhere where all this can be described by a small set of parameters. This does not have to be one MetaData ColorSpace collection, I am just defining an example category of ColorSpace. Now I am  throwing ASC CDL and LOG in same sentence (A reference color correction versus a technical descriptor about the image).In the regular metadata suite scenario there could be a CDL thing but it would not ask the host to modify how it gets image to effect etc, it's just meta-data.

3. Then there is a whole space of nuances (what personally I call normally color profiles ). Now we are in domain of specifics like Adobe RGB, sRGB, rec 601, rec 709, etc.  Assuming all this is in RGB(A), then shouldn't this be a mode that has an associated option menu. Unless you write the NTSC color safe plugin sort of thing it's not always needed, we have reached a level of specialized usage. This already becomes complicated, does a plugin need to collect a list from host and filter out the ones they know... 

4. And then we can have an even more dense thing. This might include not working in RGB(A) and the chromacities, tonal curves / "roll off" and what not become critical. This is about my pay grade here :)  --  ACES etc

Additional Notes:

       Unlike other "metadata" types, because there is an implied getImage converted,  in this case I can't imagine someone else than the host populating that. So that tells us that this maybe does not fit with the MetaData suite, is it's own suite.

        Under that hypothesis, such suite would do the following work:

        - At description time indicate (hand-shake) support for ColorSpace suite

        - Pre-Render associate a particular ColorSpace collection by name to a clip (default clip property is RAW==NULL or hosts populate a Project preference?). There is only one clip property, color managed or not and there is ColorSuite passing back ClipHandle with the ColorSpace MetaData Collection set?

        - As for implementation, would an host populate the Effects Controls somehow with an option menu for input clips I said are color manageable?  Or it the effect responsible for that and then once we introduced color profiles and actual color spaces does this assume the effect can consult a "dictionary" of host supported ColorSpace/Profile Image MetaData Collections?  So then a need to walk through all ColorSpace available.

Does effect need to create parameters and slave them to a ColorSpace MetaData set?

         - If I create a color parameter and host provides a color picker as a bonus do I always get RGB values internally from my color parameter and are they transformed by host or we need the pixel level equivalent getPixel instead of getImage sort of thing? That is what value do I get if my input is linear and my output is sRGB.

*: Sidecar question as I never used that feature, is if the Parametric Parameter a place to store LUTs.?

*: Just discussing Image Colors, there might be all kinds of meta-data that are related to color, example someone might figure a nice way to forward to me exposure entry in an EXIF or something. But then it's not an Image conversion/transform, it's just a or like a parameter (often something you could capture and write as parameter in the effect). So it does not fit this-here.

Phil Barrett comments on 1 Feb 2017

Thanks for the comments! Unfortunately it looks like I won't be able to join the discussion this evening. But I'd like to respond to some of the comments that have been made so far.

Pierre says "colourspace is ambiguous". I tried to be unambigious. I shall try again. By "colour space" I specifically mean

a defined tone curve (explicitly or implicitly as two formulae to convert to and from linear)
and ideally three defined primary chromaticities (x,y)
and possibly a defined white point (x,y), e.g. D65

The tone curve definition encodes the black and white points by defining the mapping to/from linear 0.0 and 1.0. This includes legal-ranging if that is in effect, by squashing and offsetting.

The tone curve definition is independent of the bit depth of integer image formats, and expresses values scaled so that the maximum code value (255, 1023, 65535 etc) is treated as 1.0.

I specifically did not use the word "gamma" since that is just one way of defining one class of tone curves, and is often inaccurately used.

As an example of a complete definition, here is a Baselight colour space which has the ST 2084 (a.k.a. Dolby PQ) tone curve, Rec.2020 primaries and a D65 white point. We define it as

Tone curve formulae (ST 2084):

float convertToLinear(float val)
{
    float f = max(pow(abs(val), 1.0/78.84375), 0.8359375);
    return sign(val)*pow((f-0.8359375)/(18.8515625-18.6875*f),6.277394636);
}

float convertFromLinear(float val)
{
    float f = pow(abs(val), 1.0/6.277394636);
    return sign(val)*pow((18.8515625*f + 0.8359375)/(1.0 + 18.6875*f), 78.84375);
}

Primaries (Rec.2020): (0.708,0.292), (0.170,0.797), (0.131,0.046)
White point (D65): (0.3127,0.3290)

I am not suggesting that OFX need necessarily go this far, and I am certainly not suggesting that we define OFX constants for a large number of supported tone curves, but this should give an indication of the depth of information that is available from some hosts.

This proposal does not include any mention of colour correction descriptions (e.g. CDL, lift/gamma/gain etc), and I would strongly argue that it should not. The proposal is just about definining the colour space of image data passing through OFX plugins.

"Do we need to start this discussion by [asking] Is Linear the reference space of OFX?" - yes I think that would be a great starting question. Baselight assumes this and converts images to linear for OFX, but I would far rather this was made explicit, and/or plugins could have a way to request this behaviour.

"Host native LOG (e.g. FilmLight)" - Baselight is not "natively log". Baselight has a generalised colour space handling mechanism which means that images being processed can be in any defined colour space. That's why we have a deep understanding of this domain and why I have raised this proposal.

Defining the colour space of colour parameters is definitely an issue, and indeed it's something we're only now starting to address in our applications.

Defining the handling of the alpha channel in an RGBA image would be good. I suggest that the only sensible path is to say it is always linear with 0=transparent and 1=opaque.

Yes this needs to apply to OpenGL textures (and any future extensions to OpenCL, CUDA etc as well) as well as to CPU images.

Gary Oberbrunner comments on 1 Feb 2017

I'm no expert on this, but there are very well-defined ways to specify color space now, thanks to OpenColorIO profiles (for example). It seems to me we could piggyback on a standard open profile spec. Input clips would be tagged with their color profile, and the host would specify the desired output profile. Yes, this is very detailed, but anything less isn't really useful IMHO.

I'm agnostic about whether the spec should provide negotiation between host and plugin for color space. On one hand, if the plugin is colorspace-aware, it can (and should) just convert from/to the host's declared space. On the other hand, if the host is already converting from its format to OFX format, converting colorspace at the same time is more efficient. If the hosts convert, plugins are easier to write ;-) But performance (minimizing conversions) is important.

Pierre comments on 5 Feb 2017

Here's notes from last meeting -

-> Some “simple” color space enumeration clip properties could be immediately useful (Log-like [e.g., S-Log, ACEScc], Gamma-like [e.g., Rec.709, Rec.2020], linear 0-1 [needed?], linear HDR [e.g., ACES]).

-> Some “detailed” color space clip properties could be useful down the road (named gamut [e.g, “Rec.709”, “Rec.2020”, “ACEScc”), named curve [e.g., “Rec.709 gamma”, “HLG”, “PQ”]). Perhaps even primaries coordinates, optional white point. Hard to pass curve definition math though.

-> It would be useful for a plug-in to indicate which simple color spaces it can work in, and hosts converts if possible (e.g., Lift/Gamma/Gain color corrector prefers Log-like or Gamma-like curve and cannot work in Linear). Alternatively, this could be indicated in getClipImage like some other clip properties are. In either case, it is optional and the host does not have to do this conversion.

-> Can a plug-in change the color space of an output clip? Many hosts would not be able to deal with this; can we table for now?

(Pierre and Fabien add): there is also the topic of color picker - color parameter relation to that

Situation of discussion on 12 oct 2017

Still discussed on https://groups.google.com/forum/#!topic/ofxa-members/Gye7kLeHdhY

Solution unknown

[revisionfx]

PR #63 was created

[garyo]

This has been superseded by Issue #102