Color palettes

[justinlaughlin]

Refactor Palettes for better customization

This PR:

• Provides a more flexible implementation for color palettes, primarily to allow specification of palettes both at compile and run time.
• Compile time palettes are defined under lib/base_palettes.cpp and exported as BasePalettes. Some of these were generated by functions and are now just defined explicitly.
• Run time palettes can be specified using the command line argument -p <palette_filename>; an example palette file with the proper structure is under share/palettes.txt (this includes all of the palettes mentioned in the old version of the PR).
• Adds a few structs/classes
  • Struct RGBAf (to represent RGBA in [0,1] float format)
  • Struct Palette (holds vector<RGBAf> colors and string name)
  • Class PaletteRegistry (holds vector<shared_ptr<Palette>>)
• Updates PaletteState; previously it required the global variables Num_RGB_Palettes, RGB_Palette_Sizes, RGB_Palettes, and RGB_Palette_Names. Now it uses a pointer to a PaletteRegistry. By default, a PaletteState will be initialized with the global variable BasePalettes but it is not necessary.

Misc notes:

• Palettes can be loaded with explicit alpha channels now; right now this doesn't do anything. I think this would require some changes to PaletteState::GenerateAlphaTexture.
• Right now because BasePalettes is updated before any PaletteState are created, they initialize properly, but because textures are created by PaletteState::Init this could lead to problems if palettes are loaded in at a later point in the run. Moving the functionality to Palette, eg Palette::as_texture() with caching could be a solution. (there are several other methods that should be moved to Palette so that PaletteState is just in charge of the state).
• Palettes can now be retrieved by name but this isn't used anywhere yet.

Original PR

There is a nice resource from Fabio Crameri that provides a lot of color-vision deficiency friendly and perceptually-uniform color maps [see issue 268, and reference]. This PR adds most of those colormaps into GLVis.

Here is an example of batlow

I wrote a small script to help munge the colormap files into something copy-pasteable. I'm not sure if it's worth adding to the repo but I posted it as a gist. The colormaps added in this PR are listed in the gist:

cmnames = ['batlow', 'batlowW', 'batlowK', 'glasgow', 'lipari', 'navia', # continuous
           'oleron', 'bukavu', 'fes', # multi-sequential
           'hawaii', 'buda', 'imola', # discrete
           'oslo', 'nuuk', 'lajolla', 'bamako', 'davos', # categorical
           'bilbao', 'lapaz', 'acton', 'turku', 'tokyo',
           'broc', 'cork', 'vik', 'lisbon', 'tofino', 'berlin', # diverging
           ]

[justinlaughlin]

It looks like tests 15 and 17 are failing because they cycle color palettes backwards using P, so they get around to the new palettes. Maybe we should restrict palettes that cycle to a subset of the total? Or update tests?

[v-dobrev]

This seems like going a bit overboard with the palettes. 😁 The size of palettes.cpp is essentially doubled in terms of lines!

[justinlaughlin]

I started with just a few, but it was taking a while so I wrote a script to help, and then I figured might as well add most 😆. I think we had 43 before and this adds 28? Could pare it down too...

[justinlaughlin]

Some suggestions from gm:

• Add option for user to point to custom colormap file
• Keep "cyclable" colormaps the same for backwards compatability

Extra colormaps + user defined ones would still be accessed through F6?

[tzanio]

To clarify

Add option for user to point to custom colormap file

The idea will be to have an optional colormap file (could be configurable at build time, or just "colormaps.dat" in the same directory as the executable) and for glvis to check for that file at startup and if found to add the additional colormaps at the end of the default list.

This way, we preserve the default behavior, while allowing for local customization.

[tzanio]

@justinlaughlin, I think it will be still cool to add an optional input file for this. Happy to chat if you want to give it a try.

[tzanio]

Approved modulo the renaming suggestions I put in the comments.

We can also consider adding some additional palettes, e.g.

• The ColorCET perceptually uniform colour maps from here.
• The VisIt's colortables from here.
• The MatPlotLib colormaps, see here.

[najlkin]

I have a point for discussion here. Alpha channel is now supported by palettes technically, but except two special external palettes there is no way how to use this feature. So how about adding some option to add alpha channel to existing palettes? For example, there could be an extra option after pressing F6, which would add a simple gradient from 0 to 1 to your selected palette and you could optionally set the repetition/reversion in a similar way to the (non-alpha) color palette? 🤔 That would work nicely in most cases without the need to have special palettes for that 😉 .
Technically, we could have two classes for palettes with one base, where one would be for RGB only palettes and the second one for RGBA palettes instead of defaulting the alpha channel during loading. This extra option would be offered only for the former one.

[justinlaughlin]

I have a point for discussion here. Alpha channel is now supported by palettes technically, but except two special external palettes there is no way how to use this feature. So how about adding some option to add alpha channel to existing palettes? For example, there could be an extra option after pressing F6, which would add a simple gradient from 0 to 1 to your selected palette and you could optionally set the repetition/reversion in a similar way to the (non-alpha) color palette? 🤔 That would work nicely in most cases without the need to have special palettes for that 😉 . Technically, we could have two classes for palettes with one base, where one would be for RGB only palettes and the second one for RGBA palettes instead of defaulting the alpha channel during loading. This extra option would be offered only for the former one.

I like your suggestion - but I think it would make more sense as an extension of the existing "global" alpha modifier (ie the modifier changed byk, K, ,, <) rather than the palette alpha channel itself. Palette repetition/reversion is already a global state (represented in PaletteState; all Textures are regenerated when palette repetition/reversion is changed).

IMO the main use for custom alpha maps is to highlight certain parts of the data (kind of like more customizable level sets), especially when there is lots of occlusion eg in volumetric rendering. The NVIDIA IndeX plugin has some nice examples. But, in order to do that you need a more sophisticated alpha editor (paraview lets you define it as a piecewise linear function). I think what you want to highlight is often pretty specific to the data/palette.

TBH I think the global modifier is already good as-is; it could get kind of complicated if we add too many features (what if you reverse the palette, but don't want the alpha channel reversed too?). I think some useful directions for the future may be 1) more formats to load from (paraview exports palettes as a json iirc), 2) a piecewise editor, 3) support in pyglvis (it would be quite easy to modify your palette as a numpy array).

[najlkin]

Sure, some easier ways how to define palettes would be nice and may include that alphas 😉 . But I was thinking of something basic at the level of the default palettes, where you also cannot specify any levels you want highlight, but still most of the people are fine with that. Most of the palettes are linear, highlighting one end of the scale, so that would match the 0 to 1 gradient, or they have uniformly distributed levels, which can be achieved by repetition of the alpha gradient.
I agree that the global solution makes more sense, I just mean that I would not try to mix it with the alpha channel of the palette and only add that when it is absent, so the logic may stay relatively simple.

[justinlaughlin]

Approved modulo the renaming suggestions I put in the comments.

We can also consider adding some additional palettes, e.g.

• The ColorCET perceptually uniform colour maps from here.
• The VisIt's colortables from here.
• The MatPlotLib colormaps, see here.

I added the cet maps share/palettes-cet.txt.

For VisIT maybe in a future PR we should add a loader for their .ct specification? Looking at it briefly it looks like they allow for custom spacing of points and also different versions of the spec. If we need to parse it we might as well integrate that into GLVis.

I'd need to dig a bit more on matplotlib's license (they say it is a mixture of the licenses of each contributor?); they do have a nice list of externally maintained colormaps that maybe we can consider adding from in the future?

[justinlaughlin]

Technically, we could have two classes for palettes with one base, where one would be for RGB only palettes and the second one for RGBA palettes instead of defaulting the alpha channel during loading.

We could also use the check Palette::IsTranslucent() which checks if any alpha != 1.0.

Most of the palettes are linear, highlighting one end of the scale, so that would match the 0 to 1 gradient, or they have uniformly distributed levels, which can be achieved by repetition of the alpha gradient.

Now that I think about it more, I'd prefer a less context dependent solution. What if a user uploads a palette with no alpha but wants to add a linear mask? I think the issue is that F6 already asks for a lot of information at once as is; I'd personally prefer if we implemented something like #287 first, and then add this alpha adjustment idea as a command.

I suggest we separate this into a different PR. Many GLVis commands are already kind of confusing (IMO 🫣) and I think this would add more confusion. E.g. k, K sets one kind of alpha mask (I assumed it was uniform, but once I read the source I realized its a two-sided exponential curve (see plot); with a peak shifted by , and <). If F6 is (sometimes) used to set a linear mask (and another key else for uniform? something else for piecewise linear?) that is too many different keys + context to keep track of.

[najlkin]

Wow 😳 , I ...and probably all other users 😄 ... did not know it is complicated like that 🙃 . But when you look at that, it is actually close to what I was proposing with the linear gradient. So definitely I would not add another layer of logic to the alpha channel, but it is another reason to implement the logic in palettes to unify the whole thing. Also incorporating the settings to the F6 menu would give better control over it to the user. Anyway, we may defer it to another PR, this is good as it is 😉

[najlkin]

Almost there 👍

[najlkin]

I thought this will go to Texture, i.e., it will hold the actual OpenGL texture 🤔 . It would also save some memory, because texture_data are needed only temporarily to prepare the textures 😉 .

[justinlaughlin]

I also had this in mind but was holding off to wrap this up 😅.. but I agree with you that it would be cleaner. I'll make this change - some other things like global alpha texture (from PaletteState::GenerateAlphaTexture) will also need to be refactored similarly.

[justinlaughlin]

See 4d4be89

[najlkin]

Cool, a step in the good direction 👍 A few things:

• You forgot about first_init
• CI complains about ordering of inits in constructor
• Get rid of PaletteTexIds, textures can generate the ids by themselves. Not batched, but that does not matter much, I believe.
• The statics of Texture should be private. It uses only rgba_internal from the formats, alphas do not belong there. WIP, I guess. 🚧
• Methods SetCycle/Color should be private as they set the texture to a semi-invalid state without update of the size. I do not see any separate usage, so maybe you may merge them with UpdateTextureSize()? 🤔