Creating overlay maps in Voyager Story

[gjcope]

We've had support for rendering overlay maps for some time now, but looking to add overlay creation to the authoring features.

I've started this by merging in some existing 'paint' code here (https://github.com/Smithsonian/dpo-voyager/tree/dev-overlay). There is still quite a bit to do/clean up but looking for some feedback/discussion before moving forward:

This earlier implementation was based on the idea of 'target zones'. It's a relatively common use case on our end to create overlay maps that identify specific regions on an object with a solid color. The previous proof-of-concept then allowed you to tie state changes to clicking these 'zones'.

I think this implementation should be more general, as overlays can be used in many ways. We still need to be able to support this 'zone' concept though and previously it was tightly tied to creating the map (as you can see in the branch above) so I'm not sure where it fits in now.

@sdumetz

[gjcope]

https://github.com/Smithsonian/dpo-voyager/tree/dev-overlay is updated with a refactor to remove 'target' functionality and support creating multiple overlays per object. "Targets" will be reintroduced in the future ActionManager.

Questions/TODOs:

• How will we handle derivative quality change? Previous functionality resized the active overlay on change but I'm not sure that's what we want now.
• An indicator is needed for when overlays are "dirty" (need saving)
• Currently creating static names for overlays. Seems like a worthwhile tradeoff for dealing with renaming files and potential errors there, but introduces the possibility of a conflict/overwrite.
• Memory management could probably be improved and we might consider limiting the total number of overlays possible.

[gjcope]

Removed the support for automatic derivative quality generation/saving and only showing overlays for the active derivative quality. I think it was really helpful if that's how you intend to use the overlays, but I don't have the data to support that notion. This way it is simpler and there is still a path create multi-res maps if you need to.

[gjcope]

One other topic - previously in "paint" we supported drawing directly on the texture in a flat "unwrapped" window as well as on the model. The core support is still there, but there was never any demand for it so it would be great to get opinions on if it's worth clouding the UI with.

[sdumetz]

I’d add that support for unwrapped editing would probably be better served by an external editor anyway.
Power users will probably want to fine-tune their maps in some cases, but they will also probably need better compression and a complete set of tools, brushes, whatever.
Ensuring a download->edit->upload workflow is easy to set up might be a good (as in: easy to implement and maintain) fallback that ensure all edge cases can be efficiently tackled externally.

[gjcope]

Sounds good. I've removed that support.

[gjcope]

@sdumetz I think https://github.com/Smithsonian/dpo-voyager/tree/dev-overlay is at a good state to push to RC-45. It's not fully featured, but I think feature complete enough to provide functionality for users wanting to create and modify maps within Voyager Story. It can always be added to/improved. If you have a chance to take a look your feedback would be appreciated.

[sdumetz]

Hi,
I tested it a bit this morning and had some trouble.

1. On models that have none of the configured overlay quality levels (Low, Medium, High), saving the image does nothing.
2. On some models, the paint "spills" on vertices far from the cursor. Possibly has to do with bad UVs rather than the implementation itself? See this statue for an example
3. If model has no imageSize, the map is saved with a size of 0. Since the property is not marked as required in model.schema.json, and not set when creating a model by drag&drop, a default value should probably be used?

Beside those troubles that are admittedly mostly due to our scenes being of dubious quality, I really like where this is going.

[gjcope]

@sdumetz Thanks for taking a look! Were you possibly looking at an older commit in that branch? I think 1) and 3) should no longer be an issue as quality level is now only used for naming and imageSize is not used. We are pulling the size from the active diffuse map and defaulting to 4k if it doesn't have one.

For 2) yes definitely, this is bleed over UV seams as the painting algorithm is pretty simple to keep it performant. I definitely think there is room for improvement here.

[sdumetz]

Were you possibly looking at an older commit in that branch?
I was, sorry about that :/

I see no problem with having a degraded experience on non-optimized mesh at first and see if we can improve in further releases once we see actual usage patterns.

[sdumetz]

Now with an up-to-date branch.

I seem to have bumped into some kind of race condition (?) : When I load a scene with an overlay (not visible by default), it's availability is sometimes not registered. It shows neither in the Overlay tabs nor in Settings.

If I try to create an overlay again I get this error :

TypeError: Cannot read properties of undefined (reading 'fromFile')
    at CVOverlayTask.getTexture (CVOverlayTask.ts:141:20)

On a more personal note:

I don't like that the overlay is tied with the quality level. That's not how I would expect the paint to work from a user perspective. Derivatives are generally expected to be transparent design-wise: If I put an annotation, it's there on every Quality levels.

I know it makes a lot of sense from a 3D model perspective and am not event sure if you could generate a map for every derivative (we probably couldn't even detect if a model shares UV coordinates across derivatives or not) , but if it was possible that would greatly improve the experience.

The yellow brackets identifying the selected object are hidden in "paint" mode. It makes identifying the target object a bit hard in multi-models scenes. Maybe adding them back isn't the best option though and you can hide or make transparent everything except for the painted object?

[gjcope]

Do you have any more info on the bug or a test file? I've tried several (both created in and outside of Story) and refreshed many times but I'm not able to trigger the issue.

Re: quality levels: yes, that was one of my first questions/concerns in this thread. If you look at previous commits, we were generating maps for each L/M/H derivative on every save but that's very specific to our workflow. We know that those derivatives always share UVs. As you mentioned, that will not be true always. Also, I think there are some cases (like Thumb) where we would not want the overlay created. I agree that it's not optimal to have to recreate a map, but I'm not sure of a good solution here. It requires having a detailed knowledge of the model creation which I don't think all users will. Happy to take any suggestions on it though.

Putting the selection brackets back in is not a big issue. I think it is cleaner for painting without, but understand the concern.

[sdumetz]

we were generating maps for each L/M/H derivative on every save but that's very specific to our workflow.
When we make derivatives, we tend to only change the texture resolution and reuse the same mesh as we found geometry simplification to generally be painful and contribute marginally to size and performance gains. So that makes two cases where this would work. I never saw any multi-resolution user-generated model get uploaded so that's another thing we shouldn't worry too much about. Still I agree the hypothetical case where someone's model would have per-level UVs is to be taken care of.

Maybe we could use the default user-created overlay for every quality level, still allowing for a per-level overlay to be uploaded/created by the user?

On a realated topic, I'm really not fond of how the Overlay is currently tied to a model's quality level. The ability to select an overlay is tied with what's currently displayed on-screen, while the "settings panel" is tied to the whole model, including every derivatives. It's an inconsistency that I think will come back at us and bite very hard in the future. In particular that's going to be very problematic with my plan to propose a more dynamic use of LOD selection in multi-object scenes.

[gjcope]

We commonly have derivatives with different geometry resolutions (Thumb, AR) so I think it's a real use case. Using the same map for all levels means bad UVs in these cases and potentially excessive texture memory usage if the default is High, and the active derivative is lower.

On a realated topic, I'm really not fond of how the Overlay is currently tied to a model's quality level.
This is sort of inherent to how derivatives work, no? All of the assets for a model are tied to the model's quality level. A number of the attributes in the "settings panel" are shader settings, so independent of any assets/model. I suppose if we moved to a system that one conceptual overlay with a file per-derivative, it could be managed in that way but that quickly gets really complicated with the above use cases when you want to not have an overlay on a derivative or need to update a specific map for different UVs.

Interested to understand what issues this would bring up with your LOD system.

[sdumetz]

Something that switches more often between quality level is going to trigger those missing Overlays more often.

It's already a problem to have the overlay potentially disappear when a model's quality is changed (as discussed, we have no good solution for this right now), but additionally if the system switches back the model to it's original quality afterwards, fact we wanted the overlay (now available again) to be shown has been lost.

Once again, there is no obvious solution because we can't assume Overlays to be reusable across quality levels but I'd like the UI to properly reflect this need. For now if I want to add an overlay for every Q. level I have to:

1. go to Overlay tab, draw
2. Switch derivative in the Derivatives tab
3. Draw again
4. Repeat for every Q. level

So there is nothing in the Overlay tab to indicate me that, for example, I only have the "High" overlay for this model.

This is especially awkward because in the scene file overlays are not name-indexed but numerically assigned so xxx-overlaymap-high.jpg is in fact automagically connected to xxx-overlaymap-medium.jpg, etc... if they exist. It's just that the the Overlay tab won't let me know if they do and/or create them.

[gjcope]

Something that switches more often between quality level is going to trigger those missing Overlays more often.

Got it. I thought we were talking about a different concern, but this is fundamentally the same question about how to create maps for the quality levels.

I agree that somehow automatically generating these maps (in a way that fits the previously discussed criteria) would be ideal, but I'm not sure how missing data in that sense is much different from missing data when you are doing something like adding alternate language content.

I'm not sure I follow your last point. The file names indicate the quality level the map was created for. There should actually be no connection between the maps of different levels as it is currently implemented. To do that we would need a higher-level overlay concept that grouped maps as I previously mentioned. This seems like it would greatly complicate things.

I think this still needs more consideration. Another aspect of automatically generating multiple levels of quality is editing. What happens when you edit a medium or low res map? Does that change somehow propagate to other levels? I would say not likely. Which means you are back to having to edit each level anyways...