mini ui library / util class

[uvwxy]

Add a utility class / libary / .. to collect UI related draw utils

• labels for buttons
• centered time string
• date formatting
• list view / settings view (an object oriented structure with callbacks to add menu elements and then react to toggle states or button presses on the element)

[uvwxy]

The first implementation should be inside https://github.com/Open-Smartwatch/lib-open-smartwatch

Moving it to a separate repo can be discussed/done later.

[max9901]

Shall we add these utilities directly into the "Graphics2DPrint" class ? doesn't seem like a lot of extra features.

And do want automatic centering on the whole watch or do we want to center around the position of the cursor only which makes it very easy to get the center of the screen anyway?

[uvwxy]

If there are print String related operations, they should go into the Graphics2DPrint class.
If it's UI components a separate object oriented structure is better.

Centered around the cursor. This way we can place the cursor, and then select the alignment (left, right (default), centered).

[simonmicro]

Are there any class hierarchy ideas already? As far as I see there is only the Graphics2DPrint, which just extends the ArduinoGraphics2DCanvas by the text functionalities? So what about some kind of (button, text (size, positioning), display (for relatively positioning these objects)) classes? Especially to prevent pixel-based positioning to make i18n more fluent...

[max9901]

As far as i know not! if u have ideas i'm happy to help implement them!

[simonmicro]

Okay, before we dive further into the implementation lets establish some ground "rules"/concepts I would like to follow:

• The UI & layouts should be only configured on loading the app or starting the system, during rendering should be no support to move elements. This way we can make many things more static and do not need to recalculate the alignment during rendering very often.
• I would like to see as many relative positioning as possible (so we position elements via 80% of width and 20% of height) -> no pixels anymore. This ensures we are easier portable later on (and also we can resolve these alignments during adding the elements and then could forget them).
• I've noted we all love to place labels / icons near the physical buttons, so how about we introduce some constants to better fit their positions (and therefore use the same everywhere)?
• Icons. Stopwatch UI #20 already displays some possible designs (and hey, I think they look very good!) - so how about also introducing some kind of icons lib (I mean self-written, more like Icon flag(Icon::FLAG) type, which then implements the needed draw calls)?
• No-one should be able to change the positions of the elements after their registration at their Layer, so we could e.g. implement partial updates (?) later on... Anyways they should still be able to report their sizes (in pixels) during runtime and also change them (e.g. texts changes and now needs more/lesser space)...

Therefore I propose the following class hierarchy:
Untitled Diagram.drawio.zip

Any drawable should also get some sort of alignment, so we could set its position e.g. to center (50%, 50%) and it would be centered, as the cursor specifies the center of the screen AND the e.g. image. So maybe an other enum, which addresses stuff like top-left, top-center, top-right, usw.?

A stopwatch would therefore create on its loading a new layer, with all the icons usw. and would have to allow the main.cpp to access it via a getter. The main would then go over all layers (background, app, notifications, status icons) and render them after each other.

What do you guys think?

Yay, a wall of text! And yes, "Dear Imgui" is out there, but it is way more dynamic and therfore needs way more resources during runtime - maybe we could orient ourselfs on their work?

[max9901]

I like this idea and i think it could be beneficial. I do however not really get how u propose to build it.

Also a concern on the memory foodprint. How would u propose to store all the layer information ? It is the drawables std:array right? What is the amount of memory a simple "hello world" string would cost as drawable ?

Also main.cpp needs to preform the render which means connecting the "layers" to the gfx_2d library ? or where does the gfx lib fit into the picture?

(~~somewhat unexperience programmer, especially in the multiply person project area. (I do however also have a research job that requires me to learn it. so im motivated to learn!))

[simonmicro]

I do however not really get how u propose to build it.

Ouch - give me a hint where I lost you 😀 . I plan to use as much compile-time optimization as possible. So e.g. the icons draw() just has a long if-else chain going through all the enum possibilities, so no need to have a separate thingy for the icons. The ui elements could be part of the apps class instances itself, so they are created on load (new) and cleaned up on switching (delete). I know this could introduce memory fragmentation, but I really hope we do not allocate anything during running those apps, beside stuff on the stack...

It is the drawables std:array right

Hmm, the Layer could use std::arrays, they could also just use plain-old c arrays. I just wanted to stretch, that we do not need abstract datatypes (like a vector) there. The layers size should be known at its creation (std::array enforces that, as it is compiled using exactly its template-size) and can't be changed later anyways. This should further prevent any kind of memory fragmentation.

What is the amount of memory a simple "hello world" string would cost as drawable ?

Hmm, such string does not need to be stored inside the drawable - as it is constant anyways the Text would only need to take a const char* ptr anyways. When I think about it: Any Text would need two unsigned chars shorts (for the pixels), one bool (needs redraw) and an unsigned charfor the text size. Maybe one moreunsigned char` for the alignment enum. With the pointer to the text overall 2+2+1+1+2(?) 7 to 8 bytes overall size. Should be good enough?

Also main.cpp needs to preform the render which means connecting the "layers" to the gfx_2d library ? or where does the gfx lib fit into the picture?

Everywhere? I do not propose to replace it! Writing the text, setting the cursor and drawing rectangles are still its job. My idea more tries to encapsulate the drawing order (layers) into their own objects. Also the cursor placement is just encapsulated inside the respective drawable, so the dev could just type place a text at 23% width, 24% height and do not need to place the cursor correctly depending on the text size, alignment and the remaining space on the screen. I have to admit that I do not really have an idea what exact features the gfx lib provides (except these fundamental ones). As far as I know the gfx lib does not provide layer support, but I would like to add one, as this simplifies removing and replacing whole parts of the rendering-stack. Want to show a notification? Enable the layer above the app and update the texts! Want to show a little loading-icon above the app? Just swap the respective layer.

For example - how does the main work: It goes trough every layer and calls their draw(). Those call the draw() on the elements and finally they call the gfx functions to draw the needed content.

Also this all should allow us to utilize the two cpu cores we have here (later on). One running the ui render-pipeline when needed and the other one could start taking over background tasks (like managing wifi/ble, running services and therefore updating its own "loading"-layer).

(~~somewhat unexperience programmer, especially in the multiply person project area. (I do however also have a research job that requires me to learn it. so im motivated to learn!))

Okay 🤓 - I do work on several projects and have also a job as software developer, BUT this does not mean my ideas are perfect nor appropriate (I do have to admit that I tend to over-engineer stuff)! I do have a little bit experience working in a team, but I'm also always happy to learn. When anyone has any questions or improvements - just comment them!

@max9901 Also when you can speak english, we could chat about all this over on @uvwxy Discord! This way we both know our abilities and can better delegate work. E.g. for #54 @uvwxy works on the ui & server, while I thinker with the compiler, as he has more experience with his own creation, while I wanted to learn more about #defines. Later on we come together and merge our work together to finalize the feature.

tl;dr Explanations what I mean and invite for suggestions!

[uvwxy]

I'm working on a basic UI helper class right now. let's discuss on discord where we can go from there. :) 👍

[xasz]

Any glimbse on this :) ?

[simonmicro]

Currently I / we work on the finalization of #54, then I plan to work on #77. @uvwxy will work on other stuff (two services). I then plan to look further into this (I consider this here as a early-stage-draft)... So all the progress is currently the osw_ui.cpp - you may want to extend that one or look there...

[xasz]

• The UI & layouts should be only configured on loading the app or starting the system, during rendering should be no support to move elements. This way we can make many things more static and do not need to recalculate the alignment during rendering very often.

I am not sure but i think there will be needs for building Layout on Runtime.
For example if there i am thinking about alot of Apps where an external API/Rest-Service could define the layout.

I do not see much in the osw_ui.cpp
Is there any implementations for ur Diagram above?

[uvwxy]

I really like the proposed sprite system of #90

I think we should implement possible widges in the same way

[xasz]

If i did understand it correctly it is quite the same. As i have not found any code, i did start some really really basic test implementation. If u call it Sprite or Drawable - Thats a discussion worth :)
I have a Drawable / Label / ListView and Plan on having some more Elements Like Button / Icon / and i can implement a "Sprite". But my priorioty is on really simple to build ui/menus but i think it is kind of the same at that level.
#90 If u plan to sprite system like for a 2d game i would displace this from the ui system which most game engines does aswell.
the ui just does not have that much "unused" rendered sprite live a tile based level in a game could have features like simple frustum cullum. But i think the additional overhead in such things compared to the potential unessecary drawn things are not worth it on ui layer.

If anyone want to look at some testing code just have a look at my UIAstraction branch in /src/ui and apps/tools/ui_test. I have a semi working Drawable/ListView/Label.. more will come if u are interested.

[SeanReg]

My recommendation would be to use Sprites as the base class for any UI components. UI Components like Button/Icon/etc would/should have much of the same basic properties as a sprite (position, angle, size, layers, etc) and build on top of it via inheritance.

With the Sprite system I also want to add the concept of a Group. Which holds multiple Sprites all offset from a common origin. The group can then be translated/rotated/scaled all as a single group. I think this would be useful for UI components as well.

[xasz]

My recommendation would be the "same" but not call it sprite. In my world a sprite is a 2D Graphic (for example for games). My superclass would be a Drawable as @simonmicro described it and a Sprite is a derviced class which could hold a graphic or animation. I am planning on different Layout components, which could be Grid, VerticalLayout, HorizontalLayout. I have not planned any rotation support for now but surely offset etc if drawables are nested, which u can already see in the ListView i provided. There the nested Items position get controlled by the Listview and the local coords are ignored.

I am on planning a UI utility which works really fast on menu/settings/input creation. In my opinion this is a Ui class and sure it can hold Sprite/Icon/Image, call it however you like. A sprite is an image/bitmap and nothing more and should not be a superclass. The base class which holds position/size etc should be something really generic which can than be implement by a sprite, textfield, label, button.

Edit:
What is your goal @SeanReg . A "GameEngine" or a "Ui Component Builder".
I think if your goal is a "game/graphic engine" we should do seperate things even if they are somewhat the same, the features we need in the far end goal are really different. I think there is a reason why unity/libgdx/.. and others are do seperate classes on UI and graphics.

[simonmicro]

Hey, nice seeing you guys working on this!

But I would like to add an other thought to the discussion: Please be more static!
Our RAM (Heap&Stack) is not big - especially when you create on-the-fly ui elements. When you do this (and therefore not on the stack) we will run into problems with heap fragmentation. For example: A MQTT app gets an answer and displays some new strings - that's fine, but then it makes a call into some deeper HAL-stuff and this (for it self) has to create some permanent allocations on the heap. When the app is now unloaded it will cause tiny holes in our heap space! Imagine now multiple apps are doing this and we've got a problem! Also we have memory problems currently: When you enable some BLE features we have to disable any framebuffer rendering - I hope we can further reduce our memory needs, so at some point in the future this is should not be necessary anymore.

When you think it is absolutely needed to add dynamical add/remove stuff into the renderer - go for it! But please think twice before proceeding. After all an e.g. MQTT app will know all its label positions during starting and could just show/hide them collectively using e.g. a Layer. This way everything stays on the stack and we should be fine ™️ ...

[xasz]

Thats a good thought. Tbh i am "new" to c++ and this level of saving perfomance.
I am not sure what u mean by be more static. I think it is in the responsibilty of the one who actually "implement" the menu, if he setup all the layout and fixed elements in the setup() and then all is fixed or if he will update it in the lifecycle of the app.
Could you give me an example of what would be bad and what would be static :)
Thank you.

[simonmicro]

NP! 😀

A "static" implementation would require you to specify during compile-time what the layers size is, also you would fill such a layer with its constructor and then never be able to "remove"/"add" anything without destroying it.

For example - this could it look like:

Text t1("wowo", 0.5, 0.5, otherParameters, ...);
Text t2("wowo", 0.5, 0.5, otherParameters, ...);
Layer<2>(t1, t2);

Note that we have not used anything on the heap - the example-texts are only on the stack (as we are not using any pointers). The Layer itself gets compiled with enough space to hold two drawable pointers (maybe use std::array under the hood), therefore also gets resolved during compile time and lifes on the stack.

A more dynamic implementation would be:

Layer l;
Text t1("wowo", 0.5, 0.5, otherParameters, ...);
Text t2("wowo", 0.5, 0.5, otherParameters, ...);
l.add(t1);
l.add(t2);

Note there is no limit how many texts the layer could hold and would therefore need to use something like a std::vector and lifes on the heap! With the "static" way the dev, which would use it later on, would be forced from the beginning to know how everything (could) look like and will us reduce headaches later on (like, when I open app A before app B, it crashes with "out of memory" - but app A does clean everything up properly and the free heap space is enough - where is the problem?).

Basically on our low-end hardware I would encourage you to use the heap as less as possible. It is convenient, but it tends to get fastly out of hand.

Ofc. there is a good chance that I overthink this - feel free to point this out too!

[xasz]

Thanks for the little excurse :) I am on your side and get your point.
How big the problem would be to use a dynamic implementation i really can't tell - really don't know :) But the advantages on the usability would be big. But i don't see any problems to have both world, at least for some point.
If we give both options. There really is not any Point to have a "StaticLabel" and a "DynamicLabel" and a for e.g. DynamicVerticalLayout and a FixedVerticalLayout Implementations, which are compatible.

[SeanReg]

I am on planning a UI utility which works really fast on menu/settings/input creation. In my opinion this is a Ui class and sure it can hold Sprite/Icon/Image, call it however you like. A sprite is an image/bitmap and nothing more and should not be a superclass. The base class which holds position/size etc should be something really generic which can than be implement by a sprite, textfield, label, button.

Edit:
What is your goal @SeanReg . A "GameEngine" or a "Ui Component Builder".
I think if your goal is a "game/graphic engine" we should do seperate things even if they are somewhat the same, the features we need in the far end goal are really different. I think there is a reason why unity/libgdx/.. and others are do seperate classes on UI and graphics.

You seem to be confusing concepts here... A Sprite is just a bitmap, you're correct. And UI elements on an OS are made with bitmaps... Just because game engines "have Sprites" does not mean that an Operating System's UI isn't backed by bitmaps (it is). In Unity, all UI components and Sprites are Behaviors which contain transforms, they are therefore derived from the same concept. In LibGDX Sprites and Widgets aren't exactly the same because UI elements are DOM based. They are rendered completely differently. At the end of the day, all Sprites and UI have positions, angles, layers, etc and ARE bitmaps (all raster images are). Therefore they should come from the same base class even if that base class is not called "Sprite".

But I would like to add an other thought to the discussion: Please be more static!
Our RAM (Heap&Stack) is not big - especially when you create on-the-fly ui elements. When you do this (and therefore not on the stack) we will run into problems with heap fragmentation.

I agree, this is a big issue. However, the Stack size isn't that big (I think 8k). So we need to also be careful about what goes on the stack too. My current thought is that larger resources that are contained within an app should mostly be heap bound. That way when the app starts it can alloc the resources and when it stops it can free the resources. Hopefully nothing gets created externally while the app is running. This should (in theory) mean that the heap, in terms of fragmentation, would look the same before app start() and after app stop(). Thoughts on this?

[xasz]

@SeanReg I think we are talking about the same. And sure, the UI Component in the and is a bitmap, but in the end i think we are justing using other terms and i just would not name the superclass sprite, we can if others think we should :)
For my app what i am planning to make i need some simple to use ui/text/label/slider/buttons, that is the reason why i am in this issue ;) Because i looked if there is some. Aslong as there is no other/better implementation i will keep on working on mine.
But i appreciate any help/tipps on how to add better perfomance or what to chance and i will try ;)

[simonmicro]

hat big (I think 8k). So we

Good point, but do you have further references on that? As far as I know we can use with our stack as much as we want (will be subtracted from the heap) and any stack size limitations for the esp32 are caused by yourself, when you create any xTaskCreate*. I'm just unable to find any references to this limit right now.

This should (in theory) mean that the heap, in terms of fragmentation, would look the same before app start() and after app stop(). Thoughts on this?

Currently yes? But I would not trust our future selfs to keep that in mind. When I'm thinking about this... What's about the framebuffer management? It should be on the heap and is therefore theoretically able to change over time - I recall @uvwxy saying something about tile based rendering of it? I guess I'll create a service for logging heap sizes asap, so I can monitor the systems usage over time...

[SeanReg]

As far as I know we can use with our stack as much as we want (will be subtracted from the heap) and any stack size limitations for the esp32 are caused by yourself, when you create any xTaskCreate*. I'm just unable to find any references to this limit right now.

Yes, so this is a little tricky because of the arduino-esp32 framework which we use. When the device boots, a Task is created (loopTask) which the start()/stop()/loop() run in the context of. That task has a fixed stack size:

https://github.com/espressif/arduino-esp32/blob/master/cores/esp32/main.cpp

However, I agree the precomputed static resources should just come from global memory and not from the context of any Task.

Currently yes? But I would not trust our future selfs to keep that in mind.

Eh yeah, can't disagree with that one.

[xasz]

@simonmicro If u have the power to create a service which can monitor heap sizes and idk if possible fragmentation, i could use that to run some tests when i finsh some more complex elements. I am trying to keep thinks on the stack as much as possible without the for now, at least for me, needed dynamic elements or think about it if i can implement them in two ways, like static stack and dynamic heap version which than can be used if really needed.

But is it really a problem? As far as i understood if we go in deepsleep the device really shuts down and boots again. Then the heap is rebuild on a new bases (if i understand correctly) and there will be some app switches, but not like running for days.
i do not want to says that we should not think of it and let the bad code happen, but just not to overthink it.

[simonmicro]

@simonmicro If u have the power to create a service which can monitor heap sizes and idk if possible fragmentation, i could use that to run some tests when i finsh some more complex elements.

Fantastic timing! I've just implemented a new service structure with exactly such a subservice last night (this is WIP, so make sure to not pull any newer versions, otherwise you'll may get the new wifi service :P ): https://github.com/Simonmicro/open-smartwatch-os/tree/feature/wifiService

As far as i understood if we go in deepsleep the device really shuts down

Yes, but have you noticed the wake up times recently? @uvwxy is currently working on a light sleep version - then we will keep our memory and can directly resume our work after the interrupt!

[xasz]

tbh. did not use the watch the last days because of the bad wake up times :) will need to try it again.
I hope i got some time this week and want to try to "finish" a first implementation with a layout/button/label/slider. When i am ready i would appreciate to have a review a a discussiona bout the code quality/perfomance and if we can build on it or not :)

[xasz]

I have build a small UI Class/Concept. For now there is a List/Button/Label/Slider:
Instragram: https://www.instagram.com/p/CPTD-h_BBTP/?utm_source=ig_web_copy_link
App-Code: https://github.com/xasz/open-smartwatch-os/blob/UiAbstraction/src/apps/tools/ui_test.cpp

Visuals need some work, but there are some little helpers for width and height for child elements.
I plan on add a staticList or other layouts. But i think before i spend more time on it i would like to know, whats your opinions on the concept and if @uvwxy want to add this to the osw. I will spend more time on doc and helpermethods if we want this concept in the osw and not only for my app needs :)