Provide access to graphics written to the screen

[nvaccessAuto]

Reported by afsmel on 2011-02-17 23:34
while NVDA PRODUCES SOUNDS TO LET USER IDENTIFY THE LOCATION OF THE POINTER ON A SCREEN, I FIND IT WILL EMPOWER THIS WAY OF DEALING TO LET NVDA PRODUCES ANOTHER SOUND TO INFORM THE USER THAT THE MOUSE IS IS ON A GRAPHIC NOW, SPECIALLY WHEN THE HEAD OF THE ARROW IS LOCATED IN THE CENTRE OF THE GRAPHIC SO THAT ANY CLICK WILL BE FUNCTIONABLE. FURTHER, NVDA MAY HELP WITH INFORMING THE USER WITH THE REQUIRED MOVEMENT, AND HOW IT IS TO BE DONE, THIS IS IN BEGINNER MODE ONLY, THIS WILL GIVE THE USER THE ABILITY TO DEAL WITH GRAPHICAL INTERFACE SOFTWARES, WITHOUT RELYING ON MSAA. PARTICULARLY, IF USER CAN ASSIGN CERTAIN TEXT TO THE GRAPHICS HE NEEDS TO DEAL WITH IN A SOFTWARE, AND STORE THEM TO A USER DEFINED FILE, FOR WHEN LOCATING THEM IN ANOTHER TIME, NVDA INFORMS THE USER WITH THE ASSIGNED TEXT TO THIS GRAPHIC, ALSO, USER MAY BE IN NEED TO STORE THIS FILE ON A STORAGE MEDIA TO INSTALL IT AFTER HAVING ANOTHER VERSION OF THE SOFTWARE, OR ON ANOTHER SYSTEM.

[nvaccessAuto]

Comment 1 by jteh on 2011-02-21 00:22
The scope of this ticket is too large, as it covers multiple major enhancements and doesn't clearly define them. I'm narrowing it to just cover providing access to graphics written to the screen, as this is needed before any of the other ideas can be considered.
Changes:
Changed title from "empowering use of mouse with nvda" to "Provide access to graphics written to the screen"

[LeonarddeR]

@jcsteh: Would it be technically possible to get access to graphics and shapes on the screen for programs which expose output accessible using display model / screen review?

[jcsteh]

@michaelDCurran is the display model expert, so he'd know better. Having said that, I believe it is possible to detect graphics. One problem is calculating a persistent identifier for them. Without that, you can't label them.

P4 because GDI is becoming less and less prevalent, but there are still legacy use cases that need this.

[michaelDCurran]

We could certainly track when an icon is drawn to the screen. We could expose either its file path, or some kind of hash of the data.

[LeonarddeR]

@michaelDCurran commented on 19 jul. 2017 04:13 CEST:

We could certainly track when an icon is drawn to the screen. We could expose either its file path, or some kind of hash of the data.

Would all of this require additional hacking into the C code, or is there already a Pythonic way here to get graphics info? I'm asking because at @BabbageCom, relying on display model info is still quite prevalent, and it would really help if we could at least identify images by hash or file path as you mentioned.

[Adriani90]

Any update on this issue? Does anybody start working on it?

[Adriani90]

This could be a great improvement for the golden cursor addon, at least this would allow for a good test environment before implementing something similar into the core.

cc: @josephsl, @Robert-J-H

[Robert-J-H]

I'm not sure that I understand the proposal.
There are Windows functions that can get the resource ID for icons, cursor and bitmaps. The file path is probably not very useful since a lot of resources are just packed into the executable. Admittedly, it would be advantageous to have at least the areas of the graphic clips and hashes could be done with e.g. md-5 checksums or whatever is fastest. Given the dynamic and animated nature of today's graphics, it won't probably be very reliable.

[cary-rowen]

There are many software developed in China that do not follow the barrier-free standards. MSAA is completely invalid. It would be very good if the proposal can be reached. I am looking forward to it.

[josephsl]

Hi,

I'm no longer maintaining Golden Cursor add-on, so this is beyond my area at this time (sorry).

[lijianwei2019]

Any update on this issue?

[cary-rowen]

If this issue is completed, it will undoubtedly be a great feat for software with unfriendly accessibility.

[Adriani90]

Software is nowadays updated more frequently than in the past in general, and changes can occur quite often in terms of grafical interface, UX etc. This is because new technologies evolve, screens got more diverse (e.g. touch screens, monitors etc.) and the software will have to adapt to these technical changes. Grafics drawn onto the screen change their positions, even sometimes their dimentions etc. I think having to struggle in NVDA with such changes (e.g. adjusting everytime the code to matchh the new design of the software etc.) is really difficult to maintain and very probable too sofisticated or even impossible for a blind person to rely on. A sighted person would have to regularly look into these changes of grafical interfaces and grafical designs and maintain NVDA's code accordingly. But maybe I am wrong.
I think there are following possibilities to handle legacy use cases which might still need a feature like this:

• Ask a local company to consult you and maintain a local mirror of NVDA adjusted for your use case, or developing an add-on
• Invest more effort and energy in pushing software developers to adapt to accessibility standards (this is what we do in Europe since years and it shows that pushing for accessibility in a structured and constructive way often results in regulations which force software developers to think about adjusting their products (see e.g. the EAA - european accessibility act, or the BSFG, a national law that will start next year in Germany).
• Request an add-on in the community which could be maintained by the community

I know it is not the most satisfactory option we have, but I think the NVDA core cannot maintain such a feature in the long term. So we should maybe invest rather time in other features that are also very important and should definitely be part of the core, e.g. ne navigation paterns such as by indentation etc.

I am converting this to a discussion in case you still have any arguments on how maintainability could be achieved. Please feel free to participate in the discussion.