Pointer confinement support (opt-in) #17169
Conversation
New environment set call to enable a bit more sensible handling of absolute pointing devices (pointer and lightgun). With the confinement enabled, pointing devices will not return neither -0x8000 nor (0,0), which was anyway dependent on the input driver, instead they will stay at the extreme edge.
|
This was already merged before I saw it but here's my response. As a core developer, while this would solve my issues I have with RetroArch, I really don't think this is an issue in libretro that needs to be fixed there. Introducing a new environment call to work around a bug in RetroArch seems like a Band-Aid that only pollutes libretro with unnecessary environment calls. The buggy behavior of RetroArch that this new environment call fixes is not documented, until now it has not been described by libretro.h. So unless another frontend were to copy the buggy code in question, it very well might not have the same issue. So, do we know of any other frontend that has copied this buggy behavior of RetroArch? The only frontend I know that supports mouse inputs is RALibretro which does not at all have this buggy behavior. It already always clamps the coordinates reported to the core to -0x7fff,0x7fff and doesn't treat 0,0 as anything but the cursor being perfectly in the screen center.
I have looked through search result of RETRO_DEVICE_POINTER in the libretro organization to try to figure out which cores might be affected by this. Of course this isn't every libretro core in existence but it covers a large amount of them. As far as I can tell no core relies in 0,0 being off screen. A few cores just ignore it but as far as I can tell this is just to work around the bug in RetroArch. Here's my finding on cores that just ignore 0,0:
Here's two cores that look suspiciously like they would treat 0,0 special but carefully reading through the code reveals it's not really doing that: While the code parts look different, I'm pretty sure they are based on the same code at some point. They both have this suspicious code part: // Handle offscreen by checking corrected x and y values
if ( gun_x == 0 || gun_y == 0 )But looking at the calculation of So as far as I can tell the issues are purely in RetroArch and in its many input device drivers. The effort should be spent in unifying the behaviors of both
I think a good idea would be to write down the behavior of each input driver in regards to both |
I use behavior a bit with melonDS DS, but I'm fine with rethinking that assumption. Especially since the screen resolution is unlikely to be large enough for a player to accidentally hit that exact position...and the touch screen isn't likely to even contain this point, anyway. |
|
@JesseTG Aren't you emulating a touch screen though? Shouldn't an emulated touch screen only think it is touched while the mouse button is down or while a real touch screen is being touched? The issue at hand should only affect mice (and lightguns) that can hover on and off screen. If RetroArch reports a mouse button being held down (or a finger being touched) the pointer is always on screen with coordinates inside the screen. RetroArch doesn't report a mouse or finger being pressed to the core while off screen. Looking through your code I think this isn't affecting the actual emulation but the rendering of an on-screen cursor, right? I personally think it's fine if the cursor is still drawn while the real (host) mouse cursor is off one side of the screen. It would even still report to mouse clicks while pressed off-screen so this would make it easier to tap on things near the edge, just like it would make RTS style games in a DOS core much more playable. |
Sure...except there are games that expect you to hold down the stylus at the edge of the screen. I've gotten tickets about this before.
Sure, but then how would I tell where the "stylus" is being held? |
It sounds like you explicitly want to know whether the mouse is in the black area around the screen in RetroArch. Of course that black area around the screen does not exist if the viewport (windowed or fullscreen) matches the core's video aspect ratio (depending on scaling options in RetroArch). Maybe for that introducing a new Can you confirm though that if RetroArch were to stop reporting an off screen mouse position as 0,0 (and clamp to -0x7fff to +0x7fff), the emulation of melonDS DS would not break, it would just continue to draw an on-screen mouse cursor at the edge while the real mouse is off the edge. Also it would report touches at that edge when the mouse is clicked while in the black area around the screen. |
|
Aw. Should have remembered to mark it as draft. (Code was/is fine from my part, but there may be some different conclusion than my original solution.)
Umm... yeah. Not the cleanest approach. But, as you mention, so far these edge cases were undocumented and a bit random at places. Introducing an unconditional change in the frontend behavior can make things better, but also worse, if something was overlooked and some cores will start acting weird. As RetroArch does not have a huge automated test machinery for detecting regressions (only users who can get annoyed), my preference would still be for the API extension. But if the final decision is to revert this PR and just have it as a new default, then so be it.
Regarding input drivers, I scanned through them and there are only a few exceptions, Switch does its own thing (I have not looked into that in detail), Vita and WiiU do not seem to have the translation, others will use I am not familiar with ScummVM, but my guess would be that it would prefer to use the relative mouse device, and only fall back to the absolute pointer when it must, and on WiiU there is no support for relative mouse, while on Switch it looks as if it only refers to physical mouse, touch input is not converted. |
|
Ok, I'm not too upset if this stays merged. As mentioned, it will solve my issues with RetroArch as a core developer and it is something that can be done today as opposed to doing the investigation which might just take time and lead to nothing. So in the end maybe this is the most realistic solution with how libretro is tied to the reference frontend RetroArch.
Did you look at Android? When I tried it (just by logging the data returned for Another thing that needs to be checked and made sure is that pressing things on the overlay in RetroArch (the on-screen buttons that are enabled by default on mobile phones but can optionally be enabled on any platform) will never be reported to the core. Ideally mis-presses near the overlay should not be reported to the core, unless the overlay is directly over the screen. So testing and making sure the behavior of these scenarios:
|
|
I'll revert it for now so we can explore a solution that both of you can agree with |
This reverts commit 90ee413.
Not just whether. Where. I can map such input to the edge of the touch screen without losing information (e.g. direction) |
|
Now I'm lost 😅 Nothing that is implemented now or in the past, or has been suggested to be implemented, ever gives you the information exactly where off the screen the mouse is. Of course you always know where the mouse is over the core's video rectangle by polling X/Y and checking if those are in a valid range. If we were to change that the current behavior (which returns 0,0 as soon as the mouse leaves any edge of the core's video rectangle) to return the coordinate clamped -0x7fff to +0x7fff, and add a |
|
As I understand clamping should make it possible to map the pointer position to the edges, i.e. no contradiction with what is needed for MelonDS DS. Any recommended content for checking behavior in actual games? |
The melonDS DS ticket I linked to (posted by @DarkSide1305) contains two brief videos. This works the way OP expected it to. I think the recorded behavior uses This second one is what happens now in melonDS DS, using Do these videos help you understand the problem I'm facing? Or would you still like me to suggest a game for you? |
|
Oh boy, more variables :) 2 questions:
|
|
The Decks Touchpads just operate the OS Mouse |
|
As for your second question...
That's right. From RetroArch's perspective, nothing unusual is going on. All the linear algebra (screen layouts, cursor coordinates, touch screen coordinates...) is happening inside the core. |
|
After some investigation... here's how the input drivers currently treat mouse, pointer, and lightgun. Input drivers missing from the list do not handle either one. Mouse can have a maximum of 5 buttons and 2 scroll wheels,
|
|
@zoltanvb Wow, nice work! Which commit or release of RetroArch did you check this on? |
|
On kind of current code, slightly newer than this PR. |
|
Still thinking about the conclusion, but looking at some specific points:
This is actually what happens if the result of the current viewport translation is just passed on to the client. It's just that most desktop drivers will do a check and return 0,0 instead.
Some input drivers support that, but it is giving me itches seeing it in the pointer query case branches. We could at least introduce Also, if offscreen is true, then x/y could still be clamped, and not set to -0x8000 as it carries no more information. At least with pointer, since for lightgun it is supposed to return -0x8000 (but actually doesn't, with the exception of the overlay lightgun), so maybe that could also be unified in the API. On a slightly related note, going fully out-of-bounds is not always possible, so the core shouldn't really rely on that.
I have found only 1 that does not (wiiu), and udev has its own logic for the same, but everywhere else that function is used well.
For this one, I can not easily resolve the conflict between not reporting touches outside the viewport, vs. allowing the user to do the touch slightly outside (where there may be overlay elements, etc.). |
Maybe this is where the environment call from this PR would've helped? Or one that's similar. |
Description
New environment set call to enable a bit more sensible handling of absolute pointing devices (pointer and lightgun).
There was a discussion on Discord a while ago, which concluded that it would be better if the returned coordinates would be clamped to
(-0x7fff,0x7fff), and not report(0,0)(or-0x8000, in some cases) as an indication of being outside the active content. The most easily understandable example is RTS games in DOS, where scrolling is done by moving the mouse to the edge of the screen, and that is tricky with absolute pointers. Another improvement may be offscreen detection in lightgun mode.Since there are some cores that may rely on (0,0), the new behavior was made optional behind a new environment API call, with the actual state being stored in runloop (as opposed to a configuration item).
Remote Retropad was updated with a debug switch that enables turning this option on. Behaves as expected, running the content in a 16:9 monitor and moving the pointer to the black padding bars, cursor stays stuck to the edge if the option is enabled, but jumps back to 0,0 if not enabled.
The change in video_driver.c should cover the issue without having to touch all input drivers. I tested with wayland and udev.
Reviewers
@schellingb