Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(input/linux): add support for more virtual input devices #2606

Merged
merged 39 commits into from
Jun 16, 2024

Conversation

Hazer
Copy link
Member

@Hazer Hazer commented May 29, 2024

Description

This is a direct continuation of #2315 @ABeltramo 's amazing work.

  • Rebased to master (f0a00ae)
  • Adopted new vue components
  • Broken the inputtino.cpp into namespaces by kind
  • Renamed input.cpp to legacy_input.cpp

I had to rename the branch for CI to work, this is a continuation of #2261

Added support for the followings new virtual devices:

  • Nintendo Switch pro joypad

  • DualSense (PS5) joypad, including:

    • Gyroscope
    • Accelerometer
    • Touchpad
    • LED
    • Battery status
    • Force feedback? (is this supported in Moonlight already? I have to check..)
    • Microphone and Headset jack? (we should be able to plug into that as well, to be checked..)

And generally completely refactored all virtual input code using inputtino as a library

Requirements

The DualSense implementation requires access to /dev/uhid, (more on this later..) this is easily done by setting the right permissions, similar to how we deal with udev already:

First we’ll add our user to the input group:

sudo usermod -a -G input $USER

Then we add an udev rule to allow access to /dev/uhid:

echo 'KERNEL=="uhid", GROUP="input", MODE="0660"' | sudo tee /etc/udev/rules.d/99-uhid.rules

Finally, we have to make sure that the uhid kernel module is loaded at boot:

echo "uhid" | sudo tee /etc/modules-load.d/uhid.conf

Code details

The main file is src/platform/linux/inputtino.cpp I've kept the original input.cpp so that we can easily compare the two implementations and possibly add what's missing but the idea would be to completely replace that implementation with this. To switch from one implementation to the other you can just set the Cmake variable -DSUNSHINE_USE_INPUTTINO=OFF.

Unfortunately we have to deal with uhid in order to properly emulate gyro, acceleration and touchpad; there's a bit of a rationale into why this is not possible with udev here. This might be a deal breaker for you guys but I hope it's not; if that's the case, we can still emulate a PS5 joypad using udev without supporting gyro and acceleration.

I think bringing in uhid has also other benefits, since it sits at a lower level than uinput, we can really emulate any kind of usb device and possibly even completely integrate usb over IP if this will ever be considered as an addition to the protocol.

As for the code itself, Inputtino has got unit tests using libinput for mouse, keyboard, pen and touchscreen and SDL2 for testing joypads. These aren't complete (yet) but should cover most of the codebase.

Future plans

  • I'd like to give it a go at implementing gyro and acceleration support for a Nintendo joypad too.

  • Ideally, since we are emulating the real joystick USB messages, this should also be a good foundation for a Windows virtual driver implementation. I know nothing about this but I'd like to do more research on it and possibly add a multiplatform implementation to inputtino.

    • I know of the EV license and all the requirements, for some reason it's hard to convince companies to invest in this without a functioning prototype first 😅
  • Add support for trackpads in Moonlight? Generally it seems to not be the same as a touchscreen: it moves relatively instead of absolutely and has got different gestures.

What's missing?

Most notably the XTest fallback that is present in input.cpp if this is something that is used we can easily add it here as well. Everything else should be implemented, please let me know if something is missing or misbeaving.

I'm probably forgetting something else..

Issues Fixed or Closed

Might help with #1720

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Dependency update (updates to dependencies)
  • Documentation update (changes to documentation)
  • Repository update (changes to repository files, e.g. .github/...)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated the in code docstring/documentation-blocks for new or existing methods/components

Branch Updates

LizardByte requires that branches be up-to-date before merging. This means that after any PR is merged, this branch
must be updated before it can be merged. You must also
Allow edits from maintainers.

  • I want maintainers to keep my branch updated

third-party/inputtino Outdated Show resolved Hide resolved
src_assets/common/assets/web/configs/tabs/Inputs.vue Outdated Show resolved Hide resolved
cmake/compile_definitions/linux.cmake Show resolved Hide resolved
docs/source/about/setup.rst Outdated Show resolved Hide resolved
docs/source/about/setup.rst Outdated Show resolved Hide resolved
Copy link

codecov bot commented May 29, 2024

Codecov Report

Attention: Patch coverage is 3.57143% with 378 lines in your changes missing coverage. Please review.

Project coverage is 8.70%. Comparing base (5f1a17f) to head (920cca6).
Report is 152 commits behind head on master.

Files with missing lines Patch % Lines
src/platform/linux/input/inputtino_gamepad.cpp 2.66% 144 Missing and 2 partials ⚠️
src/platform/linux/input/inputtino.cpp 3.22% 60 Missing ⚠️
src/platform/linux/input/inputtino_keyboard.cpp 0.00% 33 Missing ⚠️
src/platform/linux/input/inputtino_pen.cpp 0.00% 29 Missing ⚠️
src/platform/linux/input/inputtino_mouse.cpp 0.00% 26 Missing ⚠️
src/input.cpp 0.00% 22 Missing ⚠️
src/platform/linux/input/inputtino_touch.cpp 0.00% 20 Missing ⚠️
src/platform/linux/input/inputtino_common.h 0.00% 17 Missing ⚠️
src/config.cpp 0.00% 5 Missing and 5 partials ⚠️
src/platform/windows/input.cpp 40.00% 7 Missing and 2 partials ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff            @@
##           master   #2606      +/-   ##
=========================================
- Coverage    9.88%   8.70%   -1.19%     
=========================================
  Files          88      94       +6     
  Lines       18070   17366     -704     
  Branches     8605    8251     -354     
=========================================
- Hits         1787    1512     -275     
+ Misses      13374   12994     -380     
+ Partials     2909    2860      -49     
Flag Coverage Δ
Linux 6.49% <1.63%> (-1.94%) ⬇️
Windows 3.82% <15.55%> (+0.04%) ⬆️
macOS-12 ?
macOS-13 9.72% <5.12%> (-0.01%) ⬇️
macOS-14 10.02% <5.12%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/input.h 0.00% <ø> (ø)
src/platform/common.h 34.56% <100.00%> (+0.81%) ⬆️
src/platform/macos/input.cpp 36.57% <14.28%> (-0.22%) ⬇️
src/platform/windows/input.cpp 1.06% <40.00%> (+0.65%) ⬆️
src/config.cpp 4.31% <0.00%> (-0.53%) ⬇️
src/platform/linux/input/inputtino_common.h 0.00% <0.00%> (ø)
src/platform/linux/input/inputtino_touch.cpp 0.00% <0.00%> (ø)
src/input.cpp 0.23% <0.00%> (-0.01%) ⬇️
src/platform/linux/input/inputtino_mouse.cpp 0.00% <0.00%> (ø)
src/platform/linux/input/inputtino_pen.cpp 0.00% <0.00%> (ø)
... and 3 more

... and 4 files with indirect coverage changes

@Hazer Hazer requested a review from ReenigneArcher May 29, 2024 21:58
@Hazer Hazer force-pushed the feat/inputiino branch 3 times, most recently from 23df8b9 to 2898ae8 Compare May 30, 2024 01:02
@ReenigneArcher
Copy link
Member

@ABeltramo any idea how to solve these errors on Fedora 40? https://github.com/LizardByte/Sunshine/actions/runs/9295282809/job/25581974710?pr=2606#step:10:3266

We're using gcc-14 on that distro, I'm guessing that's part of the problem.

@Hazer Hazer force-pushed the feat/inputiino branch from 91ea6f1 to 0c78c68 Compare May 30, 2024 01:52
@Hazer Hazer mentioned this pull request May 30, 2024
11 tasks
@ABeltramo
Copy link
Contributor

I've quickly skimmed thru the logs:

error: 'clamp' is not a member of 'std'

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

@Hazer
Copy link
Member Author

Hazer commented May 30, 2024

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

@ABeltramo Seems to work! But now it needs the same fix in uinput/pentablet.cpp too.

@azuwis
Copy link

azuwis commented May 30, 2024

I was testing this PR in my setup, and found some problems of the virtual motion sensor.

Sunshine ad93e89
Moonlight 5.0.1
DualSense Wireless Controller 054C:0CE6

After setup uhid, on the server side (sunshine), I have:

/dev/input/event30:	Sunshine DualSense (virtual) pad
/dev/input/event31:	Sunshine DualSense (virtual) pad Motion Sensors
/dev/input/event256:	Sunshine DualSense (virtual) pad Touchpad

on the client side (moonlight), I have:

/dev/input/event15:	DualSense Wireless Controller
/dev/input/event16:	DualSense Wireless Controller Motion Sensors
/dev/input/event17:	DualSense Wireless Controller Touchpad

But in the game tested, the motion sensor is unreasonablely fast to use.

After compare the output of evtest /dev/input/event*, it seems there are some problems with ABS_RX ABS_RY ABS_RZ.

So, I run evtest on the server and client at the some time to capture ABS_RX:

On the server side:

# Sunshine DualSense (virtual) pad Motion Sensors
$ evtest /dev/input/event31 | awk -F '( |,)' '/ABS_RX/ {print $3, $14}' > sunshine.txt

On the client side:

# DualSense Wireless Controller Motion Sensors
$ evtest /dev/input/event16 | awk -F '( |,)' '/ABS_RX/ {print $3, $14}' > local.txt

Edit sunshine.txt and local.txt to align the timestamp, and use gnuplot the render them:

set term png
set timefmt "%s"
set xdata time
set grid

set output "sunshine.png"
plot "sunshine.txt" using 1:2 with lines

set output "local.png"
plot "local.txt" using 1:2 with lines

sunshine

local

The 2 plots represent the same movement of the dualsense controller, the server side seems wrong to me.

@ABeltramo
Copy link
Contributor

This is probably the most in-depth bug report I've ever received, thanks @azuwis !
I know there's something wrong with the actual values of gyro and accelerometer as I was mentioning here but thanks for confirming it, I have to check where the error is between Moonlight reported values and the virtual pad implementation

@ABeltramo
Copy link
Contributor

sounds like it's missing #include <algorithm> I've just pushed a potential fix, can you upgrade the submodule to bc6afda ?

@ABeltramo Seems to work! But now it needs the same fix in uinput/pentablet.cpp too.

My bad, I've added that too!

@Hazer
Copy link
Member Author

Hazer commented May 31, 2024

I have to check where the error is between Moonlight reported values and the virtual pad implementation

@azuwis thanks for your report!

@ABeltramo I'm splitting in smaller parts the upper input layer too, so it will be easier to understand, here is the PR #2594 if you want to use it

And the (hopefully) last changes missing to build on all platforms: games-on-whales/inputtino#4

@ABeltramo
Copy link
Contributor

@Hazer I've pushed a fix for Gyro and Acceleration, it should now be very close to the original:

gyro_capture.mp4

You have to update inputtino to the latest and change the following (I'm afraid I can't push to this PR, right?):

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, motion.x, motion.y, motion.z);

to:

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, deg2rad(motion.x), deg2rad(motion.y), deg2rad(motion.z));

(add the deg2rad conversion).

It would be great if @azuwis can give it another go after this change is in!

@Hazer
Copy link
Member Author

Hazer commented May 31, 2024

@ABeltramo sent you an invite for my fork, if you want to change directly.
Another fix for fedora: games-on-whales/inputtino#5

Updated with latest ps5 changes, can you give it another go @azuwis?

@ABeltramo
Copy link
Contributor

@ABeltramo sent you an invite for my fork, if you want to change directly. Another fix for fedora: games-on-whales/inputtino#5

Updated with latest ps5 changes, can you give it another go @azuwis?

Damn you Fedora! I'm going to install it in a VM to quickly test this compilation issues..
I don't see the deg2rad change that I've mentioned before in the commit that you pushed, could it be because I'm on mobile?
Thanks for the invite, I'll double-check everything again tomorrow!

@Hazer
Copy link
Member Author

Hazer commented May 31, 2024

@ABeltramo a few developers VS a hat, who would win?
fedora hat
So far the hat is beating both of us 🤣

I don't know, maybe it's because it's on mobile. I've updated now to games-on-whales/inputtino@99c9840, so I guess everything should be there.

PS: Fedora is building with latest release. So far.

@azuwis
Copy link

azuwis commented Jun 1, 2024

Sunshine: azuwis@4855e09, not the version in this PR, have to apply the patch @ABeltramo mentioned:

You have to update inputtino to the latest and change the following (I'm afraid I can't push to this PR, right?):

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, motion.x, motion.y, motion.z);

to:

std::get<inputtino::PS5Joypad>(*gamepad->joypad).set_motion(inputtino::PS5Joypad::GYROSCOPE, deg2rad(motion.x), deg2rad(motion.y), deg2rad(motion.z));

Gnuplot of ABS_RX:

set term png
set timefmt "%s"
set xdata time
set grid

set output "all.png"
plot "sunshine.txt" using 1:2 with lines linecolor rgb "red" title "Sunshine", \
     "local.txt" using 1:2 with lines linecolor rgb "blue" title "Local"

all

Great work! 👍

@ABeltramo
Copy link
Contributor

ABeltramo commented Jun 1, 2024

Thanks @azuwis for confirming! @Hazer I've pushed the missing bit to your branch, I think there are only a few things left for this PR:

  • Fix the Fedora build
  • Fix the PS5 trackpad integration (it seems to not move properly)
  • Decide if we want to add the automatic failover (mentioned here) when uhid/uinput aren't available and implement it

Anything else missing?

@ReenigneArcher
Copy link
Member

For your third checklist, I don't know that it's worth it. I don't think we'll be keeping both code paths for long.

@ABeltramo
Copy link
Contributor

For your third checklist, I don't know that it's worth it. I don't think we'll be keeping both code paths for long.

That sounds good to me! Shall we clean it up then? I mean, it's always going to be present in git history if anyone ever is going to need it..

@ReenigneArcher
Copy link
Member

I'm fine with that, or handling it in a separate PR right after the next stable release.

I'm just a little worried about a big change like this with so few testers.

@ABeltramo
Copy link
Contributor

That's a valid concern; whilst the DualSense stuff is definitely experimental and lightly tested users can always force an xbox joypad from the config page and that's been in use for more than a year in Wolf. I'd assume that any big issue would have been reported by now..

@ReenigneArcher ReenigneArcher enabled auto-merge (squash) June 15, 2024 23:31
@ReenigneArcher ReenigneArcher merged commit 509576d into LizardByte:master Jun 16, 2024
47 of 48 checks passed
@VoxelTek
Copy link

Hey! Just did a test of the latest version. I can confirm, the rumble issue is indeed fixed, however, the "low battery" message still appears.
image

@ABeltramo
Copy link
Contributor

Hey! Just did a test of the latest version. I can confirm, the rumble issue is indeed fixed, however, the "low battery" message still appears. image

Thanks for testing it out! How far off is the battery from what is reported in the client? I'm not sure how reliable is Moonlight (SDL I guess?) in getting that value.. It would be nice to see if it's the Moonlight control packet reporting that or something wrong in the virtual device in Sunshine.

@ReenigneArcher
Copy link
Member

@ABeltramo it looks like the percentage we give is divided by 10. Is that the issue? https://github.com/games-on-whales/inputtino/blob/2739465690c7bbd1a27cb4e285ff08f486a208e3/src/uhid/joypad_ps5.cpp#L383

@VoxelTek
Copy link

Thanks for testing it out! How far off is the battery from what is reported in the client? I'm not sure how reliable is Moonlight (SDL I guess?) in getting that value.. It would be nice to see if it's the Moonlight control packet reporting that or something wrong in the virtual device in Sunshine.
@ABeltramo

I'll have to do more investigation, I've been primarily testing using a modded Switch running a Moonlight client. I have tried with other proper controllers, however, and I do believe that those also presented the "low battery" message. I'll double-check in a bit.

When I received those errors, the Switch was most certainly on somewhere around 80-90%, or more.

@cgutman
Copy link
Collaborator

cgutman commented Jun 17, 2024

Maybe @cgutman can shed some light on which HW is currently supported as a drawing tablet?

SDL2 doesn't have proper pen support, so you'd need to test on Android or iOS using a stylus. Those should support all of the fancy pen features (pressure, tilt, hover, buttons) assuming your hardware has them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants