Weird touch behavior on mobile device

[ryo33]

I found problems on mobile device and chrome developer tool's mobile device mode. The video is recorded on chrome on macbook.
This happpens in https://desk-x.com and your demo mvlabat.github.io/bevy_egui_web_showcase, and does not happens in https://egui.rs

Problems

1. Drag position is reset on any next touch.
2. It can be moved by dragging any position.

Possible solution

Proper handling of touch release event (or something).

Screen.Recording.0004-05-07.at.8.29.25.mov

[vladbat00]

Hi! Thanks for filing an issue. Yeah, unfortunately, bevy_egui doesn't have a proper touch device support, as it doesn't react to touch events at all. I'll be happy to accept a PR if you'd like to tackle this.

[ryo33]

I'd like to resolve this. Is it hard to do? just not yet implemented?

[vladbat00]

I think it shouldn't be difficult. I believe the only system you'll need to look into is process_input. But since I've never worked with touch devices, I've no idea how many edge-cases there can be. :) It can also be a good idea to look into the egui's demo implementation for reference.

[ryo33]

Ok. Thanks. I’ll try it.

[71e6fd52]

The input event of bevy is from winit.
But winit hasn't implemented touch handling on web target. rust-windowing/winit#1673
eframe get touch event from web_sys.
So, should we include web_sys dependency to support touch?

[ryo33]

I forgot about my PR that adds support web touch event to winit.
rust-windowing/winit#2188

I think waiting for merge is fine for me.

[ryo33]

rust-windowing/winit#2188
Things advance!

[vladbat00]

Wow, nice! If web touch device support lands into winit, that will be great. Thank you for your work on this @ryo33.

@71e6fd52 if anyone wants to make a PR adding touch device support by using web_sys directly, I'm open to merging it. I'd probably just feature gate it. Once the winit PR is merged and Bevy upgrades to a newer version, we can always switch to the native support.

[gui1117]

I use winit with the PR 2188, and I modified bevy_egui to handle touch event, I got this, which seems to work, by inspiring from bevy_winit.

Here is the branch: https://github.com/thiolliere/bevy_egui/tree/gui-touch-event
The diff: main...thiolliere:bevy_egui:gui-touch-event

diff --git a/src/systems.rs b/src/systems.rs
index e484397..a0ed473 100644
--- a/src/systems.rs
+++ b/src/systems.rs
@@ -13,6 +13,7 @@ use bevy::{
         keyboard::{KeyCode, KeyboardInput},
         mouse::{MouseButton, MouseButtonInput, MouseScrollUnit, MouseWheel},
         ElementState, Input,
+        touch::TouchInput,
     },
     prelude::EventReader,
     utils::HashMap,
@@ -33,6 +34,7 @@ pub struct InputEvents<'w, 's> {
     ev_keyboard_input: EventReader<'w, 's, KeyboardInput>,
     ev_window_focused: EventReader<'w, 's, WindowFocused>,
     ev_window_created: EventReader<'w, 's, WindowCreated>,
+    ev_touch: EventReader<'w, 's, TouchInput>,
 }
 
 #[derive(SystemParam)]
@@ -40,6 +42,7 @@ pub struct InputResources<'w, 's> {
     #[cfg(feature = "manage_clipboard")]
     egui_clipboard: Res<'w, crate::EguiClipboard>,
     keyboard_input: Res<'w, Input<KeyCode>>,
+    pointer_touch_id: Local<'s , Option<u64>>,
     egui_input: ResMut<'w, HashMap<WindowId, EguiInput>>,
     #[system_param(ignore)]
     marker: PhantomData<&'s usize>,
@@ -225,6 +228,8 @@ pub fn process_input(
         }
     }
 
+    let focused_window_height = window_resources.focused_window.as_ref().map(|window_id| window_resources.window_sizes[&window_id].height());
+
     if let Some(focused_input) = window_resources
         .focused_window
         .as_ref()
@@ -269,6 +274,77 @@ pub fn process_input(
         }
 
         focused_input.raw_input.modifiers = modifiers;
+
+        for touch in input_events.ev_touch.iter() {
+            let scale_factor = egui_settings.scale_factor as f32;
+            let mut touch_position: (f32, f32) = (touch.position / scale_factor).into();
+            touch_position.1 = focused_window_height.unwrap() / scale_factor - touch_position.1;
+
+            // Emit touch event
+            focused_input.raw_input.events.push(egui::Event::Touch {
+                device_id: egui::TouchDeviceId(egui::epaint::util::hash(touch.id)),
+                id: egui::TouchId::from(touch.id),
+                phase: match touch.phase {
+                    bevy::input::touch::TouchPhase::Started => egui::TouchPhase::Start,
+                    bevy::input::touch::TouchPhase::Moved => egui::TouchPhase::Move,
+                    bevy::input::touch::TouchPhase::Ended => egui::TouchPhase::End,
+                    bevy::input::touch::TouchPhase::Cancelled => egui::TouchPhase::Cancel,
+                },
+                pos: egui::pos2(touch_position.0, touch_position.1),
+                force: match touch.force {
+                    Some(bevy::input::touch::ForceTouch::Normalized(force)) => force as f32,
+                    Some(bevy::input::touch::ForceTouch::Calibrated {
+                        force,
+                        max_possible_force,
+                        ..
+                    }) => (force / max_possible_force) as f32,
+                    None => 0_f32,
+                },
+            });
+
+            // If we're not yet tanslating a touch or we're translating this very
+            // touch …
+            if input_resources.pointer_touch_id.is_none() || input_resources.pointer_touch_id.unwrap() == touch.id {
+                // … emit PointerButton resp. PointerMoved events to emulate mouse
+                match touch.phase {
+                    bevy::input::touch::TouchPhase::Started => {
+                        *input_resources.pointer_touch_id = Some(touch.id);
+                        // First move the pointer to the right location
+                        focused_input.raw_input
+                            .events
+                            .push(egui::Event::PointerMoved(egui::pos2(touch_position.0, touch_position.1)));
+                        // Then do mouse button input
+                        focused_input.raw_input.events.push(egui::Event::PointerButton {
+                            pos: egui::pos2(touch_position.0, touch_position.1),
+                            button: egui::PointerButton::Primary,
+                            pressed: true,
+                            modifiers,
+                        });
+                    }
+                    bevy::input::touch::TouchPhase::Moved => {
+                        focused_input.raw_input
+                            .events
+                            .push(egui::Event::PointerMoved(egui::pos2(touch_position.0, touch_position.1)));
+                    }
+                    bevy::input::touch::TouchPhase::Ended => {
+                        *input_resources.pointer_touch_id = None;
+                        focused_input.raw_input.events.push(egui::Event::PointerButton {
+                            pos: egui::pos2(touch_position.0, touch_position.1),
+                            button: egui::PointerButton::Primary,
+                            pressed: false,
+                            modifiers,
+                        });
+                        focused_input.raw_input.events.push(egui::Event::PointerGone);
+                    }
+                    bevy::input::touch::TouchPhase::Cancelled => {
+                        *input_resources.pointer_touch_id = None;
+                        focused_input.raw_input.events.push(egui::Event::PointerGone);
+                    }
+                }
+            }
+
+        }
+
     }
 
     for egui_input in input_resources.egui_input.values_mut() {

Also thanks for the good work

[ryo33]

rust-windowing/winit#2188 is steady progress.

@thiolliere Nice work! Could you submit a pull-req? It should be helpful for platforms except for the Web. Also, I plan to use this feature for my product which will be released next month!

[ryo33]

Can I close this? #134 has been opened, and this cannot be resolved until winit merges it.

[ryo33]

rust-windowing/winit#2188 have been merged!

[vladbat00]

@ryo33 nice work! Glad it has been merged :)
I don't mind keeping this issue open btw, to track touch device support in web specifically.