Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor WebGL performance on RPi5 #172

Open
deregtd opened this issue Jun 25, 2024 · 34 comments · Fixed by #177
Open

Poor WebGL performance on RPi5 #172

deregtd opened this issue Jun 25, 2024 · 34 comments · Fixed by #177

Comments

@deregtd
Copy link

deregtd commented Jun 25, 2024

I'm using the browser module out of the box (v2.7.0) on my 8GB Raspberry Pi 5, with ENABLE_GPU=1 and other such niceties, on a 1080p touchscreen on HDMI, all deployed via balena:

  browser:
    image: bh.cr/balenalabs/browser-aarch64:v2.7.0
    network_mode: host
    privileged: true
    environment:
      LAUNCH_URL: "http://localhost"
      KIOSK: 0
      PERSISTENT: 1
      ENABLE_GPU: 1
      WINDOW_SIZE: 1920,1080
    labels:
      io.balena.features.dbus: "1"
    volumes:
      - browser-data:/data

My application is a webgl-based application that does map rendering, and poor performance (1fps even though the render function takes ~15ms to run) is pointing to something very wrong with the hardware pipeline on the balena block. I then tried doing the basic WebGL Aquarium sample and the browser gets 7-8fps on there with 500 fish (default mode). Even when I disable all the maps so my application is rendering a blank screen with an arrow in the center, it's still only getting 7-8fps (with a render function well below 1ms now).

I've checked chrome://gpu and all the usual stuff is enabled. Other benchmarks on a Pi5 suggest that I should be getting 30-40fps.

Any idea what's up here? It's looking like I'm going to have to bail on having a pi5 power my kiosk and just get a cheap android tablet to hit the website served by the pi remotely.

@deregtd
Copy link
Author

deregtd commented Jul 9, 2024

For any others that come here, I ended up forking the repo to my internal docker build, and just having it kick a build. All I changed was moving the base image to %plat%-debian-node:20-bookworm-run, since the node 16 images haven't been updated as recently as the node 20 ones. From there, it just apt-get installs chromium/mesa, so the rebuild gets latest version of those packages. Boom -- my simple mapping app went from 2-3fps up to 20fps, and the browser is way snappier to use. So, I think just doing a re-release of the package (ideally, updating to node 20) and calling it 2.7.1 would be a big benefit to the community, if anyone's still watching this... (balena feels kinda dead right now, so not sure if anyone is)

@basz
Copy link

basz commented Aug 29, 2024

| So, I think just doing a re-release of the package (ideally, updating to node 20) and calling it 2.7.1

So, 2.7.1. was released but was this added? I'm still not seeing GPU performance on a pi5

@basz
Copy link

basz commented Aug 29, 2024

io.balena.features.dbus: "1"

Why is that needed?

@deregtd
Copy link
Author

deregtd commented Aug 29, 2024

I'm suggesting that they do a re-release, since simply building a new release of the image will solve the problem. I forked the dockerfile to my own buildchain and simply did a rebuild/"release" and am using that image.

I honestly don't remember why I added the dbus thing. I don't see it in the example code for the browser module...

@basz
Copy link

basz commented Aug 29, 2024

Ah. That solved your performance issue on the pi5.

Gonna try that also! Thx!

@phil-d-wilson
Copy link
Contributor

@deregtd balena is not dead (far from it) but the main maintainer of this block (me) became annoyingly busy running the company. Although I would love to still do technical work, and have great intentions, I almost always get pulled into other matters.

I will, however, endeavour to get the stack updated as you suggest. My request, in return, is if you could help me test it? We need the result to be performant on Pi3, 4, 5, zero 2 W and Intel NUC.
As you can see, the testing is really the time suck not the technical work.

Phil

@deregtd
Copy link
Author

deregtd commented Aug 29, 2024

How dare you run the company! :D

I only have pi 5s here, but I can try switching to a release you kick out. If you just rebuild the release, you should end up with an effectively identical image to what I have in my docker image builder repo, so I have faith, but happy to test it out when it's ready. Just lemme know.

BTW, this block really needs an onscreen keyboard to be terribly useful for kiosks, btw. I def can't use this in any wider release of my stuff without an OSK. I just use a laptop/phone to change settings in the app right now, but that doesn't scale.

@phil-d-wilson
Copy link
Contributor

For sure, but I have to be honest that I don't have the time to implement (and test) that. I use the browser-control block from my phone or another machine - but then I don't have to change my screens often. Mostly they just display home assistant dashboards around the house. 😁

@basz
Copy link

basz commented Aug 29, 2024

I can test on pi4

@phil-d-wilson
Copy link
Contributor

@basz @deregtd as good as my word, I did endeavour! I bumped up the version of node and deployed the block to my own organisation (i.e. not balenalabs) and then pushed that to my application that all my screens around the house use, and.....everything is broken when I enable the GPU. :(

image

One thing I noticed is that the block now has a different build for RPi's and everything else, so that it can bring in the patched chromium for Pi video drivers:

if [ "${FORCE_RPI_REPO}" != "0" ] && [ "${FORCE_RPI_REPO}" = "1" -o "${MACHINE_NAME:0:9}" = "raspberry" -o "${MACHINE_NAME}" = "fincm3" ]

This takes the device type the block was built for, and if that is a RaspberryPi variant uses the "chromium-browser" instead of pure Chromium. The only block built for a RaspberryPi device type was the armv7 one:
image
So @deregtd the reason that you originally saw poor performance on your Pi5 may have been because you used the aarch64 block, which will have used vanilla Chromium.

Now I need to figure out what is not being installed as part of the "chromium-browser" package to enable the GPU without all current weirdness. Disabling the GPU makes everything (non-webGL) work fine, FWIW.

@phil-d-wilson
Copy link
Contributor

@deregtd could you push your fork to your own GH repo, so that I could take a look, please?

@deregtd
Copy link
Author

deregtd commented Aug 31, 2024

I'm pretty sure it has no changes at all from your repo. I think I just re ran your build. Here is the docker file from my repo:

ARG NODEJS_VERSION="20"

FROM balenalib/raspberrypi5-debian-node:${NODEJS_VERSION}-bookworm-run

# Install the necessary packages
COPY ./build /usr/src/build
RUN /usr/src/build/install_chromium "raspberrypi5"

WORKDIR /usr/src/app

# install node dependencies
COPY ./package.json /usr/src/app/package.json
RUN JOBS=MAX npm install --unsafe-perm --production && npm cache clean --force

COPY ./src /usr/src/app/

RUN chmod +x ./*.sh

ENV UDEV=1

RUN mkdir -p /etc/chromium/policies
RUN mkdir -p /etc/chromium/policies/recommended
COPY ./policy.json /etc/chromium/policies/recommended/my_policy.json

# Add chromium user
RUN useradd chromium -m -s /bin/bash -G root || true && \
    groupadd -r -f chromium && id -u chromium || true \
    && chown -R chromium:chromium /home/chromium || true

#COPY ./public-html /home/chromium  

# udev rule to set specific permissions 
RUN echo 'SUBSYSTEM=="vchiq",GROUP="video",MODE="0660"' > /etc/udev/rules.d/10-vchiq-permissions.rules
RUN usermod -a -G audio,video,tty chromium

# Set up the audio block. This won't have any effect if the audio block is not being used.
RUN curl -skL https://raw.githubusercontent.com/balena-labs-projects/audio/master/scripts/alsa-bridge/debian-setup.sh| sh
ENV PULSE_SERVER=tcp:audio:4317

COPY VERSION .

# Start app
CMD ["bash", "/usr/src/app/start.sh"]

@phil-d-wilson
Copy link
Contributor

@basz @deregtd good news - I found that my test device didn't have the correct DT overlay, which was what was causing the display problems. I now have many virtual fish swimming around the screen at 60FPS. :)

@deregtd
Copy link
Author

deregtd commented Sep 2, 2024

Sweet! Lemme know if you have an image you want me to test on a 5.

@basz
Copy link

basz commented Sep 2, 2024

Nice! What's the correct overlay then?

@phil-d-wilson
Copy link
Contributor

vc4-kms-v3d @basz

Why my pi5 didn't have that by default is the next sensible question (which I'm on).

@deregtd I want to test a few things and then I'll put up a test image

@basz
Copy link

basz commented Sep 2, 2024

Cuase vc4-kms-v3d-pi5 apparently is a thing too.

@deregtd
Copy link
Author

deregtd commented Sep 2, 2024

vc4-kms-v3d @basz

Why my pi5 didn't have that by default is the next sensible question (which I'm on).

@deregtd I want to test a few things and then I'll put up a test image

I've had to put that overlay on our pis through the balena panel for years.

@phil-d-wilson
Copy link
Contributor

@deregtd I'm investigating....

@deregtd
Copy link
Author

deregtd commented Sep 2, 2024

Cuase vc4-kms-v3d-pi5 apparently is a thing too.

If you use the base vc4-kms-v3d overlay, it properly uses the right version based on arch. So you can force it by adding -pi5 to the end but you don't have to.

@phil-d-wilson
Copy link
Contributor

@deregtd @basz tested and working on Pi3, Pi4 and Pi5 and a NUC, but you're also welcome to test before I push to the balenalabs org:

bh.cr/gh_phil_d_wilson/browser-armv7hf

@basz
Copy link

basz commented Sep 3, 2024

nice Phil!

one snag. Error: (HTTP code 404) unexpected - pull access denied for bh.cr/gh_phil_d_wilson/browser-armv7hf, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

@phil-d-wilson
Copy link
Contributor

phil-d-wilson commented Sep 3, 2024

@basz sorry, try now. I forgot to make it public and was using it from inside my own org.

@basz
Copy link

basz commented Sep 3, 2024

np & thx. building and deploying now. ready for when i get to my work studio. will report.

@basz
Copy link

basz commented Sep 3, 2024

pi4 and pi5 confirmed to work! awesome.

@phil-d-wilson phil-d-wilson linked a pull request Sep 4, 2024 that will close this issue
@phil-d-wilson
Copy link
Contributor

All fixed in version 2.8.0 bh.cr/balenalabs/browser-armv7hf 👍

@phil-d-wilson
Copy link
Contributor

@deregtd not dead, just spinning many plates 😉

@basz
Copy link

basz commented Sep 4, 2024

thank you very much!

@deregtd
Copy link
Author

deregtd commented Sep 16, 2024

Unfortunately, on an RPi 5, bumping from my fork ("2.7.1") up to the latest 2.9.0 causes the very slow performance to return. :(

Using image: bh.cr/balenalabs/browser-aarch64/2.9.0

Output on console on startup:

umount: /dev: target is busy.

     Step 1 - 654.10s (Deploying)
     DONE IN: 654.10s 

user.max_user_namespaces = 10000
mount: /tmp/tmpmount/console: wrong fs type, bad option, bad superblock on /dev/console, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
umount: /dev: target is busy.
Running balena base image entrypoint...
setting xserver-xorg-legacy/xwrapper/allowed_users from configuration file
balenaLabs browser version: 2.9.0
Setting CPU Scaling Governor to 'performance'
Disabling cursor
Raspberry Pi 5 detected, injecting X.org config
hostname: No address associated with hostname
xauth:  file /home/chromium/.Xauthority does not exist
xauth: (stdin):1:  bad display name "ce688f5:0" in "add" command
X.Org X Server 1.21.1.7
X Protocol Version 11, Revision 0
Current Operating System: Linux ce688f5 6.1.77-v8-16k #1 SMP PREEMPT Thu Feb  8 15:07:10 UTC 2024 aarch64
Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=2C:CF:67:37:BF:66 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 rootfstype=ext4 rootwait dwc_otg.lpm_enable=0 rootwait console=null quiet splash vt.global_cursor_default=0 consoleblank=0 cgroup_enable=memory   root=UUID=ba1eadef-acfd-4946-a06b-4dc60082e980 
xorg-server 2:21.1.7-3+deb12u7 (https://www.debian.org/support) 
Current version of pixman: 0.42.2
	Before reporting problems, check http://wiki.x.org/
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Mon Sep 16 00:43:29 2024
(==) Using config directory: "/etc/X11/xorg.conf.d"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
V3D 7.1 not supported by this version of Mesa.
V3D 7.1 not supported by this version of Mesa.
--- List Input Devices ---
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer              	id=4	[slave  pointer  (2)]
⎜   ↳ Touchplanet ETM-P64                     	id=6	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core XTEST keyboard             	id=5	[slave  keyboard (3)]
    ↳ pwr_button                              	id=7	[slave  keyboard (3)]
    ↳ vc4-hdmi-0                              	id=8	[slave  keyboard (3)]
    ↳ vc4-hdmi-1                              	id=9	[slave  keyboard (3)]
----- End of List --------
Window size set by environment variable to 1920,1080
Installed browser version: Chromium 128.0.6613.113 built on Debian GNU/Linux 12 (bookworm)
Browser API running on port: 5011
Using LAUNCH_URL: http://localhost/
Enabling GPU
Disabling KIOSK mode
Starting Chromium with flags: --disable-features=TranslateUI,--disable-component-extensions-with-background-pages,--disable-background-networking,--disable-sync,--metrics-recording-only,--disable-default-apps,--no-default-browser-check,--no-first-run,--disable-backgrounding-occluded-windows,--disable-renderer-backgrounding,--disable-background-timer-throttling,--force-fieldtrials=*BackgroundTracing/default/,--window-size=1920,1080,--window-position=0,0,--autoplay-policy=no-user-gesture-required,--noerrdialogs,--disable-session-crashed-bubble,--check-for-update-interval=31536000,--disable-dev-shm-usage,--enable-zero-copy,--num-raster-threads=4,--ignore-gpu-blocklist,--enable-gpu-rasterization
Displaying URL: http://localhost/
Chromium remote debugging tools running on port: 35173
█

Using my older working build:

umount: /dev: target is busy.
user.max_user_namespaces = 10000

     Step 1 - 3.24s (Deploying)
     DONE IN: 3.24s 

mount: /tmp/tmpmount/console: wrong fs type, bad option, bad superblock on /dev/console, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
umount: /dev: target is busy.
Running balena base image entrypoint...
setting xserver-xorg-legacy/xwrapper/allowed_users from configuration file
balenaLabs browser version: 2.7.1
Setting CPU Scaling Governor to 'performance'
Disabling cursor
Raspberry Pi 5 detected, injecting X.org config
hostname: No address associated with hostname
xauth:  file /home/chromium/.Xauthority does not exist
xauth: (stdin):1:  bad display name "ce688f5:0" in "add" command
X.Org X Server 1.21.1.7
X Protocol Version 11, Revision 0
Current Operating System: Linux ce688f5 6.1.77-v8-16k #1 SMP PREEMPT Thu Feb  8 15:07:10 UTC 2024 aarch64
Kernel command line: reboot=w coherent_pool=1M 8250.nr_uarts=1 pci=pcie_bus_safe snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1  smsc95xx.macaddr=2C:CF:67:37:BF:66 vc_mem.mem_base=0x3fc00000 vc_mem.mem_size=0x40000000  dwc_otg.lpm_enable=0 rootfstype=ext4 rootwait dwc_otg.lpm_enable=0 rootwait console=null quiet splash vt.global_cursor_default=0 consoleblank=0 cgroup_enable=memory   root=UUID=ba1eadef-acfd-4946-a06b-4dc60082e980 
xorg-server 2:21.1.7-3+rpt2+deb12u7 (https://www.debian.org/support) 
Current version of pixman: 0.42.2
	Before reporting problems, check http://wiki.x.org/
	to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Mon Sep 16 00:52:22 2024
(==) Using config directory: "/etc/X11/xorg.conf.d"
(==) Using system config directory "/usr/share/X11/xorg.conf.d"
--- List Input Devices ---
⎡ Virtual core pointer                    	id=2	[master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer              	id=4	[slave  pointer  (2)]
⎜   ↳ Touchplanet ETM-P64                     	id=6	[slave  pointer  (2)]
⎣ Virtual core keyboard                   	id=3	[master keyboard (2)]
    ↳ Virtual core XTEST keyboard             	id=5	[slave  keyboard (3)]
    ↳ pwr_button                              	id=7	[slave  keyboard (3)]
    ↳ vc4-hdmi-0                              	id=8	[slave  keyboard (3)]
    ↳ vc4-hdmi-1                              	id=9	[slave  keyboard (3)]
----- End of List --------
Window size set by environment variable to 1920,1080
dpkg-query: no packages found matching bluealsa
Installed browser version: Chromium 125.0.6422.133 Built on Debian , running on Debian 12
Browser API running on port: 5011
Using LAUNCH_URL: http://localhost/
Enabling GPU
Disabling KIOSK mode
Starting Chromium with flags: --disable-features=TranslateUI,--disable-component-extensions-with-background-pages,--disable-background-networking,--disable-sync,--metrics-recording-only,--disable-default-apps,--no-default-browser-check,--no-first-run,--disable-backgrounding-occluded-windows,--disable-renderer-backgrounding,--disable-background-timer-throttling,--force-fieldtrials=*BackgroundTracing/default/,--window-size=1920,1080,--window-position=0,0,--autoplay-policy=no-user-gesture-required,--noerrdialogs,--disable-session-crashed-bubble,--check-for-update-interval=31536000,--disable-dev-shm-usage,--enable-zero-copy,--num-raster-threads=4,--ignore-gpu-blocklist,--enable-gpu-rasterization
Displaying URL: http://localhost/
Chromium remote debugging tools running on port: 35173

The key thing I was trying to eliminate earlier was the:

V3D 7.1 not supported by this version of Mesa.

I've attached my entire image build folder for my local balena-browser build that gave me a working build with no v3d errors, at least as of 2 months ago, in case it helps:
balena-browser.zip

@phil-d-wilson
Copy link
Contributor

@deregtd could you try the arm7hf version of the block, please?

@deregtd
Copy link
Author

deregtd commented Sep 16, 2024

I tried the arm7hf version and it immediately quits with error 139. I'm running a 64-bit os on rpi 5, using the balena base image untouched. So I'm guessing it doesn't support mismatched architecture images out of the box?

@22chrs
Copy link

22chrs commented Sep 22, 2024

nice job! that is the reason for my bad performance, playing videos on a pi5 vs an x64 pc. looking forward for a release!

@phil-d-wilson phil-d-wilson reopened this Oct 24, 2024
@phil-d-wilson
Copy link
Contributor

Sorry @deregtd - I missed the replies (and got busy on company "stuff" again). I'll try to look at this again this weekend.

Also pinging @rahul-thakoor and @mtoman to similarly make them feel guilty about letting down the faithful browser block community. 😄

@deregtd
Copy link
Author

deregtd commented Oct 24, 2024

We've packaged our boat up for the winter, so I don't have a way to test this until the spring, btw. :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants