App crashes with high number of images and "performance" settings

[redmoon1945]

Version

1.0.2

OS

Linux

Description

Using the App Image and 1050 jpg file to convert (around 1 to 2 Mb each), multi-threading = performance, threads = 16, conversion starts ok, but available memory (32 Gb) is eaten up till the system crashes. Superb app by the way :-)

Steps to Reproduce

Proceed as in the description

Expected Result

Even in "performance" mode, the app should provide a notification message when lets say a certain level of system memory has been consumed (or is left), then stop the conversion, releasing the memory allocated. All that without crashing.

Current Result

The memory is consumed until system crashes.

Input Formats

jpg

Output Format

jpeg XL

Additional Information (Optional)

Here is my system info :

System:
  Kernel: 6.8.0-41-generic arch: x86_64 bits: 64 compiler: gcc v: 13.2.0 clocksource: tsc
  Desktop: Cinnamon v: 6.2.9 tk: GTK v: 3.24.41 wm: Muffin v: 6.2.0 vt: 7 dm: LightDM v: 1.30.0
    Distro: Linux Mint 22 Wilma base: Ubuntu 24.04 noble
Machine:
  Type: Laptop System: LENOVO product: 21CQ000GUS v: ThinkPad T14s Gen 3
    serial: <superuser required> Chassis: type: 10 serial: <superuser required>
  Mobo: LENOVO model: 21CQ000GUS v: SDK0T76530 WIN serial: <superuser required>
    part-nu: LENOVO_MT_21CQ_BU_Think_FM_ThinkPad T14s Gen 3 uuid: <superuser required> UEFI: LENOVO
    v: R22ET70W (1.40 ) date: 03/21/2024
CPU:
  Info: 8-core model: AMD Ryzen 7 PRO 6850U with Radeon Graphics bits: 64 type: MT MCP smt: enabled
    arch: Zen 3+ rev: 1 cache: L1: 512 KiB L2: 4 MiB L3: 16 MiB
  Speed (MHz): avg: 1346 high: 4662 min/max: 400/4768 cores: 1: 400 2: 2052 3: 400 4: 400 5: 400
    6: 400 7: 4662 8: 4624 9: 400 10: 2073 11: 400 12: 400 13: 2076 14: 2050 15: 400 16: 400
    bogomips: 86235
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: AMD Rembrandt [Radeon 680M] vendor: Lenovo driver: amdgpu v: kernel arch: RDNA-2 pcie:
    speed: 16 GT/s lanes: 16 ports: active: DP-1,eDP-1 empty: DP-2, DP-3, DP-4, DP-5, DP-6,
    HDMI-A-1, Writeback-1 bus-ID: 33:00.0 chip-ID: 1002:1681 class-ID: 0300 temp: 48.0 C
  Device-2: IMC Networks Integrated Camera driver: uvcvideo type: USB rev: 2.0 speed: 480 Mb/s
    lanes: 1 bus-ID: 5-1:2 chip-ID: 13d3:5287 class-ID: fe01 serial: <filter>
  Display: x11 server: X.Org v: 21.1.11 with: Xwayland v: 23.2.6 driver: X: loaded: amdgpu
    unloaded: fbdev,modesetting,vesa dri: radeonsi gpu: amdgpu display-ID: :0 screens: 1
  Screen-1: 0 s-res: 2560x2640 s-dpi: 96 s-size: 677x699mm (26.65x27.52") s-diag: 973mm (38.31")
  Monitor-1: DP-1 mapped: DisplayPort-0 pos: top-left model: Dell S2719DC serial: <filter>
    res: 2560x1440 hz: 60 dpi: 109 size: 597x336mm (23.5x13.23") diag: 685mm (27") modes:
    max: 2560x1440 min: 720x400
  Monitor-2: eDP-1 mapped: eDP pos: primary,bottom-r model: BOE Display 0x0a35 res: 1920x1200
    hz: 60 dpi: 161 size: 302x189mm (11.89x7.44") diag: 356mm (14") modes: max: 1920x1200
    min: 640x480
  API: EGL v: 1.5 hw: drv: amd radeonsi platforms: device: 0 drv: radeonsi device: 1 drv: swrast
    surfaceless: drv: radeonsi x11: drv: radeonsi inactive: gbm,wayland
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: amd mesa v: 24.0.9-0ubuntu0.1 glx-v: 1.4
    direct-render: yes renderer: AMD Radeon Graphics (radeonsi rembrandt LLVM 17.0.6 DRM 3.57
    6.8.0-41-generic) device-ID: 1002:1681

  Memory: total: 32 GiB note: est. available: 30.12 GiB used: 5.37 GiB (17.8%)
  Processes: 419 Power: uptime: 2h 51m states: freeze,mem,disk suspend: deep wakeups: 0
    hibernate: platform Init: systemd v: 255 target: graphical (5) default: graphical
  Compilers: clang: 18 gcc: 13.2.0 Client: Unknown python3.12 client inxi: 3.3.34

[redmoon1945]

Forgot to mention that effort=9 (I want max quality, ready to have longer processing time), Intelligent mode =OFF

[JacobDev1]

What is the highest resolution in those images you're processing?

You can enable the "Low RAM" mode to handle this edge case for now and/or use Effort 7. Effort 8 and higher require an unreasonable amount of RAM for high resolution images.

Effort 7 does streaming encoding, while 8 and higher does not. I filed an issue to libjxl requesting this feature extended in cjxl.

The "Low RAM" mode runs the conversion as if you were using a script. One image at a time.

[redmoon1945]

Thanks.
max resolution = 8256 x 5504 , which is fairly common in the world of photography (not seen as "high res")

If I use Low RAM, and effort = 9 , RAM consumption will still be "reasonable" and constrained (I have 32 GB) ? If not, then maybe the label "Low RAM" should be renamed, like e.g. "one at a time" (as opposed to "parallel") ??

Thanks for filling an issue with libjxl.
Regards

[JacobDev1]

I consider 8 k to be "high-res" because it requires over 8 GB of RAM to be processed with Effort 9. Review my libjxl issue for more details.

I will be reworking this function. The naming and the way it works will most change in the next version.

[redmoon1945]

8 GB is "reasonnable" for me :-) So I will keep effort=9.
thx, this app is unique and really built with real use cases in mind. Deepest congrats !

[JacobDev1]

Thanks. If libjxl devs add streaming encoding to cjxl, this will lower the RAM usage by a lot. If not, I'll try to work around this. We'll see.

[JacobDev1]

This issue is related to #49

For now, I'll wait for the next libjxl release with hopes streaming encoding gets added to cjxl. If that won't happen, I'll prototype solutions for this problem.

Ideally, I would like to transition to using the API at some point. However, that's a big undertaking.

[Ukhryuk-Hai]

@JacobDev1 A little tip: you can save your time if don't rush to close threads that deal with unresolved issues.

[JacobDev1]

The next update will address this problem.

"Multithreading" modes will be replaced by this option: "JPEG XL - Optimize RAM Usage". This optimizer will switch to single-worker mode when streaming encoding is unavailable. It will fix the high RAM usage at the cost of some encoding speed. It won't prevent the program from using multiple threads.