3s+ lag when switching to space with unity editor

[devidw]

When switching to a space that has a window of Unity Editor open it takes 3s or longer until the space switch actually takes place visually.

Checklist

• I've searched for duplicates. My search queries were (please don't forget to search in closed issues too):
  • unity
  • Option to hide windows instead of resizing them into the corner #235 (comment) is also mentioning it, but its not further discussed in the issue thread nor is it the subject of the issue
• I've read the documentation https://nikitabobko.github.io/AeroSpace/guide & https://nikitabobko.github.io/AeroSpace/commands
• I've searched in the documentation. My search queries were:
  • unity

aerospace CLI client version: 0.14.2-Beta 0cb8dbdfc5ee73b8cbc200f175f467ebead55201
AeroSpace.app server version: 0.14.2-Beta 0cb8dbdfc5ee73b8cbc200f175f467ebead55201

[RW21]

Probably duplicate of #131 .
I'm facing the same issue with games on the Love2d Engine.

[voidpointer0]

I have the same issue with Unity. The moment I launch it, AeroSpace starts lagging (around 10 seconds to switch between workspaces). Hope this gets fixed, I'm running Unity 90% of the time. Loving AeroSpace otherwise.

Edit: added aerospace version.

Config:
Macbook Pro M1 2020 16GB
macOS 14.6.1
aerospace CLI client version: 0.14.2-Beta
AeroSpace.app server version: 0.14.2-Beta

[jakenvac]

Also getting this when Godot (another game engine) is open. I'll try to investigate this later on today. Think this is also a duplicate of #483

[jakenvac]

Using the accessibility inspector on the Godot window also lags a lot. I'm guessing the hang up in Aerospace is the accessibility API on certain apps. Possibly something to do with OpenGL, but that's just a guess at this point.

[jakenvac]

I can confirm this also happens with Love2D. Although seems significantly more severe with Godot.

Edit: Updating here instead of spamming with more comments.

I ran a hyperfine benchmark of aerospace list-apps with and without godot running. No other changes:

Godot running:

hyperfine "aerospace list-apps"
Benchmark 1: aerospace list-apps
  Time (mean ± σ):     988.1 ms ±  34.3 ms    [User: 6.0 ms, System: 6.1 ms]
  Range (min … max):   892.2 ms … 1008.2 ms    10 runs

Godot not running:

hyperfine "aerospace list-apps"
Benchmark 1: aerospace list-apps
  Time (mean ± σ):      32.4 ms ±   3.9 ms    [User: 3.7 ms, System: 3.0 ms]
  Range (min … max):    24.9 ms …  42.4 ms    73 runs

When navigating between windows, the delay with godot seems to 'stack'. The more I navigate into and out of the godot window, the worse the lag gets until all my window management freezes for a while and then all the buffered inputs execute at the same time.

Hopefully I'll get some time to do some actual profiling on AeroSpace tonight.

-- Update: 2024-09-18

I've done some profiling, but I can't seem to pinpoint the cause of the slowdowns. It does indeed seem to happen when interacting with the accessibility APIs, and this aligns with my experience using the accessibility inspector.

Using Xcode instruments, I've found that the hangs occur during calls to the apis, but the apis themselves don't seem to take more than a few ms to complete, so I'm still not sure what's happening. My experience with instruments is limited so I might be missing something obvious.

One thing that has become apparent though is that, if possible, some of the accessibility logic could be taken off the main thread - Any slowdown in AeroSpace can completely halt further usage and even lock up the system because it's trying to catch up with switching windows, listen to inputs etc.

--

Okay so I've found at least one of the problematic accessibility calls. While inspecting a 4 second hang, I can see a call to AXUIElementCopyAttributeValue that takes 1.87s to complete. That's half of the hang accounted for at least.

I think that another problematic call is NSWorkspace.shared.runningAppications combined with the filter that checks the computed property activationPolicy. This is used when iterating over all of the manageable apps. It's not as impactful, but as activationPolicy is 'called' in a closure during an iteration, it adds up.

--

Not really discovered anything more, other than running godot in headless mode also causes the lag (maybe worse?), despite not having a window.

[jakenvac]

I've found a workaround for Godot, maybe there's something similar for Unity. In the godot editor settings, under Interface -> Editor, right at the bottom there is a Update Continuously checkbox. Enabling this seems to solve the issue.

No idea why.

Also seeing this slowdown with the Bevy game engine, too - I've spent an entire day researching this and the most I've come up with is that these game engines are running the game loops on the main thread, which is also where a traditional window loop (that answers to the AX api requests) runs, so it ends up competing and the AX api's have to wait until the app has time to respond. But I've not been able to confirm any of this.

[voidpointer0]

There's no equivalent for that in Unity, at least none that I know of. If anyone manages to find a solution, please do let us know in this thread, as AeroSpace becomes unusable for me when Unity is launched.

[jakenvac]

I don't have unity installed to test, but I asked the oracle gpt and it says unity has some similar settings:

Have a play with these settings and let us know how you get on. I basically toggled every setting in godot until I made progress 😄

[voidpointer0]

Disabled the potential performance-affecting settings (like Application.runInBackground) that weren't disabled by default, but still no luck.

Edit: Edited for more clarity.

[nikitabobko]

Okay so I've found at least one of the problematic accessibility calls. While inspecting a 4 second hang, I can see a call to AXUIElementCopyAttributeValue that takes 1.87s to complete. That's half of the hang accounted for at least.

@jakenvac on your profiling screenshot the hang happened in the disableAnimations. disableAnimations is a suspicious thing to circumvent animations that might appear in random situations #51 12e7737 (primarly when macOS built-in accessibility features are used, but not necessarily)

Blind guess. Does it become better for Godot (or whatever game engine you're testing on) if you drop disableAnimations?

diff --git a/Sources/AppBundle/tree/MacWindow.swift b/Sources/AppBundle/tree/MacWindow.swift
index a17a181..6d9e0df 100644
--- a/Sources/AppBundle/tree/MacWindow.swift
+++ b/Sources/AppBundle/tree/MacWindow.swift
@@ -149,10 +149,8 @@ final class MacWindow: Window, CustomStringConvertible {
     }
 
     override func setSize(_ size: CGSize) -> Bool {
-        disableAnimations {
-            previousSize = getSize()
-            return axWindow.set(Ax.sizeAttr, size)
-        }
+        previousSize = getSize()
+        return axWindow.set(Ax.sizeAttr, size)
     }
 
     override func getSize() -> CGSize? {
@@ -160,9 +158,7 @@ final class MacWindow: Window, CustomStringConvertible {
     }
 
     override func setTopLeftCorner(_ point: CGPoint) -> Bool {
-        disableAnimations {
-            axWindow.set(Ax.topLeftCornerAttr, point)
-        }
+        axWindow.set(Ax.topLeftCornerAttr, point)
     }
 
     override func getTopLeftCorner() -> CGPoint? {

[jakenvac]

Thanks for the suggestion, I gave it a try - It does seem to help marginally, but nothing significant.

That particular screenshot I shared does show it happening during the call to disableAnimations, but some of the other long running stack traces during that hang were various other interactions with the accessibility APIs.

I've done a bit more research in the time since I spammed this thread, and I'm still of the mind that it's because these apps manage their own event loops and constantly block the main thread - although this is based on nothing concrete, just bits and pieces I've read or experimented with.

If this does happen to be the case, then I would say it's a not AeroSpace responsibility to work around. But at the very least it would be good to come up with a way to mitigate this blocking AeroSpace from effectively managing non-problematic apps.

Some suggestions for this are:

• Using background threads to dispatch AX requests and avoid blocking the main thread, I plan to do some experiments with this particular approach.
• Implement some timeout behavior
• A config option to specify app bundles that should be completely ignored by AeroSpace (Although this wouldn't fix the slowdown on calls that aren't app specific, such as NSWorkspace.shared.runningAppications, however they're called less frequently so would still be an improvement)

[jakenvac]

For what it's worth, here are some discussions around adding support for improving accessibility support for winit (used by bevy to create a window) and SDL (used by love2d to create a window). Although the discussions being held are a bit beyond my understanding, so might not be entirely relevant.

rust-windowing/winit#2120

libsdl-org/SDL#9351

[nikitabobko]

Using background threads to dispatch AX requests and avoid blocking the main thread

Or using a dedicated thread per application. I don't know if AX API are thread safe. Most probably they are generally not. But I have hopes that AX infrastructure for different applications shouldn't share any mutable state.

Apparently the yabai developer tried it and it worked... koekeishiya/yabai#2377 (comment) so I'm rooting for this approach. I didn't test it though

[voidpointer0]

Any rough estimate when you will be investigating/patching this?
Just asking because I'm using Unity with AeroSpace daily and I don't intend to go back to Yabai.
Might switch to PC for Unity work until this is fixed if we're talking months rather than days.

[jakenvac]

I can't speak for @nikitabobko but It's not an insignificant amount of work in my opinion. I think it will take some time. And as for my attempt, I've not had time to play with threads and the AX api yet, let alone implement a proper fix.

[voidpointer0]

Yeah, looked like it. Thank you for the swift response. At the moment I'm just restarting Unity several times per day. There's one upside to the slowdown - it prevents me from getting distracted from work, haha.
I've waited years for something of AeroSpace's quality on macOS, so I'll gladly wait more, especially as the beta works great for me otherwise.

[nikitabobko]

I share @jakenvac vision, it's not going to happen soon

[jakenvac]

Just for the sake of documentation and if anyone else wants to have a go at tackling this, I've ran into another app this happens with, the iOS simulator. Going forward I'll maintain a list in this post rather than constantly notifying everyone subscribed to this issue.

• iOS Simulator
  • introduces a noticeable delay, but manageable as long as you don't spam your hotkeys.
  • I don't know why (or if it's just in my head) but seems to be less noticeable if you float the simulator window using a callback
• Godot
  • By default introduces significant delay that can essentially break aerospace as keybinds get extremely backed up and execute rapidly when finally caught up
  • Can be mitigated by enabling the update continuously editor setting, but the issue still occurs when you actually run the game
• Love2d
  • Fairly noticeable delay, but not breaking if you're patient with your keybinds
• Unity
  • I've not tried this personally, but as the issue title suggests, introduces a ~3s delay

[PawKanarek]

Just a thought: can we somehow exclude Godot process from AeroSpace?

[PawKanarek]

I naively tried ignoring godot window by ignoring it in tree nodes discovery:

extension TreeNode {
    private func visit(node: TreeNode, result: inout [Window]) {
        if let node = node as? Window {
            if node.bundleIdentifier == "org.godotengine.godot" {
                print("ignording godot")
            } else {
                result.append(node)
            }
        }
        for child in node.children {
            visit(node: child, result: &result)
        }
    }

This led of course to unexpected behaviour like Godot window not hiding when navigating between workspaces. Lag was reduced like by half, but was still present.

[vegetablesalad]

When Android studio is hogging CPU/RAM then I can forget about switching screens. It can take up to 20 seconds.
Running games (Factorio for example) slows it down to about 2s.

[gamebox]

I'm seeing the same thing with Steam running in the background on both 0.14.1 and 0.15.2. Went from 100ms-3000ms range for aerospace list-apps to 30-60ms after closing Steam