Receiving and processing midi events when the app is not in focus

[SNewtown]

Hi,

My app (which receives and sends midi SysEx messages to a device) is working amazingly well thanks to MidiKit, can’t begin to say how useful this library is.

That said, I was wondering if it’s possible for the midi messages to continue to be received and processed while the app is not in focus. I tried adding the background task capability which adds the option for the app but it doesn’t process inbound midi when the app isn’t in focus. I know some midi monitor apps can still receive and log midi messages when the app isn’t in focus so there must be a way to enable this?

Thanks,
Stéphane

[orchetect]

If enabling this entitlement doesn't work, then there must be an additional entitlement or API call needed that I'm not aware of. I'd try Googling to see if there's a solution and I can add it to the documentation.

[SNewtown]

Hi Steffan, I had tried Background Processing and Uses Bluetooth LE Accessories but not Audio, Airplay and Picture in Picture, which I just tried now and it doesn’t seem to do it. I also tried all three together. I will do more research and let you know if I find a solution. I use midi pedal assignment to turn effects on/off and this works great when the app has focus but won’t register when the app doesn’t have focus. Not a deal breaker, but would be nice if it kept processing inbound messages while not in focus. Cheers, Stephane

…

On May 22, 2023, at 10:47 PM, Steffan Andrews ***@***.***> wrote: If enabling this entitlement doesn't work, then there must be an additional entitlement or API call needed that I'm not aware of. I'd try Googling to see if there's a solution and I can add it to the documentation. <https://user-images.githubusercontent.com/11605557/240105521-e161fcfa-cbc3-43bb-af4d-68318ccd2829.png> — Reply to this email directly, view it on GitHub <#187 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7NVJGBUCX2L6VWPVGDQ5NTXHQQMHANCNFSM6AAAAAAYLEUGOY>. You are receiving this because you authored the thread.

[orchetect]

From googling it seems like it used to require a hack where you set up a dummy audio stream to convince iOS to keep the app alive. But looks like that stopped working around iOS 13?

I'm not sure there's a simple or clean way to achieve this because if your app is receiving MIDI, it's also doing other things in response to that MIDI (ie: playing sounds? performing some action?). So it may involve a combination of entitlements and/or API calls.

Apple docs:

When the UIBackgroundModes key contains the audio value, the system’s media frameworks automatically prevent the corresponding app from being suspended when it moves to the background. As long as it is playing audio or video content or recording audio content, the app continues to run in the background. However, if recording or playback stops, the system suspends the app.

[SNewtown]

Yes, I found the same… something about setting up an audio port (found it on an Audio Bus piece of documentation). I also reached out to the developer of Midi Wrench as that app does receive (and log) inbound midi messages and maybe I can get some insight in how that was accomplished. But if it’s going to be too much of a hack, I’ll just drop it, was just hoping it could be done. Will let you know if I find a way or end up dropping it.

…

On May 24, 2023, at 7:58 PM, Steffan Andrews ***@***.***> wrote: From googling it seems like it used to require a hack where you set up a dummy audio stream to convince iOS to keep the app alive. But looks like that stopped working around iOS 13? I'm not sure there's a simple or clean way to achieve this because if your app is receiving MIDI, it's also doing other things in response to that MIDI. So it may involve a combination of entitlements and/or API calls. — Reply to this email directly, view it on GitHub <#187 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7NVJGE54T55RC3W2R5F2ODXH2OC7ANCNFSM6AAAAAAYLEUGOY>. You are receiving this because you authored the thread.

[SNewtown]

OK, got this working... finally...

So you had this class (along with AVFoundation framework)

import AVFoundation

class BackgroundAudioManager {
    private var audioPlayer: AVPlayer?
    
    init() {
        do {
            // Activate audio session for background playback
            try AVAudioSession.sharedInstance().setCategory(.playback, mode: .default, options: .mixWithOthers)
            try AVAudioSession.sharedInstance().setActive(true)
            
            // Create an empty AVPlayer item
            let playerItem = AVPlayerItem(url: URL(fileURLWithPath: ""))
            audioPlayer = AVPlayer(playerItem: playerItem)
            audioPlayer?.play()
        } catch {
            print("Error initializing background audio player: \(error.localizedDescription)")
        }
    }
    
    func stop() {
        audioPlayer?.pause()
    }
}

and then in your @main, you instantiate the background manager with:

let backgroundAudioManager = BackgroundAudioManager()

and finally you turn it off on quit:

func applicationWillTerminate() {
   // Stop background audio when the app terminates
   backgroundAudioManager.stop()
}

And you only need the Audio, Airplay, and Picture in Picture app entitlement enabled with the above.

Hopefully that helps!

[SNewtown]

Thanks for the improvement suggestion! So you would call the start/stop on entering and leaving background mode? Wasn't sure if this would work or might entail missing midi messages, but I can surely try it! But I have to admit that I was also concerned with overhead of this running throughout so I like the idea of turning on/off only when required.

Thanks again!
Stephane

[SNewtown]

Works like a charm with this on the @main ContentView:

.onChange(of: scenePhase) { phase in
    switch phase {
    case .active:
        // App is in the foreground
        backgroundAudioManager.stop()
    case .inactive:
        // App is transitioning between foreground and background
    case .background:
        // App is in the background
        backgroundAudioManager.start()
    @unknown default:
        // Handle any future unknown cases
        break
    }
}

and this as the scenePhase variable:

@Environment(\.scenePhase) private var scenePhase

Thanks again!

[orchetect]

So you would call the start/stop on entering and leaving background mode? [...] I was also concerned with overhead of this running throughout so I like the idea of turning on/off only when required.

Yes, if it indeed works to have the audio run only when in background then that would be the most efficient.

Works like a charm [...]

Great to hear! Appreciate the testing on your end.

[orchetect]

I've summarized the code and added a guide to the MIDIKit docs here.

[SNewtown]

Perfect and well done!

[orchetect]

I will update the docs to say that it's not a workaround but should be used when/if MIDI triggers audio samples or audio playback. That would be appropriate for passing App Store review.

I think you're going to have to figure out based on what you're trying to do in response to MIDI, what background mode or entitlement is appropriate. It may involve some research and trial & error to see what works. But Apple doesn't like entitlements purely for the sake of a workaround.