[RFC / WIP] Preview of new audio backend API integration for frontends

[bsmiles32]

This is just for preview and I will very likely changes things in the future but
this PR shows what could be done to integrate audio backends in the console-ui frontend.

See commit message for basic description of this work.

There are still issues with backend selection persistance, robustness (ie recieved limited testing),
and no plugin were effectively ported, but I will wait for some feedback on the overall architecture before going further.

[Narann]

Wow, this reveal the amount of code needed for backends.

Once again I push for a backend folder in src folder.

In a general way, here is how I see things in general (not only in console-ui), do as you want, it's just suggestions:

src/
 backend/
  audio/
   factory.h/c
   dummy.h/c
  input/
  video/ # I know... I know...

[bsmiles32]

updated:
-audio bakcends are now in backends/audio folder
-audio_backend_factory has been generalized -> backend_factory
-uses the new frontend/core interface (eg via a DoCoreCommand).

[bsmiles32]

update.
This is the first working version of the new audio backend.
It mostly consist in a minimal port of the sdl audio plugin and therefore is still not ready for prime time. However, most of the architecture is in place for wider consumption.

The basic idea is :

• except for the dummy audio backend, all audio backends are optional. This way we don't force unwanted dependencies on users/downstream projects as they are required to enable backends at compile time. So to test the sdl audio backend you will have to enable it with the HAVE_SDL_AUDIO_BACKEND=1 Makefile variable.
• users gets to choose at runtime which backend to use (among those available). For now there is only 2 backends : dummy or sdl. Note that, the user can get a list of available backends by typing mupen64plus --audio list.
• Resamplers follow the same pattern and be {en,dis}abled at compile time depending on the wanted dependencies and be selectable via the config file (in the future). This way they can be shared between all the audio backends. For now, only the trivial resampler has been ported.

This flexibility should allow downstream projects to "customize" the ui-console to match their needs as they can disable unwanted dependencies and only enable what they need. All that without having to rewrite a full frontend, just toggle some compilation options.

Note: to compile, you need to have compatible m64p headers (ie from the https://github.com/bsmiles32/mupen64plus-core/commits/new_audio_api branch).

[littleguy77]

@bsmiles32 Cool :)

So just to clarify the big picture, if a downstream project no longer wanted to use the audio plugin system, and they did not want to use the sdl-based audio backend shown here, then they would either

• Implement their own front-end (i.e. not use mupen64plus-ui-console)
• Fork ui-console and compile their own backend into the ui-console binaries
• Use ui-console as-is, and dynamically link to their own backend at run-time

Is the third option possible? Or even desirable?

[bsmiles32]

Implement their own front-end (i.e. not use mupen64plus-ui-console)

Yes. you can do that. You just have to implement an audio backend and connect it to the core AI.

Fork ui-console and compile their own backend into the ui-console binaries

That's also an option. Furthermore, depending on the usefulness/quality of this new audio backend you might want to upstream it (even if it is disabled at compile time). That way, you don't have to maintain the backend in your fork as upstream takes care of that for you. Other downstream projects can benefit from your backend.

Use ui-console as-is, and dynamically link to their own backend at run-time

Not really possible as ui-console will eventually remove the requirement on dynamic libraries.
Indeed, dynamic libraries only make sense if they can be truly shared. The thing is that no other core follow the m64p plugin API, so there is no real reason to require a dynamic library here.

Note that if you develop a full new frontend, you're free to do whatever you want in order to provide an audio backend. You don't have to provide that much flexibility and can hard-code the backend that make sense for your project.

The ideal plan for the ui-console is to become the "standard" frontend for any platform.
What I still need to solve is how to easily "share" all the backends with general GUI frontends (such as M64Py). The thing is that it may or may not be easy depending on the assumption made in backend and frontend implementations. And I as move more responsibilities to the frontends, writing one becomes more complicated. Maybe we can proceed with 3 layers :

• libmupen64plus - this is the core, no strong dependencies, full control over emulation
• libm64pfrontend - ease the writing of generic frontends - somewhat customizable - provide several generic audio,input,video,resamplers... backends. can make some assumptions if they are largely accepted, can have some optional dependencies.
• mupen64plus-ui-{console,python} - generic frontends that are linked to libm64pfrontend

That way, writing a generic frontend is easy and backends written in libm64pfrontend can be shared.
And for frontends that need more control over emulation, they can directly use the libmupen64plus and code suitable backends.
We should also make sure that both libmupen64pls and libm64pfrontend are buildable both dynamically and statically so that "integrators" can choose they're preferred way of distributing a m64p solution (ie on desktop go will dynamic libraries, on embedded platforms just distribute a single executable with statically linked libs).

Not sure if it makes sense / simplify things, but that's the only solution I have so far :( I am open for suggestions.

[littleguy77]

@bsmiles32 Thanks, I like your ideas. I was assuming that dynamic linking wasn't in the plans, but wanted to confirm. I agree with your rationale.

The three layer stack sounds good. Right now mupen64plus-ae does does some "island-hopping" between modules to get from java ui land to the mupen core.

             /-> (jni wrapper) <--> mupen64plus-ui-console
 (java ui) <-+-> (jni wrapper) <--> mupen64plus-core
             \-> (jni wrapper) <--> SDL2

With your proposed architecture, I'm imagining we could simplify this to

 (java ui) <-+-> (jni wrapper) <--> libm64pfrontend
             \-> (jni wrapper) <--> SDL2

which would allow us to clean up some of the downstream-specific hooks/hacks such as

mupen64plus-ui-console/src/main.c

Line 92 in f89efa2

#ifdef ANDROID

mupen64plus-ui-console/src/main.c

Line 616 in f89efa2

/* Allow state callback in external module to be specified via build flags (header and function name) */

If the SDL2 dependency were ever to be removed we could simplify this further to

 (java ui) <--> (jni wrapper) <--> libm64pfrontend

at which point we might even consider keeping the jni wrapper upstream (e.g. in libm64pfrontend) if someone wanted to create alternative android or java-based front-ends. Probably getting ahead of myself, but thought I'd share some big-picture possibilities with your proposed architecture.

[bsmiles32]

@littleguy77 : Regarding your big plan it all depends on what direction you want to take with your frontend. If the generic libm64pfrontend (name open to suggestion of course as I suck at names choosing) suits your needs your plan should work as expected (ideally simply wrapping libm64pfrontend). However, if the generic frontend approach doesn't work for your frontend, you will have to interface directly with libmupen64plus and develop the appropriate backends. It all depends on your use cases.

The same approach of easing development of simple apps with a helper library, while still allowing advanced usage via direct interfacing with core library as been used by other projects (OpenGL/{GLUT,SDL} for instance). Plus it should help reducing dependencies in the core by moving them into the frontend lib.

Note that I've never written a m64p frontend and I don't know all the difficulties involved writing one, so I might not be the right person for deciding what should be done here. In this work I only try to move some dependencies out the core and ease the writing/sharing of audio (and eventually input and video) backends.

[Narann]

Name suggestion:

Project name: mupen64plus-frontend

Lib name: libmupen64plus-frontend.so

This seems to be consistent with the rest.

My 2 cts.

Keep the good work! :)

[richard42]

I don't like this idea of adding another library in between the core and the front-end, on the basis of adding too much complexity. It would have to be documented and versioned and make the whole system that much more complicated, for very littly benefit. I would rather see Mupen64Plus-AE just fork the UI-console and customize it to their platform as necessary. In the past few years there hasn't been very much work done to ui-console anyway, so they're not likely to be at risk of diverging greatly from the 'reference' front-end.

[bsmiles32]

@richard42 : Fair point. My concern was more about how to share the various backends (audio, resamplers, input, ...) between ui-console and say m64py which doesn't depends on ui-console. Because now with audio and input plugins, m64py doesn't have to implement those. But if/when we deprecate plugins and use "backends" instead, m64py will have to implement those somehow since backends are tied to the ui-console. But maybe that is not really an issue... and duplicating/copying some code between the 2 projects is easier than doing a full library.

I'll keep working on that audio stuff some more to reach feature parity with the sdl-audio-plugin and update/rebase this PR accordingly.

[richard42]

For the audio, I would also recommend leaving the resampling in the core. The front-end should only be responsible for setting up the hardware and delivering the final audio samples.