Discussion: OSVR on Android - server auto start vs shared service app

[JeroMiya]

Currently on Android, OSVR has an implementation of the server auto-start API that starts a server in-process with the client app. The server runs in its own thread. It does not require a java layer to initiate - the same C++ code on Android will work on PC/Mac/Linux.

Chase Cobb from Razer recently announced some work he's done on an OSVR service that handles the OSVR server:
https://www.reddit.com/r/OSVR/comments/4iq4vz/osvr_android_update_02102016/

This issue is to discuss the two approaches and get community feedback, as well as to document decisions going forward. The current plan for OSVR on Android is for client apps to use the auto-start API as a black-box to ensure the server is running. The implementation would be to start the server in the same process (preferably using a memory stream instead of IP looback for the VRPN connection) in its own thread. The server would then look in a shared location, or use a binder, to access a shared server configuration if it exists, or fall back to a reasonable default configuration. The benefits to this approach are that the same C++ code works on Android as it does on PC/Mac/Linux. It also keeps client applications self-contained and stable - only optionally taking advantage of a shared configuration. It's more friendly to users, who are no longer required to download a separate app to get a VR app to work. It's more battery friendly - if you close an OSVR app, you can be sure that the server isn't still running in the background.

The shared service/app is the alternative. In this approach, the server runs as an Android service. It requires client apps to use a Java layer to initiate the connection to the server. OSVR apps depend on the OSVR app/service being installed to work, and must require the user to download the app if not already installed. The OSVR service app can be updated independently of client applications, which may break existing applications not tested on newer server versions. Depending on how the scheduler is tuned on a given Android device (they all differ in this regard), the background service may be punished if it takes up too much resources relative to the current foreground app, which may result in additional latency and hitches. The imaging plugin for Android cannot work with a shared service/app as it currently uses a special in-process-memory VRPN message type that only works when the server is in the same process as the client. The binder API is not intended for very large amounts of data, so the imager cannot be adapted to run over the binder interface.

I am heavily leaning towards the original plan - the in-process server thread started by the auto-start API. Feel free to discuss.

[chase-cobb]

Chase here, from Razer, providing my two cents. Apologies for the incoming wall of text. :)

Having the community weigh in is a great idea! Now let’s break the approaches down and provide as many facts as possible. I would like to add as a side note that both approaches below have an “auto-start” component to the proposal, but they handle the implementation in different ways.

Auto-start API (JointClientKit Approach?)
Coupling the server in the same process as an application retains the overhead of server maintenance without exposing the benefits of the client/server architecture to developers. On other platforms (PC/Linux/OSX), the thing OSVR does well is provide an environment where developers can benefit from improvements to the underlying server without forcing recompiles. This means end-users don’t have to constantly update client applications and developers can be confident in the client API provided by the OSVR client kit.

Another pitfall for the JointClientKit approach, especially on mobile, of the server running in the same process (which can be duplicated by backgrounding an app and starting a new one) is that it will consume more resources (compute/power/thermal/memory) than a single instance running in as a service. This also means it will duplicate data in a user’s device storage and possibly reduce the number of applications they can install.

A couple things previously mentioned that are worth discussing :

• The server would then look in a shared location, or use a binder, to access a shared server configuration if it exists, or fall back to a reasonable default configuration.
  • This is already handled by the Bound OSVR Android Service approach and is not exclusive to the JointClientKit approach.
• The benefits to this approach are that the same C++ code works on Android as it does on PC/Mac/Linux
  • However, this approach must be verified on Android devices without root and permissive SELinux permissions. It is my understanding that SELinux issues, and non-standard USB mount points/permissions, will prevent the USB stack from working in a robust way. Android’s USBManager must be used in order to provide a working solution that developers can depend on across Android versions. This implies that we must provide something like a client-side java interface for USBManager and maybe a new hidapi implementation for Android.
  • This approach seems to have the exact same implications on the core c++ server codebase as the bound service approach.
• It's more friendly to users, who are no longer required to download a separate app to get a VR app to work
  • It is trivial to detect a missing service and request that a user download it. This is very easy to do in Android and from an end-user experience it results in < 5 taps on the screen.
• It's more battery friendly - if you close an OSVR app, you can be sure that the server isn't still running in the background.
  • This implies that server can’t be stopped from running in the background if it is running inside an Android service. Our prototype implementation demonstrates that we can indeed stop the service.
  • Having the ability to run multiple server instances regularly is not more battery friendly.
  • The OSVR Android service can be started and stopped by any application and abides by Android lifecycle standards. In the video that I linked to, you can clearly see the toasts that are presented as the Service starts and stops automatically.

Bound Android Service
Keeping the server running as a shared object that is encapsulated by an Android service is the most performant and flexible approach. It retains all of the benefits of the client/server model, while providing the core benefits from the JointClientKit approach.

I’d like to make a few more clarifications to points previously made :

• It requires client apps to use a Java layer to initiate the connection to the server.
  • This layer is very thin and can be made invisible to the developer.
  • As I previously mentioned, both approaches will require this for USB reasons at the very least.
• The OSVR service app can be updated independently of client applications, which may break existing applications not tested on newer server versions.
  • This is very true, and it would force OSVR to improve its testing processes, versioning approach, and release process. This seems like a good thing.
  • This is only a problem if OSVR changes existing functionality underneath developers. We should be avoiding this, or at the very least constantly testing for regressions.
  • With either approach developers are going to face this issue. Even in the JointClientKit approach, developers are going to upgrade their server to get new features. If existing features are changed, or regress, developers will be affected. Neither approach is regression-proof.
• Depending on how the scheduler is tuned on a given Android device (they all differ in this regard), the background service may be punished if it takes up too much resources relative to the current foreground app, which may result in additional latency and hitches.
  • This is taking things to an extreme level but I’d like to play devil’s advocate. If a given device can’t run one instance of the server without hitches how can it be expected to potentially run multiple instances?
  • In the service model the server runs in its own thread inside the service, and the service has no other logic. This should be a non-issue.
  • Do you have an example of an Android Service application that displays the behavior you’ve described?
• The binder API is not intended for very large amounts of data, so the imager cannot be adapted to run over the binder interface.
  • While the binder does have limits on data size, images can certainly be passed.
  • Just because there is a binder implementation doesn’t mean we are bound by it in the way we move data. (Forgive the pun)

Both approaches are similar at their core, as I see it. The key differences being :

• Efficiency
  • As the server is currently constructed, the Service approach with a single server instance running will scale more efficiently potentially consumes far less resources (compute/power/thermal/memory) .
• Developer overhead
  • The JointClientKit approach requires developers to recompile whenever new server releases come out that they would like their application to benefit from.
  • With the bound approach, developers never have to compile the server.
  • The bound service approach means that developers only recompile when the client API changes.
  • Regressions in base functionality will affect developers in either approach. We must be more diligent with the release strategy.
  • Both approaches seem to point out the need for more robust client/server versioning and test strategies.

I am definitely in favor of the bound service approach on Android. In my opinion, it provides ease of use for developers and end-users. They only need to worry about the client application and we take care of the rest. It is also the most future-proof, since it uses Android APIs to sandbox, manage permissions, and handle resources that may very between devices and android versions.

It is also worth clarifying, are you suggesting the move to JointClientKit for all platforms? If not, how should a developer treat the same client code when moving between desktop and mobile platforms?

[JeroMiya]

Thank you Chase for taking the time to post and for working on this. To clarify, what I am advocating for is sort of a hybrid model. I think it is very important that OSVR applications work independently of each other and of any shared applications we might deploy (e.g. for configuration). It should also be easy for developers to integrate into their app - as easy as the Cardboard SDK is now.

Although I understand the desire to make OSVR-Android more like the PC/Mac/Linux version in terms of having an independent server app, I think you vastly underestimate the negative effect of a separate app hard requirement would have on any OSVR apps - both from users and from developers. It's important to remember that Android is not like a desktop operating system, and you need to think of the ecosystem differently. I can tell you with certainty that few experienced or high-end Android developera are going to go for a separate app requirement. The vast majority of users simply will not go through the hassle of installing another app after installing your app. They'll just downrate the app, write a nasty comment, and then uninstall your app without playing it. It just isn't done in serious apps. The best you can hope for is an optional requirement - the way that the Cardboard SDK works. That is, your SDK works just fine on its own, but may be enhanced by another app that provides shared services or configuration.

Secondly, a few clarifications/responses to your post:

• The size of the extra libraries required to bundle the server functionality is pretty small. It's only a few kb.
• The point about regressions is that the separate app can be updated in a potentially breaking way before the app developer has had a chance to update or even test their app with the new server. It's not just breaking changes - maybe it's a performance difference, or a latency difference, etc... We can't test every OSVR app every time we release an update. Bundling the server libs with the app means the developer controls when to update the server libs and gets the chance to test before the update goes through. They don't have this chance with a separate app. This is a HUGE deal with any major developer releasing a serious game/app and I can tell you from experience this would make OSVR a non-starter for a LOT of devs.
• The imager interface isn't just single images, like from a camera, but live video. That's about 20-30fps of 1080p or higher video coming through in some cases. Shoving that data through the binder interface would require at least two extra copies of all that data (one to get it into the stream and one to get it out of the stream). Even recent android hardware wouldn't be able to keep up. It would also saturate any "pipe" no matter the implementation - that's one of the reasons why we use a shared-memory implementation on PC and in-process memory messages on Android. In both cases, all that's sent over the VRPN connection is a handle/ID for the frame data in memory (no copies).
• There shouldn't be multiple server instances running with either proposal. The idea is that each app would pause when it goes into the background using the traditional Android lifecycle events. Although, now that you mention it, it may mean we need an additional auto-start API function to indicate that the client app is "pausing" to give ourselves a chance to pause the server thread if necessary.
• OSVR is a distributed server architecture - that's what makes it possible to use hydra or vive controllers in an OSVR-Android app. This includes a second server process on the same device, like your android service. So, there's no reason we couldn't support both scenarios, as long as apps can still work without the OSVR service app installed. But that's not the primary use case for OSVR-Android. The primary use case is the same as the Cardboard SDK - we're just going a little bit beyond what Cardboard offers.

Keep in mind that we are competing against three SDKs on android. The first two are the Cardboard SDK and the GearVR SDK. The Cardboard SDK comes in a self-contained bundle of libraries, which also support a shared configuration (to "pair" the SDK with the current viewer). This is exactly what I'm proposing for OSVR by default. The GearVR SDK works more like what you're describing, but in that case the developer can count on the services already being installed, because they have OS/Hardware level integration, which we don't, and because they need it to be installed to download gearvr apps in the first place, which we can't count on. The third SDK is going to be whatever Google provides for AndroidVR or whatever they end up calling it. It will be even easier to integrate because it will be built-into the SDK itself. If our SDK ends up being a pain to use compared to these alternatives, developers simply won't bother.

[cobbchaseRazer]

OSVR has a responsibility to allow third party hardware manufacturers to update their plugins and issue hotfixes. The JointClientKit approach creates complexity for maintenance of plugins for hardware developers. This means they have no control over the perception of their products in this model because hardware plugin fixes, and updates, may never make their way into the applications.

The Google Cardboard SDK is not so different from the implementation I am in favor of. It also requires an additional application, the Cardboard app, in order for an end user to change viewer configurations. It uses the Cardboard app as a service to manage configurations. I have searched the play store for user reviews that needlessly downvoted applications because of the need for an external application, but found none. Also, developers don’t seem to mind it.

The “shared configuration” is a single protobuf file that multiple clients can use to calculate their distortion correction meshes at runtime, and is forced in Cardboard. Cardboard also uses sensor manager (an Android service) to poll the IMU and provide their 3DOF estimations. Cardboard doesn’t work with sensors directly like we do. They also poll Android for screen intrinsics to allow some other functionalities. They work with a few Android services and ship a client library, much like my proposal.

As a quick recap:

• The CardboardSDK is not a solution that is completely encapsulated by their client-loaded SDK libraries.
• Requires external Cardboard app to swap default viewer.
  • Manages the “shared configuration” and acts as a discovery platform for Cardboard content.
  • If it is not installed on device and user tries to swap viewer configuration it takes only 3 taps on screen to install Cardboard application.
  • Doesn’t seem to deter developers or end-users.
• Requires Android Sensor Manager (an Android Service) to read IMU.
  • A regression in Sensor Manager, or incorrect sensor implementation from a device manufacturer, would have a negative impact on all Cardboard applications. It is a solid test and release cycle that allows.
• Cardboard’s “shared configuration” is a single protobuf used globally by client applications to configure only the lens distortion correction, as far as I am aware.
• Cardboard also provides a front-end for users to generate configuration files via a web gui.
  • https://www.google.com/get/cardboard/viewerprofilegenerator/

Given the evidence above, I don’t think there would be a developer/end-user objection to using the bound service model. Cardboard and GearVR both use something similar. It is easy for developers of both hardware and software, as well as end-users.

I also to discuss regressions and our test/release approach. It was never implied that we should test every released OSVR application with each OSVR server release. The reason we don’t need to test every application is that we already have access to the APIs used by those applications, and we can test those. If an application exposes an edge case, then we fix it and issue a hotfix that can be live within just a few hours. At the end of the day, if we are confident in the stability of the APIs this shouldn’t be this big of a concern.

Unless hard facts have been officially released by Google about any AndroidVR projects, lets avoid speculating.

[leemichaelRazer]

The technical points outlined above by Chase are valid and should be examined closer. Behind it all, however, I think we might want to get back to the core of the discussion, which is the architectural implications. As I see it, there are a few different stake holders involved:

• OEM developers who will take the OSVR platform to implement and distribute as part of their core services
• OSVR peripherals developers who wish their devices to work with all OSVR instances
• Application developers
• End users

What works best for this collective set of stakeholders and how will we know if we’ve done the right thing (as sometimes our personal views don’t necessarily represent those of the target – an easy trap to fall into) is best determined by collecting some data.
Here are some selected points to consider:

• OEM developers – these parties can take a service and install it as part of their core image. Further they may indeed be able/want to introduce detection methods to ease setup of the platform. With this, they can update and distribute whenever they choose to take advantage of any server changes and also ensure that all OSVR apps will always have a quality experience on their platform.
• OSVR peripheral developers – new devices will appear all the time. The promise of OSVR is that once a new device is enabled on the OSVR platform, users need not get a new version of an app to use it.
• App developers – Similarly, as new devices and features become available, the need to re-compile and redistribute will be mitigated.
• End users – Regarding ease of use, installation, etc… I think we’ve discussed several issues already.

In all cases, it might be prudent to get some real requirements from these stakeholders before forming a final position.

Michael
VP Software Razer/OSVR

[JeroMiya]

Thanks Michael, Chase. I think we are starting to align towards a solution that meets everyone's needs. I'll try to explain what I mean:

To clarify, I am not opposed to having optional services and apps, including services from OEMs and peripheral developers that host their own OSVR plugins. For example, there were plans in place to have a shared server configuration file along with an OSVR config utility that enabled users to change server configuration (including adding "external" devices from other OSVR servers, including potentially from a local service on the same device). I am only opposed to making that application required for the app to run at all.

The Cardboard SDK works like this. Cardboard SDK apps are self-contained in this sense. If you uninstall the Cardboard app, cardboard apps continue to function. You can also install cardboard apps without the cardboard app installed. Yes, the Cardboard app is required to pair cardboard apps with your cardboard viewer, and I'm OK with that for OSVR as well (see above comments about the plans for an OSVR-Config app for Android). But, if you don't have the app installed, Cardboard apps will continue to work with reasonable defaults. Some apps prompt users to install the cardboard app if it's not installed when you press the gear button in the middle, but it's optional. Like I said, I'm OK with optional apps that enhance or extend existing OSVR apps (either for configuration or adding additional plugins from OEMs or peripheral developers) as long as they are optional, and preferably auto-configured if possible.

The only plugins I would bundle with the OSVR-Android SDK would be plugins that use official Android SDK APIs and built-in services. For these plugins (like the android sensor plugin or the imager plugin), the Android OS acts as a hardware abstraction layer and hardware OEMs can update their drivers at the OS level. For other plugins, such as those that might talk directly to a device connected locally to USB, the idea is that you'd expose those to OSVR apps as an external device.

For OEM developers and peripheral developers, I think we can meet their needs a slightly different way. Instead of running a centralized OSVR server service, or running multiple OSVR server instances and using the external devices functionality to expose those devices to OSVR apps, you could instead write a kind of android "plug-and-play" system. This would be a plugin that works a little like the external devices functionality already built-in, but would be more automatic and require no pre-configuration.

The idea is that OEMs and peripheral developers could wrap their OSVR plugins in a thin service just like the one you created to run the server (we could provide the boilerplate code - they just need to write an OSVR plugin just like normal). Then, you'd have a global registry that OEM developers could add their background services to. The plugin on the client app side would then dynamically register those devices at runtime and handle talking over the binder interfaces to each of them. This way, multiple stakeholders could install their own custom OSVR plugin services (not just the OEM or the OSVR team distributing the OSVR app in the play store), and there's less overhead than running multiple server instances and talking over IP loopback. I think this represents a decent "hybrid" solution that addresses everyone's needs.

[leemichaelRazer]

The approaches that you are suggesting do represent a possible solution to the matter at hand. However, you are suggesting a departure from the architectural design of the system for the other platforms. If a single server/app approach were the best solution from the user’s perspective for initial deployment, then we would have to seriously consider extending this same approach for the other platforms. Because of this, I am not OK proceeding just yet.

Regarding management of deployment of the peripheral device interface components, distributing these as separate mini-services or even separate components could make the end user’s system much more difficult to manage and we’d have to be very careful how we manage data bandwidth across whatever boundaries. In fact, some of the recent discussions that we’ve been having with device manufacturers have them exploring methods with us to achieve tighter integration for the purposes of lower system latency as well as code management on their side. The message that we have gotten is that ideally they have OSVR to provide the infrastructure to deploy their device interface components without developing and maintaining their own service.

The distribution and update of components and the core server is a complex problem that needs careful attention. Based on the open questions (and design tradeoffs) that are being surfaced, I think that is prudent to have a proper requirements doc and design vetted by the community before any further changes get checked into the main branch. This should be the standard we set for all OSVR submissions.

Michael
VP Software Razer/OSVR

[CoolGames]

I have tried to follow the Android discussion with the main objective being to have OSVR HDK 1.3 run on the attached Razer Forge TV (Android TV) device.
Since this seems to be the best way to have a head mounted display as a consumer product bundle as opposed to XBox, Nintendo, Steam or Playstation and associated incompatible implementations.
Also I am seeing Steam VR with the Steam Controller on PC/MAC/Linux not working on any other platform.
If this is a place to express my feelings on where to go for end gamers and users.
My goal is interactive training using game technologies and not games for entertainment.
This was as we did with our patented expensive CRT video overlay touch devices in the 1980's for military, medical and tourist industries.
The only change now is the mobile market and head mounted VR displays with motion feedback instead of CRT touch on desktop.
Please add this variation when considering architectural decisions.

Lee T. Davy
Cool Games EVENT VIDEO
Retired applications engineer

[JeroMiya]

The aggregation plugin proposal isn't going against the the general architecture of OSVR. It's expected that some platforms may require special circumstances. Implementing those special circumstances as a plugin would probably be the least intrusive and recommended way to implement this functionality, as opposed to, for example, implementing a custom VRPN connection type on Android that uses the binder interface.

Also, there are already other similar plugins that aggregate devices from other sources. For example, the SteamVR plugin aggregates any devices with OpenVR plugins. The multi-server plugin aggregates a few devices that we have "built-in" support for (HDK, Hydras, etc...). So a plugin on Android that aggregates devices from a special central registry (or alternatively from a single external app, if that is preferred given the performance or startup/shutdown requirements - it's up to the OEM), then that is not "out of the ordinary" for OSVR, and wouldn't lead to changes to the way we handle OSVR on other platforms.

[JeroMiya]

@CoolGames Thanks for your feedback! The Forge TV (and other standalone android devices like shield TV or similar custom hardware) is an important use case for OSVR and is something we are actively working on.

[mebalzer]

I have noticed activity @JeroMiya in the builds, but nothing towards an integrated solution that was coming about two months ago as well support for the internal IMU. I have had repeated success with the server and your Android demo on NVIDIA Shield except for an IMU. I was told I could connect to it via a VRPN, but I feel this would not be ideal direction to go for orientation tracking and chose to use a separate USB based IMU with an Android library. Ideally it would be great to be able to plug it in and the HDK 1.3 external IMU replaces internal Google sensors (or in the Shield TV add them) so the headset could be used with Google Cardboard or the upcoming Google VR. In fact it seems to make more sense to just skip the OSVR layer completely and rely on Google VR, which has the ability to talk to other sensors. As mentioned, the screen works great on its own albeit no distortion or chromatic correction, timewarp, or low persistence mode. But none of these are available in the current OSVR Android server either (or at least a month ago)

Anyway, I do hope we are looking at Google VR and if so, let me know and I will be will to help test and implement anything you are working on with Nexus 6P, Pixel C, and towards the end of the month the Shield TV.

[JeroMiya]

Hello @mebalzer, @russell-taylor is currently working on RenderManager for Android this month, and I've done some work on fixing up the C API for RM-OpenGL, so things are coming along on that front.

There isn't currently an android driver for the HDK that makes it appear as a generic IMU to the OS (which would allow the GoogleVR SDK or our own OSVR-IMU plugin to work with it out of the box). What's missing is an alternate OSVR plugin for Android that uses app-space APIs to talk to the HDK over USB. @chase-cobb may be able to comment on the status of that project, if any. The current plugin uses a private API to talk to the USB device, but unless the app is "rooted" the OS will prevent the plugin from talking to the HDK (this is true whether the server is embedded with the client or you install a centralized server app). So for instance, if you wanted to setup a shield TV that works with the HDK, you'd have to build your own custom image with modifications to start the OSVR server as a init.rc daemon with the appropriate SELinux permissions. Obviously this isn't ideal, which is why a more "android-native" approach to USB is needed for the HDK.

As for Google VR: I'd like to incorporate it as a supported OSVR plugin, maybe even replace the current sensor tracker plugin I wrote as a kind of stop-gap. I would welcome community contributions on this - though let me start a new github issue with some details I'd like to flesh out before work begins (such as enabling GVR context sharing between the plugin and the client app, so you can continue to use their audio SDK at the same time as OSVR).

[JeroMiya]

@mebalzer here is the new GoogleVR plugin issue: OSVR/OSVR-Android-Plugins#1

[JeroMiya]

@chase-cobb So, I think we need to come to a conclusion on this decision soon. I was thinking of modifying my earlier proposal so that we can support all of the use cases we've identified. That is, the server auto-start functionality would do the following:

Auto-start logic on Android:

1. If there a local server running (in the form you propose - e.g. a binder interface, or a locally running server on the default port), do NOT start an in-process server - instead allow the client kit to connect to the already running server.
2. If there is not a local server running, no centralized server is installed, etc..., then start an in-process server, as I have proposed.

Client Context Startup:

1. If the binder-interface server is running, connect to it.
2. If a local server is running on the default port and the app is allowed to do IP loopback, connect to the local server over localhost.
3. Attempt to connect to the in-process server, if one is running.

This will allow a single app binary to work in different environments:

1. A custom Android image with an installed central OSVR server that runs on startup as a daemon. This is sort of the all-in-one, custom bundled hardware+software scenario (museum kiosks, arcades, that sort of thing).
2. An off-the-shelf android box (Shield TV/Forge TV) where you can connect one or more HMDs, but you still have an app store - maybe even the official Google Play store. User can update the server and (hopefully) add plugins without apps needing to rebuild.
3. Everywhere the GoogleVR SDK would be used today - phones in cardboard compatible headsets, daydream VR phones and all-in-one daydream VR devices.

[cobbchaseRazer]

@JeroMiya I agree that a decision must be made. Provided the discussion above, I still believe that the bound service is the only way to go without fragmenting the user base.

While there could be a perceived benefit to the JointClientKit model at startup, it also works as a silent failure-case for people that haven’t installed the proper Service. This could easily cause fragmentation in the ecosystem, and removed the ability for new hardware developers to reach the entire OSVR user base. This also has a high chance of preventing software improvements in the server from reaching the full user base and puts the burden of server updates on the client developer.

Also, in the bound service approach, the server only runs when a client application is foregrounded and bound to the Service. It would fail any pseudo-code that checks to see if the server is running before attempting to bind to the Service.

As a quick recap of what has been said so far :

Some of the measurable benefits to the Service approach

• Easy for end-users, hardware developers, and device manufacturers.
  • Easier to provide a uniform user experience.
  • Installs without root permissions.
  • For device manufacturers, this is a complete turnkey solution. Doesn't require any custom BSP work.
  • Provides a solution for working with Usb devices.
    • When using the standard OSVR-Core path, both HIDAPI and Freespace for usb communication are invalid and get SELinux denials at runtime. Using the Android APIs for Usb device management is the ONLY future-proof solution.
  • Can easily install and automatically update from the Play store.
    • It is critical that we have an update strategy that is simple and reaches everyone. The Service approach provides a simple solution that is already an Android convention.
• More resource friendly than each client running an internal server.
• Hardware developers have access to all of the OSVR user base when releasing new hardware. Likewise, users have the most peripheral options for all of their games.
  • Hardware developers are also guaranteed to be able to issue fixes to all released applications if needed. This means keeping the user experience intact across all applications.
• Client applications no longer need to declare OSVR related permissions. The service is properly sandboxed acts as a proxy for permissions and removes the burden from client developers.
• Removes overhead from client applications developers, since they don't have to recompile when server updates are issued.
• Easily add and manage support for other Android services
  • This means that any developer of Android hardware or software that has already built a Service or data provider can more easily add support for OSVR through the Service by way of a simple adapter.
• The Service exposes an activity that can be used as a GUI to help with some of the outstanding issues mentioned below.
  • Configuration management and shared configuration.
  • Discovery of OSVR content for the Android platform. This can be done by adding a meta tag to the client app manifest.
    • Helping users discover content is a win for everyone involved in OSVR.
  • Management of individual plugins.
    • Not all plugins will work on all devices. For instance, not all devices have an internal IMU.

Perceived downsides to the Service approach

• Requires a single, one-time, apk install that can be somewhat automated by the client layer with minimal user interaction

Downloading a single free APK from the Play store is simple and painless. The Service could, alternatively, be pre-installed by a device manufacturer as a turnkey solution. This seems to be less of an imposition that any other platform supported currently by OSVR, and it doesn't sacrifice the scalability of the platform.

[JeroMiya]

A few counterpoints:

• The in-process server is not a silent failure case. First of all, the auto-start API does not give the client any indication of failure, by design, and the client is not aware of which server has started. No matter where the server is running, the indication of failure is the same: either client context fails to start up successfully, or /me/head fails to report. With my proposal, you simply get an extra chance to work (in the case there is no central server but the app is running on a phone with an IMU - a.k.a. 99% of android users) before failing.
• It is not the case that multiple in-process servers will be running simultaneously, so resource consumption will be the same. Application lifecycle management is not relevant to this decision as there is a solution for this either way. I was thinking of adding a "pause"/"resume" API to the auto-start API to allow for more efficient lifecycle management, but this would work for both an in-process server and a centralized server.
• USB SELinux issues are not relevant to this decision. Whether we use an in-process server or a centralized service, we both agree that the OSVR plugins for USB devices (or more likely Bluetooth devices) must be rewritten to use Android APIs to avoid the SELinux restrictions. Such plugins could run in either scenario - in-process server or osvr-server service, so this is not a point in either side's favor. Some plugins may only work when "blessed" by the system image - e.g. in turnkey solutions. In this case, you would install your central service, and the same app would work without modification (or even recompilation), given my suggestion to add logic to check for a central service before starting an in-process server.
• Your centralized server approach does NOT in fact allow new hardware developers to add or update their own plugins because there is only one OSVR-Server service and it is not modifiable by third parties. And, if it were, you'd have a giant security hole because plugins are native code, loaded into the server with no additional verification (not even code signing). Only the maintainer of the service app, if it's in the App store, or the hardware manufacturer, if the service is baked into the device's OS image, have control over what plugins are included or when they get updated. Nobody else. I did suggest above a way for hardware manufacturers to do exactly that, with their own individually updated services and a central registry or auto-detection system. This solves the problem for hardware manufacturers without compromising the security of the server service or individual apps, but there were some concerns about performance when connecting to multiple binder services. And, even if this is a viable solution, it can be used either from a centralized server or an in-process server in the same way, so it makes no difference for the purposes of this decision. And yes, aggregating plugins are absolutely part of the overall OSVR architecture. For example, the SteamVR and OpenHMD plugins are both aggregating plugins.
• The Play store app guidelines do not allow you to use services from other apps to bypass app permissions in your own app. Additionally, the app store is not going to just let us (the maintainer of the OSVR service app) expose protected data like the camera to any app that asks for it, regardless of its own permissions. We would just get rejected (and consequently, under your service-only proposal, break every OSVR app out there). This is only viable for turnkey solutions where the OSVR service is installed as a system level service and thus app store guidelines are not relevant. I concede this is an important use case, and it's accounted for by adding a "check for central server first" step in the auto-start API.
• The centralized configuration GUI is going to be available whether the server is running as a service or in-process, so it's not a point in favor of either side.
• The need for each developer to update their application to get a server update is not going to be a major problem because the plugins included in the in-process server are all going to be using abstractions (e.g. via the Android sensor API, GoogleVR SDK, etc...), and thus will be very stable and relatively unchanging over time.

Also, I failed to mention this before, but there are going to be special cases where you MUST run the server in-process. Initially, this will include GearVR apps (Oculus store requirements dictate you must use their SDK for tracking and the compositor, which we can accommodate with the in-process server), but may include the GoogleVR SDK (client has to share a context for the audio portion of the SDK, and maybe the compositor).

[leemichaelRazer]

Hi All, bringing these issues to the surface is great. However, resolving this topic requires a discussion with the different stakeholders (other partners, product management, and engineering) and we would be better served in a venue where we can collectively review the requirements, design, and implementation. All of these topics must also be treated within the context of the other platform targets to ensure consistency across the board. The decision is best made when we have consensus by as many of the community members as possible. We're listening.

[JeroMiya]

Some developer feedback against a separate server: OSVR/OSVR-AndroidServerLauncher#4

[Bundenth]

Apologies for crashing the party after the lengthy discussion. I got here referred from one of my requests to make the android OSVR server launch automatically with a client application to avoid having the end users running multiple applications.

My two cents is that it is paramount that the end users do as little configuration as humanly possible -ideally they should be good to go with just downloading the client app (game, etc.). I am not sure which of the options here would be preferred, but I just wanted to highlight that the way it seems to work now -having to configure, download and run a separate server on android before running any OSVR app- is tremendously overkill.

From an app/plugin/asset developer's perspective, being able to start the server from within the app seems the most sensible way forward, though I must confess I do not have the full picture in front of me and I do not fully understand the consequences of the server running as a service of the client application. I will spend some time going backwards on this conversation and see if I can offer any useful comments.

Thanks all!

[cobbchaseRazer]

Hi @Bundenth, thanks for the response.

Just a quick word about the service. It only requires a user to install it and they don't have to run it themselves, ever. It starts automatically in the background when a client application starts. If you have used GearVR, it is very similar in approach.

I encourage you to read the entire thread to get a better understanding of both proposals and ask clarifying questions if needed. This is an open discussion. 👍

I'll leave this link to bound services in Android so you can read more, if it helps. :)

[JeroMiya]

Yep, and just to clarify, the discussion has morphed since it started. Right now the two sides overlap. Here's a summary:

Side 1: Bound Service ONLY

• Run the OSVR server as a bound service, contained within a separate app from the app store, independent of any OSVR client apps.
• The OSVR server app can be updated independently of OSVR client apps.
• The OSVR server app's bound service is active only when an OSVR client app is using it.
• OSVR client apps are required to check for the existence of the OSVR bound service app, request the user install it from the Google Play store if it is not installed.
• OSVR client apps do not function without the bound service being installed.

Side 2: Bound Service, if available, else in-process server fallback

• The Side 1 scenario still works, if the bound service app is installed and available, except that the bound service app is now optional. OSVR client apps may continue to work without the bound service app installed.
• If and only if the bound service app is not installed or otherwise not available, the auto-start API starts an in-process server within a thread of the OSVR client app.
• In-process server will work with all standard Android orientation sensors and/or Daydream compatible devices. So, no hardware-specific plugins, generally speaking.

Both Side 1 and Side 2:

• Server configuration is shared between all OSVR apps.
• There will be a server configuration GUI as a separate app. In Side 1 it is required, Side 2 it is optional. Both will use reasonable defaults without manual configuration.
• Can access the entire OSVR supported devices list through the external devices functionality (OSVR server running on a separate machine) without necessarily consuming local CPU/GPU resources those plugins might require (e.g. image processing).
• Possibility of enabling third-party hardware plugins as their own, independent, bound services, given some form of central registry or auto-discovery system. Limited perhaps by CPU/GPU/Memory resources.

[Bundenth]

Thanks for the summary!

It seems there will be a bound service on both options -which is much better than the separate server app at the moment. The only discussion seems to be on whether or not to have the extra option to run an in-process server -for which I do not have a strong opinion, though from a user perspective, having a default functionality without the need of installing is always appealing, but perhaps the cons outweight this single pro.

[JeroMiya]

The bound service will exist in a separate app, not in the OSVR client app. It would work just like the current OSVR_AndroidServerLauncher app, except it will use a more efficient bound service instead of IP loopback. Option 2 with the in-process server fallback is the only way to get out of the separate app requirement. I'll edit the summary to make this clear.

[janoc]

Hello everyone,

I am coming from a bit different background than what has been presented here so far. For us it is important to be able to use an external OSVR server with a mobile client, not having them collocated on the mobile device. We are using Android HMDs and tablets together with external tracking systems and using OSVR to serve also metadata to the application.

I have not seen much attention being paid to supporting this use case yet and, in fact, there is my still unmerged but closed (?) pull req for it: #331

While I recognize that hosting an OSVR server directly on the device is useful, there are also plenty of use cases where it actually complicates things, especially with the complicated configuration from a file - e.g. how can the user application tell the server which configuration to load, for example? Two applications could need different aliases, coordinate transformations or talk to an external server. There is no API for that at the moment - everything for every possible application has to be either hardwired in the config file or autodetected at runtime (impossible for many things). So if the server is encapsulated on the device as e.g. a service, this will need to be addressed.

Many Android devices like GearVR or tablets have limited external connectivity (and most hw doesn't talk Bluetooth), so assuming that any controllers/peripherals or trackers will be directly connected to the device and autodetected is very limiting. Another common situation is that the middleware needed to make a certain device work is supported only in Windows (SDKs ...), so making a plugin for Android is not even an option. In these cases talking to an external server that combines the various devices is much simpler than trying to configure the internal one for relaying the data from the external one (and to uselessly add latency and complexity while losing information).

Can this use case be considered as a first class citizen, please?

[JeroMiya]

@janoc Thanks for contributing! A couple of points:

The osvr_server_config.json is not intended to contain application specific configuration, including aliases or transformations. The user needs to be in control of this at the server level, not the application level. My recommendation for this at the application level is to introduce a layer of abstraction where you can setup your own aliases and transformations on top of what comes from the client kit natively. Maybe this is something we could support at the client kit level? It's possible. It's a little out of scope, maybe.

OSVR was intended to support distributed servers through the external devices mechanism. When you configure an external device, the client kit connects to that server directly, bypassing the local server. In your case, your local server would have no plugins, and you can set your polling time super high, so that the OSVR server consumes no CPU resources.

As for bypassing the local server completely and connecting to a remote server, I think I remember there being an undocumented workaround for this involving environment variables but I don't remember the details. We don't recommend this for most users, so it's sort of a special one-off workaround. If that's not working on Android, we can maybe look into a different way to do it (a server config option perhaps).

[janoc]

Hello Jeremy,

I think you didn't understand my point about the server config. My point is that if the configuration is managed on the server that is running as a shared service, then the application needs to be able to tell the server to load or change to a specific configuration (e.g. containing an external device connection or some parameters). Right now there is no way for the client to tell the server to do so and the application is at the mercy of whatever the server has configured or autodetects. That is like demanding that 3 games from different vendors agree on a common way of configuring I/O - ain't happening if one expects milimeters and Y-up and another meters and Z-up, for example.

Re external devices - this works unfortunately only at the level of VRPN connections. As far as I know, there is no way to fully forward the OSVR connection to an external OSVR server (preserving the access to the device tree metadata, aliases, etc. of the remote server). That is of limited utility.

The undocumented workaround is the OSVR_HOST env variable, but that is that - a kludgy workaround. Unfortunately there is no official, non-kludgy way of doing it (apart from using the low level API which exposes it, for some reason), so we are maintaining a local set of patches adding this functionality. It does work on Android, but this kind of basic thing should be supported out of the box, IMHO. OSVR strives to be a superset of VRPN, so it should at least support the most common VRPN use cases properly.

[JeroMiya]

@janoc Yes, I agree that the OSVR_HOST env variable is a little kludgy. If we're going to support a particular scenario (e.g. telling client kit a specific host/port to connect to for the server), it should be through API or configuration and not hidden workarounds, and then use documentation to tell developers "warning - this API is for a specific use case, for all other usage, do this other thing".

Regarding configuration swapping, I should have clarified - server configuration selection (or live switching) is outside of the scope for the ClientKit. That's a bit more server/client coupling than I'm comfortable with and it could potentially cause havoc. That doesn't mean you're out of luck, but it does mean you'll need an alternative means to implement the switching or live reloading. Perhaps a supervisor process on your server machine that responds to requests over the network by starting or restarting the server with a specific configuration. Then your client applications can make those requests prior to initializing the client context.

[janoc]

@JeroMiya Well, the host/port is supported already, all it needs is a trivial patch in the ClientKit which I have submitted some time ago to expose it. Re documentation - I think the design philosophy behind the server and how to use it for various use cases should be better documented in general. That could potentially avoid some of these discussions.

Re config switching - perhaps it shouldn't be in the ClientKit itself but a separate specialized API, no problem with that. However, an unified framework that would permit the client (or even a completely separate application - think control panel/dashboard) to demand some control things from the server would be very useful to have. I am thinking about stuff like loading of a specified config or asking the server to restart, for example and also to make sure that e.g. the server gets restarted if it crashes or is registered to start on boot (e.g. registering as a service on Android and in Windows, SystemD in Linux, etc).

This is especially useful for people writing portable applications between desktop and e.g. Android. Abstracting the platform-specific complexities (e.g. process or config management) would make the life of the developer a lot simpler and help prevent the inevitable emergence of various hacks because the most common uses will be covered. It is one of my major beefs with deploying VRPN - good tool, but this type of "practicalities" were not handled at all and required various ad-hoc solutions.

[ashleydb]

Hi,

I was directed to this thread by OSVR DevRel. I'm new to OSVR, but if you are looking for input here are some thoughts from a developer's POV, in no particular order.

• Dex limit. We hit this on several Android projects and end up having to scrape stuff out of our builds. I'm not sure how big the OSVR library is or whether the external app approach would actually make much difference here, but in general huge libraries are a concern.
• External apps on non-Google Play devices. There is mention above about making an external app available on the Google Play store, which everyone could easily update from. True for most, but not everyone. Would there be an equivalent copy available on common app stores in China or on other Android devices where Google Play isn't available, (at least not by default, e.g. Amazon Fire devices)?
• Extension. With either approach mentioned, how would someone add support for a new device, (HMD, controllers, etc.)? Someone in this case might be the device manufacturer or it might be an app developer.
  • Is the service app going to be open source?
  • Would you expect someone to write an OSVR plugin for that HW and submit that to the service app's repo?
• Compatibility across all platforms is probably less important than compatibility across "similar" platforms, i.e. mobile should be the same across Android & iOS, (as much as possible,) but that could acceptably be a bit different from PC or console. From both a dev and player POV.
  • Others may disagree if they are trying to put a small game on all platforms, but we tend to make different, tailored games on mobile vs. PC/console. Not sure if VR will change that.
  • OSVR doesn't currently support iOS though. No plans for that? GoogleVR does, at least for Cardboard.
• What is the story with permissions? What permissions might the services actually need and how would that work? Would the app itself need, for example, the camera permission before the service that required it would work? I couldn't find a definitive answer on that from some quick Android doc reviewing, so perhaps some quick test code would be good.
  • This is probably fine, but good to be clear on what the UX would be.
• In general I think we would prefer to have everything contained within the app and not rely on external dependencies or have the player jump through any hoops, however if we could make the UX clean, consistent and relatively simple then it could be acceptable to require the external app. It is only really a user burden on the first VR app they launch. Having said that, if that becomes necessary even to support something that "should just work", e.g. the most common cases of default Google Cardboard/Daydream setups, then that could be a bad thing. Players should be willing to go through the extra hoops if they wanted to have a more exotic setup, (e.g. adding a leap motion controller to their Android phone.)

I hope that makes some sense. Happy to be challenged on any of that.

Ash