Speedup on Android

[eyalhoc]

I'm running the app on Samsung S8.
I get about 0.5 fps for the highest quality and around 2 fps for the lowest.
I'm trying to speed it up but have little experience with this.
From what I found in the documentation GPU will give about 2x but will only work with tflite.
Does anyone intend to contribute in this direction or can help with some guidance?

[FilippoAleotti]

Hi
I already did in the past a similar application using tflite. Unfortunately, the app was supposed to be “shot and elaborate”, so not in streaming. However, I’ve already planned to add a tflite-version as soon as I have some spare time. The main difference is about the tf engine invocation, since the api is changed.

[eyalhoc]

Hi Filippo, Did you publish the other app, can I see it? Thanks, Eyal

…

On Mon, Jul 8, 2019, 20:10 Filippo Aleotti ***@***.***> wrote: Hi I already did in the past a similar application using tflite. Unfortunately, the app was supposed to be “shot and elaborate”, so not in streaming. However, I’ve already planned to add a tflite-version as soon as I have some spare time. The main difference is about the tf engine invocarion, since the api is changed. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFH4BLJVB5UJJZ7LNUIKG3DP6NYIPA5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNX2EI#issuecomment-509312273>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH4BLJDRCU7JVD2R24MASDP6NYIPANCNFSM4H65OEFA> .

[FilippoAleotti]

Hi Eyal
No, I didn’t published that app. In the next days I’ll try to upload a tflite version of this app
Best regards

[eyalhoc]

That's great thanks

…

On Mon, Jul 8, 2019, 22:57 Filippo Aleotti ***@***.***> wrote: Hi Eyal No, I didn’t published that app. In the next days I’ll try to upload a tflite version of this app Best regards — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFH4BLITFEOA5IRH66HEGTDP6OL3LA5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOFWLA#issuecomment-509369132>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH4BLLBBJFZ5AU2C3RWH73P6OL3LANCNFSM4H65OEFA> .

[eyalhoc]

Hi Filippo, Any news? Is there something I can do to help?

…

On Mon, Jul 8, 2019, 23:01 Eyal Hochberg ***@***.***> wrote: That's great thanks On Mon, Jul 8, 2019, 22:57 Filippo Aleotti ***@***.***> wrote: > Hi Eyal > No, I didn’t published that app. In the next days I’ll try to upload a > tflite version of this app > Best regards > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#1?email_source=notifications&email_token=AFH4BLITFEOA5IRH66HEGTDP6OL3LA5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOFWLA#issuecomment-509369132>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AFH4BLLBBJFZ5AU2C3RWH73P6OL3LANCNFSM4H65OEFA> > . >

[FilippoAleotti]

Hi Eyal
I tried two days ago but right now the app crashes (probably due to a bug in the code I added). Unfortunately, I’m quite busy now and I have really few time to spend on debugging. If you have time to take a look at it, I can push the bugged tflite version in a separate folder/branch

[eyalhoc]

I'm not very strong in java but I have time and will be happy to try and fix it. Please send me a link

…

On Fri, Jul 12, 2019, 20:36 Filippo Aleotti ***@***.***> wrote: Hi Eyalhoc I tried two days ago but right now the app crashes (probably due to a bug in the code I added). Unfortunately, I’m quite busy now and I have really few time to spend on debugging. If you have time to take a look at it, I can push the bugged tflite version in a separate folder/branch — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFH4BLNQPWMLGZU5EEAHLDDP7C6JDA5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ2MWXI#issuecomment-510970717>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH4BLLWKPMEEMDWLGBBKI3P7C6JDANCNFSM4H65OEFA> .

[FilippoAleotti]

Just created a tflite branch. I uploaded also the script I used for exporting the model

[eyalhoc]

I checked the tflite model it works for me. On my Galaxy S8 inference is around 2-2.5 seconds per frame.
I tried to use GPU and I get this error:

java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: WARNING: op code #55 cannot be handled by this delegate.  Only the first 2 ops will run on the GPU, and the remaining 134 on the CPU.GpuDelegate Prepare: ReadValue: value is a constant tensor: 193Node number 136 (GpuDelegate) failed to prepare.

[eyalhoc]

I updated to:
implementation 'org.tensorflow:tensorflow-lite:0.0.1-gpu-experimental'

Got this error:
java.lang.IllegalArgumentException: Internal error: Failed to apply delegate: Next operations are not supported by GPU delegate:
MAXIMUM: Operation is not supported.
First 2 operations will run on the GPU, and the remaining 134 on the CPU.TfLiteGlDelegate Prepare: ReadValue: value is a constant tensor: 193Node number 136 (TfLiteGlDelegate) failed to prepare.

[FilippoAleotti]

Thank you for your support. So, the problem is the maximum operation used in the leaky_relu. A possible solution could be replacing the leaky_relu (defined in script/layers.py) with the native tensorflow op tf.nn.leaky_relu, which should be supported by tflite https://www.tensorflow.org/lite/guide/ops_compatibility.

If you want to try again, I've already updated the frozen tflite-model in the asset folder using the native operation

[FilippoAleotti]

Hi Eyal, have you got some news? I’ve spent some time on debugging but I can’t figure out why the app crashes. However, with the native tf.nn.leaky_relu the maximum operation problem is gone

[eyalhoc]

Ho Filippo, I found multiple issues. First the tflite network works well. The problem was with gpu delegation. Beyond the non supported layer it gave a warning that almost all layers will work on cpu. So, I'm not sure it will boost performance. Second, I looked at the quality of the depth detection and it looks very different than the pb model. Different as in much lower quality. Don't know why this happens since you didn't quantize the weights.

…

On Tue, Jul 23, 2019, 09:04 Filippo Aleotti ***@***.***> wrote: Hi Eyal, have you got some news? I’ve spent some time on debugging but I can’t figure out why the app crashes. However, with the native tf.nn.leaky_relu the maximum operation problem is gone — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFH4BLLJPCLC75QDMUA4O23QA2NP5A5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2SAT3Q#issuecomment-514066926>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH4BLJCLC5IRE3VM2LPZI3QA2NP5ANCNFSM4H65OEFA> .

[FilippoAleotti]

Thank you for your reply

About the layers on gpu, I notice it but it sounds strange since we do not use any particular operation. For instance, the mlcore model for iOS is generated and runs on gpu without problems.

About the quality, are you sure you are not using a lower resolution map (for instance, the quartese resolution one)?

However, if I can help you in some way just ask

[eyalhoc]

I tried your tflite freeze, I'm quite sure you used the higher quality. I'll try to review it again. Have you updated the tflite file in GitHub?

…

On Tue, Jul 23, 2019, 22:28 Filippo Aleotti ***@***.***> wrote: Thank you for your reply About the layers on gpu, I notice it but it sounds strange since we do not use any particular operation. For instance, the mlcore model for iOS is generated and runs on gpu without problems. About the quality, are you sure you are not using a lower resolution map (for instance, the quartese resolution one)? However, if I can help you in some way just ask — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AFH4BLJR4DY2ZNKPTQJXVCDQA5LT7A5CNFSM4H65OEFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2UFWKI#issuecomment-514349865>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH4BLIYT42EIKL6EYEIIZDQA5LT7ANCNFSM4H65OEFA> .

[rdcdt1]

hi
it's a beautiful project and it's sad that it's run so slow on android
i look and the app only use 1 thread maybe with mutlithreading it will be faster.
i find a project that use multithreading or the gpu (https://github.com/danielecolautti/PyDNetMobileMultithreading), it's improve a lot the speed (10 fps) but the depth map is wrong.