Download resources (models, datasets) on-demand

[anhappdev]

We now have a lot of large files. The app should download the files only when it needs them and not at app launch.

• Download all resources on-demand (feat: download all resources (models, dataset) on-demand #931)
• Download resources for each benchmark separately
• Support download in background
• Update URLs to use Cloudflare instead of GitHub (feat: use Cloudflare instead of GitHub for hosting models and datasets #935)

[freedomtan]

Does moving large files to some S3 buckets help? It depends on the cloud provider we use (note that S3 is not Amazon).

Maybe it's the limitation on client side (limitation of Android Apps).

@nathanw-mlc where is the right place to place big models (github, S3 buckets, or others).

[nathanw-mlc]

Files served to the app should be in S3, especially since our Cloudflare R2 buckets use Cloudflare's CDN to distribute data to users all around the world at high speeds.

However, I also agree with Anh that the app should not download everything the moment you launch it. That is really strange behavior for an Android app. The first thing you expect to see when launching a new app is not the app downloading a whole bunch of data without even asking you first. What if you first launch the app while not connected to Wi-Fi and the app chews through your mobile data plan? Data should be downloaded on an as-needed-basis, and the app should also notify users that proceeding will require downloading X amount of data.

[anhappdev]

@nathanw-mlc Can you please create a public bucket for hosting the models and datasets, maybe named app-resources?

[nathanw-mlc]

@anhappdev I've created a public bucket named mlcommons-mobile-mlperf-app-resources. Your current keys should have Write access. The files will be publicly downloadable at resources.mlperf-mobile.mlcommons.org/<file-path-inside-bucket>. Let me know if you need anything else.

[anhappdev]

@anhappdev I've created a public bucket named mlcommons-mobile-mlperf-app-resources. Your current keys should have Write access. The files will be publicly downloadable at resources.mlperf-mobile.mlcommons.org/<file-path-inside-bucket>. Let me know if you need anything else.

I uploaded a test file and tried with the URL:
https://resources.mlperf-mobile.mlcommons.org/test/pic1.png
but it did not work.

I didn't know you need to create a new domain for the bucket. I think we can also just create a new directory named app-resources in the same level with the app-models in the mlcommons-mobile-wg-public bucket. So we can have the URL:
https://mobile.mlcommons-storage.org/app-resources/test/pic1.png
What do you think?

[nathanw-mlc]

@anhappdev I've created a public bucket named mlcommons-mobile-mlperf-app-resources. Your current keys should have Write access. The files will be publicly downloadable at resources.mlperf-mobile.mlcommons.org/<file-path-inside-bucket>. Let me know if you need anything else.

I uploaded a test file and tried with the URL: https://resources.mlperf-mobile.mlcommons.org/test/pic1.png but it did not work.

My bad, I forgot to add -storage at the end of the URL. It would be https://resources.mlperf-mobile.mlcommons-storage.org/test/pic1.png.

I didn't know you need to create a new domain for the bucket. I think we can also just create a new directory named app-resources in the same level with the app-models in the mlcommons-mobile-wg-public bucket. So we can have the URL: https://mobile.mlcommons-storage.org/app-resources/test/pic1.png What do you think?

Yes, each subdomain corresponds to a single bucket. A new directory in the mlcommons-mobile-wg-public bucket would be great.

[anhappdev]

Yes, each subdomain corresponds to a single bucket. A new directory in the mlcommons-mobile-wg-public bucket would be great.

@nathanw-mlc Ok. I choose to use the mlcommons-mobile-wg-public bucket so we have all public files for the mobile app in the same place. You can delete the mlcommons-mobile-mlperf-app-resources bucket. Sorry for your effort to create it.