Merge pull request #154 from 781flyingdutchman/parallel

Parallel downloads
781flyingdutchman · Oct 5, 2023 · a258302 · a258302
2 parents 36fdd1d + 794b201
commit a258302
Show file tree

Hide file tree

Showing 38 changed files with 4,732 additions and 2,104 deletions.
diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml
@@ -0,0 +1,3 @@
+# These are supported funding model platforms
+
+github: [781flyingdutchman]
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,9 @@
+## 7.10.0
+
+Add `ParallelDownloadTask`. Some servers may offer an option to download part of the same file from multiple URLs or have multiple parallel downloads of part of a large file using a single URL. This can speed up the download of large files.  To do this, create a `ParallelDownloadTask` instead of a regular `DownloadTask` and specify `chunks` (the number of pieces you want to break the file into, i.e. the number of downloads that will happen in parallel) and `urls` (as a list of URLs, or just one). For example, if you specify 4 chunks and 2 URLs, then the download will be broken into 4 pieces, two each for each URL.
+
+Note that the implementation of this feature creates a regular `DownloadTask` for each chunk, with the group name 'chunk' which is now a reserved group. You will not get updates for this group, but you will get normal updates (status and/or progress) for the `ParallelDownloadTask`.
+
 ## 7.9.4
 
 Enable compile for Web platform (through stubbing - no actual web functionality).

diff --git a/README.md b/README.md
@@ -16,7 +16,7 @@ Monitor progress by passing an `onProgress` listener, and monitor detailed statu
 
 Optionally, keep track of task status and progress in a persistent [database](#using-the-database-to-track-tasks), and show mobile [notifications](#notifications) to keep the user informed and in control when your app is in the background.
 
-To upload a file, create an [UploadTask](https://pub.dev/documentation/background_downloader/latest/background_downloader/UploadTask-class.html) and call `upload`. To make a regular [server request](#server-requests), create a [Request](https://pub.dev/documentation/background_downloader/latest/background_downloader/Request-class.html) and call `request`.
+To upload a file, create an [UploadTask](https://pub.dev/documentation/background_downloader/latest/background_downloader/UploadTask-class.html) and call `upload`. To make a regular [server request](#server-requests), create a [Request](https://pub.dev/documentation/background_downloader/latest/background_downloader/Request-class.html) and call `request`. To download in parallel from multiple servers, create a [ParallelDownloadTask](https://pub.dev/documentation/background_downloader/latest/background_downloader/ParallelDownloadTask-class.html).
 
 The plugin supports [headers](#headers), [retries](#retries), [requiring WiFi](#requiring-wifi) before starting the up/download, user-defined [metadata](#metadata) and GET, [POST](#post-requests) and other http(s) [requests](#http-request-method), and can be [configured](#configuration) by platform. You can [manage  the tasks in the queue](#managing-tasks-and-the-queue) (e.g. cancel, pause and resume), and have different handlers for updates by [group](#grouping-tasks) of tasks. Downloaded files can be moved to [shared storage](#shared-and-scoped-storage) to make them available outside the app.
 
@@ -191,6 +191,7 @@ FileDownloader().configureNotification(
 - [Notifications](#notifications)
 - [Shared and scoped storage](#shared-and-scoped-storage)
 - [Uploads](#uploads)
+- [Parallel downloads](#parallel-downloads)
 - [Managing tasks in the queue](#managing-tasks-and-the-queue)
   - [Canceling, pausing and resuming tasks](#canceling-pausing-and-resuming-tasks)
   - [Grouping tasks](#grouping-tasks)
@@ -570,6 +571,12 @@ Once the `MultiUpoadTask` is created, the fields `fileFields`, `filenames` and `
 
 Use the `MultiTaskUpload` object in the `upload` and `enqueue` methods as you would a regular `UploadTask`.
 
+## Parallel downloads
+
+Some servers may offer an option to download part of the same file from multiple URLs or have multiple parallel downloads of part of a large file using a single URL. This can speed up the download of large files.  To do this, create a `ParallelDownloadTask` instead of a regular `DownloadTask` and specify `chunks` (the number of pieces you want to break the file into, i.e. the number of downloads that will happen in parallel) and `urls` (as a list of URLs, or just one). For example, if you specify 4 chunks and 2 URLs, then the download will be broken into 4 pieces, two each for each URL.
+
+Note that the implementation of this feature creates a regular `DownloadTask` for each chunk, with the group name 'chunk' which is now a reserved group. You will not get updates for this group, but you will get normal updates (status and/or progress) for the `ParallelDownloadTask`.
+
 ## Managing tasks and the queue
 
 ### Canceling, pausing and resuming tasks
@@ -743,7 +750,7 @@ Several aspects of the downloader can be configured on startup:
 * Setting a proxy
 * Localizing the notification button texts on iOS
 * Bypassing TLS Certificate validation (for debug mode only)
-  
+
 Please read the [configuration document](https://github.com/781flyingdutchman/background_downloader/blob/main/CONFIG.md) for details on how to configure.
 
 ## Limitations

diff --git a/android/build.gradle b/android/build.gradle
@@ -2,7 +2,7 @@ group 'com.bbflight.background_downloader'
 version '1.0-SNAPSHOT'
 
 buildscript {
-    ext.kotlin_version = '1.8.0'
+    ext.kotlin_version = '1.8.22'
     repositories {
         google()
         mavenCentral()
@@ -25,7 +25,7 @@ apply plugin: 'com.android.library'
 apply plugin: 'kotlin-android'
 
 android {
-    compileSdkVersion 33
+    compileSdk 33
 
     compileOptions {
         sourceCompatibility JavaVersion.VERSION_1_8
@@ -50,12 +50,11 @@ android {
 dependencies {
     implementation "androidx.work:work-runtime-ktx:2.8.1"
     implementation "androidx.concurrent:concurrent-futures-ktx:1.1.0"
-    implementation "androidx.preference:preference-ktx:1.2.0"
+    implementation "androidx.preference:preference-ktx:1.2.1"
     implementation 'com.google.code.gson:gson:2.8.9'
     implementation "androidx.core:core-ktx:1.10.0"
     implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
 }
 repositories {
     mavenCentral()
 }
-
diff --git a/..._downloader/BackgroundDownloaderPlugin.kt → ...bflight/background_downloader/BDPlugin.kt b/..._downloader/BackgroundDownloaderPlugin.kt → ...bflight/background_downloader/BDPlugin.kt
@@ -12,10 +12,7 @@ import androidx.core.app.ActivityCompat
 import androidx.core.app.NotificationManagerCompat
 import androidx.preference.PreferenceManager
 import androidx.work.*
-import com.bbflight.background_downloader.TaskWorker.Companion.keyEtag
-import com.bbflight.background_downloader.TaskWorker.Companion.keyNotificationConfig
-import com.bbflight.background_downloader.TaskWorker.Companion.keyStartByte
-import com.bbflight.background_downloader.TaskWorker.Companion.keyTempFilename
+import com.bbflight.background_downloader.TaskWorker.Companion.taskToJsonString
 import com.google.gson.Gson
 import com.google.gson.reflect.TypeToken
 import io.flutter.embedding.engine.plugins.FlutterPlugin
@@ -51,7 +48,7 @@ import kotlin.concurrent.write
  * Manages the WorkManager task queue and the interface to Dart. Actual work is done in
  * [TaskWorker]
  */
-class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAware,
+class BDPlugin : FlutterPlugin, MethodCallHandler, ActivityAware,
     PluginRegistry.RequestPermissionsResultListener {
     companion object {
         const val TAG = "BackgroundDownloader"
@@ -76,6 +73,7 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
         var activity: Activity? = null
         var canceledTaskIds = HashMap<String, Long>() // <taskId, timeMillis>
         var pausedTaskIds = HashSet<String>() // <taskId>
+        var parallelDownloadTaskWorkers = HashMap<String, ParallelDownloadTaskWorker>()
         var backgroundChannel: MethodChannel? = null
         var backgroundChannelCounter = 0  // reference counter
         var forceFailPostOnBackgroundChannel = false
@@ -93,14 +91,11 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
          */
         suspend fun doEnqueue(
             context: Context,
-            taskJsonMapString: String,
+            task: Task,
             notificationConfigJsonString: String?,
-            tempFilePath: String?,
-            startByte: Long?,
-            eTag: String?,
+            resumeData: ResumeData?,
             initialDelayMillis: Long = 0
         ): Boolean {
-            val task = Task(gson.fromJson(taskJsonMapString, jsonMapType))
             Log.i(TAG, "Enqueuing task with id ${task.taskId}")
             // validate the task.url
             try {
@@ -116,20 +111,25 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
                 return false
             }
             canceledTaskIds.remove(task.taskId)
-            val dataBuilder = Data.Builder().putString(TaskWorker.keyTask, taskJsonMapString)
+            val dataBuilder = Data.Builder().putString(TaskWorker.keyTask, taskToJsonString(task))
             if (notificationConfigJsonString != null) {
-                dataBuilder.putString(keyNotificationConfig, notificationConfigJsonString)
+                dataBuilder.putString(TaskWorker.keyNotificationConfig, notificationConfigJsonString)
             }
-            if (tempFilePath != null && startByte != null) {
-                dataBuilder.putString(keyTempFilename, tempFilePath)
-                    .putLong(keyStartByte, startByte)
-                    .putString(keyEtag, eTag)
+            if (resumeData != null) {
+                dataBuilder.putString(TaskWorker.keyResumeDataData, resumeData.data)
+                    .putLong(TaskWorker.keyStartByte, resumeData.requiredStartByte)
+                    .putString(TaskWorker.keyETag, resumeData.eTag)
             }
             val data = dataBuilder.build()
             val constraints = Constraints.Builder().setRequiredNetworkType(
                 if (task.requiresWiFi) NetworkType.UNMETERED else NetworkType.CONNECTED
             ).build()
-            val requestBuilder = OneTimeWorkRequestBuilder<TaskWorker>().setInputData(data)
+            val requestBuilder =
+                if (task.isParallelDownloadTask())
+                    OneTimeWorkRequestBuilder<ParallelDownloadTaskWorker>()
+                else if (task.isDownloadTask()) OneTimeWorkRequestBuilder<DownloadTaskWorker>()
+                else OneTimeWorkRequestBuilder<UploadTaskWorker>()
+            requestBuilder.setInputData(data)
                 .setConstraints(constraints).addTag(TAG).addTag("taskId=${task.taskId}")
                 .addTag("group=${task.group}")
             if (initialDelayMillis != 0L) {
@@ -179,6 +179,8 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
         suspend fun cancelActiveTaskWithId(
             context: Context, taskId: String, workManager: WorkManager
         ): Boolean {
+            // cancel chunk tasks if this is a ParallelDownloadTask
+            parallelDownloadTaskWorkers[taskId]?.cancelAllChunkTasks()
             val workInfos = withContext(Dispatchers.IO) {
                 workManager.getWorkInfosByTag("taskId=$taskId").get()
             }
@@ -189,7 +191,6 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
             for (workInfo in workInfos) {
                 if (workInfo.state != WorkInfo.State.SUCCEEDED) {
                     // send cancellation update for tasks that have not yet succeeded
-                    Log.d(TAG, "Canceling active task and sending status update")
                     prefsLock.write {
                         val prefs = PreferenceManager.getDefaultSharedPreferences(context)
                         val tasksMap = getTaskMap(prefs)
@@ -200,7 +201,7 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
                             )
                             TaskWorker.processStatusUpdate(task, TaskStatus.canceled, prefs)
                         } else {
-                            Log.d(TAG, "Could not find taskId $taskId to cancel")
+                            Log.d(TAG, "Could not find task with taskId $taskId to cancel")
                         }
                     }
                 }
@@ -317,21 +318,25 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
                 "killTaskWithId" -> methodKillTaskWithId(call, result)
                 "taskForId" -> methodTaskForId(call, result)
                 "pause" -> methodPause(call, result)
+                "moveToSharedStorage" -> methodMoveToSharedStorage(call, result)
+                "pathInSharedStorage" -> methodPathInSharedStorage(call, result)
+                "openFile" -> methodOpenFile(call, result)
+                // ParallelDownloadTask child updates
+                "chunkStatusUpdate" -> methodUpdateChunkStatus(call, result)
+                "chunkProgressUpdate" -> methodUpdateChunkProgress(call, result)
+                // internal use
                 "popResumeData" -> methodPopResumeData(result)
                 "popStatusUpdates" -> methodPopStatusUpdates(result)
                 "popProgressUpdates" -> methodPopProgressUpdates(result)
                 "getTaskTimeout" -> methodGetTaskTimeout(result)
-                "moveToSharedStorage" -> methodMoveToSharedStorage(call, result)
-                "pathInSharedStorage" -> methodPathInSharedStorage(call, result)
-                "openFile" -> methodOpenFile(call, result)
+                // configuration
                 "configForegroundFileSize" -> methodConfigForegroundFileSize(call, result)
                 "configProxyAddress" -> methodConfigProxyAddress(call, result)
                 "configProxyPort" -> methodConfigProxyPort(call, result)
                 "configRequestTimeout" -> methodConfigRequestTimeout(call, result)
                 "configBypassTLSCertificateValidation" -> methodConfigBypassTLSCertificateValidation(
                     result
                 )
-
                 "configCheckAvailableSpace" -> methodConfigCheckAvailableSpace(call, result)
                 "configUseCacheDir" -> methodConfigUseCacheDir(call, result)
                 "forceFailPostOnBackgroundChannel" -> methodForceFailPostOnBackgroundChannel(
@@ -354,28 +359,23 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
         // by tempFilePath, startByte and eTag if this enqueue is a resume from pause
         val args = call.arguments as List<*>
         val taskJsonMapString = args[0] as String
+        val task = Task(gson.fromJson(taskJsonMapString, jsonMapType))
         val notificationConfigJsonString = args[1] as String?
         val isResume = args.size == 5
-        val startByte: Long?
-        val tempFilePath: String?
-        val eTag: String?
-        if (isResume) {
-            tempFilePath = args[2] as String
-            startByte = if (args[3] is Long) args[3] as Long else (args[3] as Int).toLong()
-            eTag = args[4] as String?
+        val resumeData: ResumeData?
+        resumeData = if (isResume) {
+            val startByte = if (args[3] is Long) args[3] as Long else (args[3] as Int).toLong()
+            val eTag = args[4] as String?
+            ResumeData(task, args[2] as String, startByte, eTag)
         } else {
-            tempFilePath = null
-            startByte = null
-            eTag = null
+            null
         }
         result.success(
             doEnqueue(
                 applicationContext,
-                taskJsonMapString,
+                task,
                 notificationConfigJsonString,
-                tempFilePath,
-                startByte,
-                eTag
+                resumeData,
             )
         )
     }
@@ -615,6 +615,50 @@ class BackgroundDownloaderPlugin : FlutterPlugin, MethodCallHandler, ActivityAwa
         result.success(if (activity != null) doOpenFile(activity!!, filePath, mimeType) else false)
     }
 
+    /**
+     * Update the status of one chunk (part of a ParallelDownloadTask), and returns
+     * the status of the parent task based on the 'sum' of its children, or null
+     * if unchanged
+     *
+     * Arguments are the parent TaskId, chunk taskId, taskStatusOrdinal
+     */
+    private suspend fun methodUpdateChunkStatus(call: MethodCall, result: Result) {
+        val args = call.arguments as List<*>
+        val taskId = args[0] as String
+        val chunkTaskId = args[1] as String
+        val statusOrdinal = args[2] as Int
+        val exceptionJson = args[3] as String?
+        val exception = if (exceptionJson != null) {
+            TaskException(gson.fromJson(exceptionJson, jsonMapType) as Map<String, Any?>)
+        } else null
+        val responseBody = args[4] as String?
+        parallelDownloadTaskWorkers[taskId]?.chunkStatusUpdate(
+            chunkTaskId,
+            TaskStatus.values()[statusOrdinal],
+            exception,
+            responseBody
+        )
+        result.success(null)
+    }
+
+    /**
+     * Update the progress of one chunk (part of a ParallelDownloadTask), and returns
+     * the progress of the parent task based on the average of its children
+     *
+     * Arguments are the parent [TaskId, chunk taskId, progress]
+     */
+    private suspend fun methodUpdateChunkProgress(call: MethodCall, result: Result) {
+        val args = call.arguments as List<*>
+        val taskId = args[0] as String
+        val chunkTaskId = args[1] as String
+        val progress = args[2] as Double
+        parallelDownloadTaskWorkers[taskId]?.chunkProgressUpdate(
+            chunkTaskId,
+            progress
+        )
+        result.success(null)
+    }
+
     /**
      * Returns the [TaskWorker] timeout value in milliseconds
      *
@@ -867,12 +911,12 @@ class ResultHandler(private val completer: CompletableFuture<Boolean>) : Result
     }
 
     override fun error(errorCode: String, errorMessage: String?, errorDetails: Any?) {
-        Log.i(BackgroundDownloaderPlugin.TAG, "Flutter result error $errorCode: $errorMessage")
+        Log.i(BDPlugin.TAG, "Flutter result error $errorCode: $errorMessage")
         completer.complete(false)
     }
 
     override fun notImplemented() {
-        Log.i(BackgroundDownloaderPlugin.TAG, "Flutter method not implemented")
+        Log.i(BDPlugin.TAG, "Flutter method not implemented")
         completer.complete(false)
     }
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		# These are supported funding model platforms

		github: [781flyingdutchman]