[iOS] Set content length if not provided by server, and using "?" to …

…get suggested filename during download

Use Ranges header or Known-Content-Length special header in the Task to be used if the server does not provide Content-Length header. This enables progress updates for downloads of known size, even if the server does not provide content length.

"?" as filename will trigger parsing of response header for Content-Disposition, or last path segment of URL

CHANGELOG.md

@@ -1,3 +1,34 @@
+## 7.11.0
+
+### Android external storage
+Add configuration for Android to use external storage instead of internal storage. Either your app runs in default (internal storage) mode, or in external storage. You cannot switch between internal and external, as the directory structure that - for example - `BaseDirectory.applicationDocuments` refers to is different in each mode. See the [configuration document](https://github.com/781flyingdutchman/background_downloader/blob/main/CONFIG.md) for important details and limitations
+
+Use `(Config.useExternalStorage, String whenToUse)` with values 'never' or 'always'. Default is `Config.never`.
+
+### Server suggested filename
+If you want the filename to be provided by the server (instead of assigning a value to `filename` yourself), you now have two options. The first is to create a `DownloadTask` that pings the server to determine the suggested filename:
+```dart
+final task = await DownloadTask(url: 'https://google.com')
+        .withSuggestedFilename(unique: true);
+```
+The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible (e.g. due to a lack of connection). If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt". You can now also supply a `taskWithFilenameBuilder` to suggest the filename yourself, based on response headers.
+
+The second approach is to set the `filename` field of the `DownloadTask` to `DownloadTask.suggestedFilename`, to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be `DownloadTask.suggestedFilename`. For example:
+```dart
+final task = await DownloadTask(url: 'https://google.com', filename: DownloadTask.suggestedFilename);
+final result = await FileDownloader().download(task);
+print('Suggested filename=${result.task.filename}'); // note we don't use 'task', but 'result.task'
+print('Wrong use filename=${task.filename}'); // this will print '?' as 'task' hasn't changed
+```
+
+### Set content length if not provided by server
+
+To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the download. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes.  The downloader will assume this content length when calculating progress.
+
+### Bug fix
+
+Partial Downloads, using a Range header, can now be properly paused on all platforms.
+
 ## 7.10.1
 
 Add `displayName` field to `Task` that can be used to store and dsipay a 'human readable' description of the task. It can be displayed in a notification using {displayName}.

README.md

@@ -234,7 +234,7 @@ A [DownloadProgressIndicator](https://pub.dev/documentation/background_downloade
 The widget can be configured to include pause and resume buttons, and to expand to show multiple
 simultaneous downloads, or to collapse and show a file download counter.
 
-To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the donwload. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes.  The downloader will assume this content length when calculating progress.  
+To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the download. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes.  The downloader will assume this content length when calculating progress.  
 
 #### Status
 
@@ -299,11 +299,11 @@ If you want the filename to be provided by the server (instead of assigning a va
 final task = await DownloadTask(url: 'https://google.com')
         .withSuggestedFilename(unique: true);
 ```
-The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible. If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt".
+The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible (e.g. due to a lack of connection). If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt". You can also supply a `taskWithFilenameBuilder` to suggest the filename yourself, based on response headers.
 
-The second approach is to set the `filename` field of the `DownloadTask` to '?', to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be '?'. For example:
+The second approach is to set the `filename` field of the `DownloadTask` to `DownloadTask.suggestedFilename`, to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be `DownloadTask.suggestedFilename`. For example:
 ```dart
-final task = await DownloadTask(url: 'https://google.com', filename: '?');
+final task = await DownloadTask(url: 'https://google.com', filename: DownloadTask.suggestedFilename);
 final result = await FileDownloader().download(task);
 print('Suggested filename=${result.task.filename}'); // note we don't use 'task', but 'result.task'
 print('Wrong use filename=${task.filename}'); // this will print '?' as 'task' hasn't changed

example/integration_test/downloader_integration_test.dart

@@ -2112,7 +2112,7 @@ void main() {
           taskProgressCallback: progressCallback);
       task = DownloadTask(
           url: urlWithContentLength,
-          filename: '?',
+          filename: DownloadTask.suggestedFilename,
           updates: Updates.statusAndProgress,
           allowPause: true);
       expect(await FileDownloader().enqueue(task), equals(true));
@@ -2698,7 +2698,8 @@ void main() {
       try {
         File(await task.filePath()).deleteSync();
       } catch (e) {}
-      task = DownloadTask(url: urlWithContentLength, filename: '?');
+      task = DownloadTask(
+          url: urlWithContentLength, filename: DownloadTask.suggestedFilename);
       final result = await FileDownloader().download(task);
       expect(result.task.filename, equals('5MB-test.ZIP'));
       final file = File(await result.task.filePath());
@@ -2793,7 +2794,7 @@ void main() {
     testWidgets('[*] Range or Known-Content-Length in task header',
         (widgetTester) async {
       // Haven't found a url that does not provide content-length, so
-      // can oly be tested by modifying the source code to ignore the
+      // can only be tested by modifying the source code to ignore the
       // Content-Length response header and use this one instead
       FileDownloader().registerCallbacks(
           taskStatusCallback: statusCallback,

example/integration_test/parallel_download_test.dart

@@ -158,23 +158,6 @@ void main() {
       expect(await fileEqualsTestFile(file), isTrue);
     });
 
-    test('[*] override content length', () async {
-      // Haven't found a url that does not provide content-length, so
-      // can oly be tested by modifying the source code to ignore the
-      // Content-Length response header and use this one instead
-      FileDownloader().registerCallbacks(taskStatusCallback: statusCallback);
-      task = task.copyWith(
-          url: urlWithContentLength,
-          headers: {'Known-Content-Length': '$urlWithContentLengthFileSize'});
-      expect(await FileDownloader().enqueue(task), isTrue);
-      await statusCallbackCompleter.future;
-      expect(lastStatus, equals(TaskStatus.complete));
-      expect(statusCallbackCounter, equals(3));
-      final file = File(await task.filePath());
-      expect(file.existsSync(), isTrue);
-      expect(await fileEqualsTestFile(file), isTrue);
-    });
-
     test('simple enqueue, 2 chunks, 2 url', () async {
       task = ParallelDownloadTask(
           url: [urlWithContentLength, urlWithContentLength],
@@ -359,9 +342,31 @@ void main() {
       statusCallbackCompleter = Completer();
       lastValidExpectedFileSize = -1;
       task = task.copyWith(taskId: "3", headers: {});
+      if (!Platform.isIOS) {
+        expect(await FileDownloader().enqueue(task), isTrue);
+        await statusCallbackCompleter.future;
+        expect(lastStatus, equals(TaskStatus.failed));
+      } else {
+        // on iOS the task fails at enqueue
+        expect(await FileDownloader().enqueue(task), isFalse);
+      }
+    });
+
+    test('[*] override content length', () async {
+      // Haven't found a url that does not provide content-length, so
+      // can only be tested by modifying the source code to ignore the
+      // Content-Length response header and use this one instead
+      FileDownloader().registerCallbacks(taskStatusCallback: statusCallback);
+      task = task.copyWith(
+          url: urlWithContentLength,
+          headers: {'Known-Content-Length': '$urlWithContentLengthFileSize'});
       expect(await FileDownloader().enqueue(task), isTrue);
       await statusCallbackCompleter.future;
-      expect(lastStatus, equals(TaskStatus.failed));
+      expect(lastStatus, equals(TaskStatus.complete));
+      expect(statusCallbackCounter, equals(3));
+      final file = File(await task.filePath());
+      expect(file.existsSync(), isTrue);
+      expect(await fileEqualsTestFile(file), isTrue);
     });
   });
 }

ios/Classes/Downloader.swift

@@ -40,6 +40,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
     static var localResumeData = [String : String]() // locally stored to enable notification resume
     static var remainingBytesToDownload = [String : Int64]()  // keyed by taskId
     static var responseBodyData = [String: [Data]]() // list of Data objects received for this UploadTask id
+    static var tasksWithSuggestedFilename = [String : Task]() // [taskId : Task with suggested filename]
+    static var tasksWithContentLengthOverride = [String : Int64]() // [taskId : Content length]
 
     public static func register(with registrar: FlutterPluginRegistrar) {
         let channel = FlutterMethodChannel(name: "com.bbflight.background_downloader", binaryMessenger: registrar.messenger())
@@ -111,6 +113,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
                 methodStoreConfig(key: Downloader.keyConfigCheckAvailableSpace, value: call.arguments, result: result)
             case "forceFailPostOnBackgroundChannel":
                 methodForceFailPostOnBackgroundChannel(call: call, result: result)
+            case "testSuggestedFilename":
+                methodTestSuggestedFilename(call: call, result: result)
             default:
                 os_log("Invalid method: %@", log: log, type: .error, call.method)
                 result(FlutterMethodNotImplemented)
@@ -484,6 +488,22 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
         result(nil)
     }
 
+    /// Tests the content-disposition and url translation
+    ///
+    /// For testing only
+    private func methodTestSuggestedFilename(call: FlutterMethodCall, result: @escaping FlutterResult) {
+        let args = call.arguments as! [Any]
+        guard let taskJsonString = args[0] as? String,
+              let contentDisposition = args[1] as? String,
+              let task = taskFrom(jsonString: taskJsonString) else {
+            result("")
+            return
+        }
+        let resultTask = suggestedFilenameFromResponseHeaders(task: task, responseHeaders: ["Content-Disposition" : contentDisposition], unique: true)
+        result(resultTask.filename)
+    }
+
+
 
     //MARK: Helpers for Task and urlSessionTask
 
@@ -550,6 +570,9 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
             os_log("Could not find task related to urlSessionTask %d", log: log, type: .error, task.taskIdentifier)
             return
         }
+        // clear storage related to this task
+        Downloader.tasksWithSuggestedFilename.removeValue(forKey: task.taskId)
+        Downloader.tasksWithContentLengthOverride.removeValue(forKey: task.taskId)
         let responseBody = getResponseBody(taskId: task.taskId)
         Downloader.responseBodyData.removeValue(forKey: task.taskId)
         Downloader.taskIdsThatCanResume.remove(task.taskId)
@@ -618,9 +641,28 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
     /// If the task requires progress updates, provide these at a reasonable interval
     /// If this is the first update for this file, also emit a 'running' status update
     public func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) {
-        guard let task = getTaskFrom(urlSessionTask: downloadTask) else { return }
+        // task is var because the filename can be changed on the first 'didWriteData' call
+        guard var task = getTaskFrom(urlSessionTask: downloadTask) else { return }
         if Downloader.progressInfo[task.taskId] == nil {
             // first 'didWriteData' call
+            let response = downloadTask.response as! HTTPURLResponse
+            // get suggested filename if needed
+            if task.filename == "?" {
+                let newTask = suggestedFilenameFromResponseHeaders(task: task, responseHeaders: response.allHeaderFields)
+                os_log("Suggested task filename for taskId %@ is %@", log: log, type: .info, newTask.taskId, newTask.filename)
+                if newTask.filename != task.filename {
+                    // store for future replacement, and replace now
+                    Downloader.tasksWithSuggestedFilename[newTask.taskId] = newTask
+                    task = newTask
+                }
+            }
+            // obtain content length override, if needed and available
+            if totalBytesExpectedToWrite == -1 {
+                let contentLength = getContentLength(responseHeaders: response.allHeaderFields, task: task)
+                if contentLength != -1 {
+                    Downloader.tasksWithContentLengthOverride[task.taskId] = contentLength
+                }
+            }
             // Check if there is enough space
             if insufficientSpace(contentLength: totalBytesExpectedToWrite) {
                 if !Downloader.taskIdsProgrammaticallyCancelled.contains(task.taskId) {
@@ -649,8 +691,9 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
                 updateNotification(task: task, notificationType: .running, notificationConfig: notificationConfig)
             }
         }
-        Downloader.remainingBytesToDownload[task.taskId] = totalBytesExpectedToWrite - totalBytesWritten
-        updateProgress(task: task, totalBytesExpected: totalBytesExpectedToWrite, totalBytesDone: totalBytesWritten)
+        let contentLength = totalBytesExpectedToWrite != -1 ? totalBytesExpectedToWrite : Downloader.tasksWithContentLengthOverride[task.taskId] ?? -1
+        Downloader.remainingBytesToDownload[task.taskId] = contentLength - totalBytesWritten
+        updateProgress(task: task, totalBytesExpected: contentLength, totalBytesDone: totalBytesWritten)
     }
 
     /// Process taskdelegate progress update for upload task
@@ -686,6 +729,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession
         else {
             os_log("Could not find task associated urlSessionTask %d, or did not get HttpResponse", log: log,  type: .info, downloadTask.taskIdentifier)
             return}
+        Downloader.tasksWithSuggestedFilename.removeValue(forKey: task.taskId)
+        Downloader.tasksWithContentLengthOverride.removeValue(forKey: task.taskId)
         let notificationConfig = getNotificationConfigFrom(urlSessionTask: downloadTask)
         if response.statusCode == 404 {
             let responseBody = readFile(url: location)

ios/Classes/ParallelDownloader.swift

@@ -21,9 +21,9 @@ func scheduleParallelDownload(task: Task, taskDescription: String, baseRequest:
                     os_log("URL not found for taskId %@", log: log, type: .info, task.taskId)
                     postResult(result: result, value: false)
                 }
-                else if !parallelDownload.start(contentLength: Int64(httpResponse.value(forHTTPHeaderField: "Content-Length") ?? "0") ?? 0) {
-                    os_log("Cannot chunk or enqueue download", log: log, type: .info)
-                    postResult(result: result, value: false)
+                else if !parallelDownload.start(contentLengthFromHeader: Int64(httpResponse.value(forHTTPHeaderField: "Content-Length") ?? "-1") ?? -1, responseHeaders: httpResponse.allHeaderFields ) {
+                        os_log("Cannot chunk or enqueue download", log: log, type: .info)
+                        postResult(result: result, value: false)
                 } else {
                     processStatusUpdate(task: task, status: TaskStatus.enqueued)
                     postResult(result: result, value: true)
@@ -85,10 +85,12 @@ public class ParallelDownloader: NSObject {
     /// the
     ///
     /// Returns false if start was unsuccessful
-    public func start(contentLength: Int64) -> Bool {
-        parallelDownloadContentLength = contentLength
+    public func start(contentLengthFromHeader: Int64, responseHeaders: [AnyHashable: Any]) -> Bool {
+        parallelDownloadContentLength = contentLengthFromHeader > 0 ?
+            contentLengthFromHeader :
+            getContentLength(responseHeaders: responseHeaders, task: self.parentTask)
         ParallelDownloader.downloads[parentTask.taskId] = self
-        chunks = createChunks(task: parentTask, contentLength: contentLength)
+        chunks = createChunks(task: parentTask, contentLength: parallelDownloadContentLength)
         let success = !chunks.isEmpty && enqueueChunkTasks()
         if !success {
             ParallelDownloader.downloads.removeValue(forKey: parentTask.taskId)