[Android] Set content length if not provided by server.

Use Ranges header or Known-Content-Length special header in the Task to be used if the server does not provide Content-Length header. This enables progress updates for downloads of known size, even if the server does not provide content length.

README.md

@@ -234,6 +234,8 @@ A [DownloadProgressIndicator](https://pub.dev/documentation/background_downloade
 The widget can be configured to include pause and resume buttons, and to expand to show multiple
 simultaneous downloads, or to collapse and show a file download counter.
 
+To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the donwload. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes.  The downloader will assume this content length when calculating progress.  
+
 #### Status
 
 If you want to monitor status changes while the download is underway (i.e. not only the final state, which you will receive as the result of the `download` call) you can add a status change callback that takes the status as an argument:
@@ -286,18 +288,26 @@ final task = DownloadTask(
 
 The downloader will only store the file upon success (so there will be no partial files saved), and if so, the destination is overwritten if it already exists, and all intermediate directories will be created if needed.
 
-Android has two storage modes: internal and external storage. Read the [configuration document](https://github.com/781flyingdutchman/background_downloader/blob/main/CONFIG.md) for details on how to configure your app to use external storage.
-
 Note: the reason you cannot simply pass a full absolute directory path to the downloader is that the location of the app's documents directory may change between application starts (on iOS), and may therefore fail for downloads that complete while the app is suspended.  You should therefore never store permanently, or hard-code, an absolute path.
 
-If you want the filename to be provided by the server (instead of assigning a value to `filename` yourself), use the following:
+Android has two storage modes: internal (default) and external storage. Read the [configuration document](https://github.com/781flyingdutchman/background_downloader/blob/main/CONFIG.md) for details on how to configure your app to use external storage instead of the default.
+
+#### Server-suggested filename
+
+If you want the filename to be provided by the server (instead of assigning a value to `filename` yourself), you have two options. The first is to create a `DownloadTask` that pings the server to determine the suggested filename:
 ```dart
 final task = await DownloadTask(url: 'https://google.com')
         .withSuggestedFilename(unique: true);
 ```
-
 The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible. If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt".
 
+The second approach is to set the `filename` field of the `DownloadTask` to '?', to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be '?'. For example:
+```dart
+final task = await DownloadTask(url: 'https://google.com', filename: '?');
+final result = await FileDownloader().download(task);
+print('Suggested filename=${result.task.filename}'); // note we don't use 'task', but 'result.task'
+print('Wrong use filename=${task.filename}'); // this will print '?' as 'task' hasn't changed
+```
 
 ### A batch of files
 

android/src/main/kotlin/com/bbflight/background_downloader/DownloadTaskWorker.kt

@@ -101,9 +101,10 @@ class DownloadTaskWorker(applicationContext: Context, workerParams: WorkerParame
                 )
                 val dirName = File(filePath).parent ?: ""
                 destFilePath = "$dirName/${task.filename}"
+                Log.d(TAG, "Suggested filename for taskId ${task.taskId}: ${task.filename}")
             }
             // determine tempFile
-            val contentLength = connection.contentLengthLong
+            val contentLength = getContentLength(connection.headerFields, task)
             val applicationSupportPath =
                 baseDirPath(applicationContext, BaseDirectory.applicationSupport)
             val cachePath = baseDirPath(applicationContext, BaseDirectory.temporary)

android/src/main/kotlin/com/bbflight/background_downloader/Helpers.kt

@@ -52,7 +52,6 @@ fun acceptUntrustedCertificates() {
         val sslContext = SSLContext.getInstance("TLS")
         sslContext.init(null, trustAllCerts, java.security.SecureRandom())
         // Create an ssl socket factory with our all-trusting manager
-        val sslSocketFactory = sslContext.socketFactory
         HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.socketFactory)
         Log.w(
             BDPlugin.TAG, "Bypassing TLS certificate validation\n" +
@@ -103,6 +102,39 @@ fun parseRange(rangeStr: String): Pair<Long, Long?> {
     return Pair(start, end)
 }
 
+/**
+ * Returns the content length extracted from the [responseHeaders], or from
+ * the [task] headers
+ */
+fun getContentLength(responseHeaders: Map<String, List<String>>, task: Task): Long {
+    // if response provides contentLength, return it
+    val contentLength = responseHeaders["Content-Length"]?.get(0)?.toLongOrNull()
+        ?: responseHeaders["content-length"]?.get(0)?.toLongOrNull()
+        ?: -1L
+    if (contentLength != -1L) {
+        return contentLength
+    }
+    // try extracting it from Range header
+    val taskRangeHeader = task.headers["Range"] ?: task.headers["range"] ?: ""
+    val taskRange = parseRange(taskRangeHeader)
+    if (taskRange.second != null) {
+        val rangeLength = taskRange.second!! - taskRange.first + 1L
+        Log.d(TAG, "TaskId ${task.taskId} contentLength set to $rangeLength based on Range header")
+        return rangeLength
+    }
+    // try extracting it from a special "Known-Content-Length" header
+    val knownLength = (task.headers["Known-Content-Length"]?.toLongOrNull()
+        ?: task.headers["known-content-length"]?.toLongOrNull()
+        ?: -1)
+    if (knownLength != -1L) {
+        Log.d(TAG, "TaskId ${task.taskId} contentLength set to $knownLength based on Known-Content-Length header")
+    } else {
+        Log.d(TAG, "TaskId ${task.taskId} contentLength undetermined")
+    }
+    return knownLength
+}
+
+
 /**
  * Return the path to the baseDir for this [baseDirectory], or null if path could not be reached
  *

android/src/main/kotlin/com/bbflight/background_downloader/ParallelDownloadTaskWorker.kt

@@ -110,7 +110,8 @@ class ParallelDownloadTaskWorker(applicationContext: Context, workerParams: Work
                             )
                             val errorContent = responseErrorContent(connection)
                             taskException = TaskException(
-                                ExceptionType.httpResponse, httpResponseCode = connection.responseCode,
+                                ExceptionType.httpResponse,
+                                httpResponseCode = connection.responseCode,
                                 description = if (errorContent?.isNotEmpty() == true) errorContent else connection.responseMessage
                             )
                             if (connection.responseCode == 404) {
@@ -181,7 +182,12 @@ class ParallelDownloadTaskWorker(applicationContext: Context, workerParams: Work
     /**
      * Process incoming [status] update for a chunk with [chunkTaskId]
      */
-    suspend fun chunkStatusUpdate(chunkTaskId: String, status: TaskStatus, taskException: TaskException?, responseBody: String?) {
+    suspend fun chunkStatusUpdate(
+        chunkTaskId: String,
+        status: TaskStatus,
+        taskException: TaskException?,
+        responseBody: String?
+    ) {
         val chunk = chunks.firstOrNull { it.task.taskId == chunkTaskId }
             ?: return // chunk is not part of this parent task
         val chunkTask = chunk.task
@@ -386,18 +392,22 @@ class ParallelDownloadTaskWorker(applicationContext: Context, workerParams: Work
     }
 
 
+    /**
+     * Returns a list of chunk information for this task, and sets
+     * [parallelDownloadContentLength] to the total length of the download
+     *
+     * Throws a IllegalStateException if any information is missing, which should lead
+     * to a failure of the [ParallelDownloadTask]
+     */
     private fun createChunks(
         task: Task,
         headers: MutableMap<String, MutableList<String>>
     ): List<Chunk> {
         val numChunks = task.urls.size * task.chunks
         try {
-            val contentLength = headers.entries
-                .first { it.key == "content-length" || it.key == "Content-Length" }
-                .value
-                .first().toLong()
+            val contentLength = getContentLength(headers, task)
             if (contentLength <= 0) {
-                throw IllegalStateException("Server does not provide content length - cannot chunk download")
+                throw IllegalStateException("Server does not provide content length - cannot chunk download. If you know the length, set Range or Known-Content-Length header")
             }
             parallelDownloadContentLength = contentLength
             try {
@@ -417,7 +427,7 @@ class ParallelDownloadTaskWorker(applicationContext: Context, workerParams: Work
                 )
             }
         } catch (e: NoSuchElementException) {
-            throw IllegalStateException("Server does not provide content length - cannot chunk download")
+            throw IllegalStateException("Server does not provide content length - cannot chunk download. If you know the length, set Range or Known-Content-Length header")
         }
     }
 }

android/src/test/kotlin/com/bbflight/background_downloader/HelpersTest.kt

@@ -3,13 +3,36 @@ package com.bbflight.background_downloader
 import org.junit.Test
 import org.junit.Assert.assertEquals
 
+val urlWithContentLength = "https://storage.googleapis.com/approachcharts/test/5MB-test.ZIP"
+var task = Task(mapOf("url" to urlWithContentLength, "filename" to "taskFilename.txt"))
+
+
 class HelpersTest {
 
     @Test
     fun parseRange() {
-        assertEquals(parseRange("bytes=10-20"), Pair(10L, 20L))
-        assertEquals(parseRange("bytes=-20"), Pair(0L, 20L))
-        assertEquals(parseRange("bytes=10-"), Pair(10L, null))
-        assertEquals(parseRange(""), Pair(0L, null))
+        assertEquals(Pair(10L, 20L), parseRange("bytes=10-20"))
+        assertEquals(Pair(0L, 20L), parseRange("bytes=-20"))
+        assertEquals(Pair(10L, null), parseRange("bytes=10-"))
+        assertEquals(Pair(0L, null), parseRange(""))
+    }
+
+    @Test
+    fun getContentLength() {
+        var h = mapOf<String, List<String>>()
+        assertEquals(-1, getContentLength(h, task))
+        h = mapOf("Content-Length" to listOf("123"))
+        assertEquals(123, getContentLength(h, task))
+        h = mapOf("content-length" to listOf("123"))
+        assertEquals(123, getContentLength(h, task))
+        task = task.copyWith(headers = mapOf("Range" to "bytes=0-20"))
+        h = mapOf()
+        assertEquals(21, getContentLength(h, task))
+        task = task.copyWith(headers = mapOf("Known-Content-Length" to "456"))
+        assertEquals(456, getContentLength(h, task))
+        task = task.copyWith(headers = mapOf("Known-Content-Length" to "456", "Range" to "bytes=0-20"))
+        assertEquals(21, getContentLength(h, task))
+        h = mapOf("Content-Length" to listOf("123"))
+        assertEquals(123, getContentLength(h, task))
     }
 }

example/integration_test/downloader_integration_test.dart

@@ -843,13 +843,6 @@ void main() {
       /// which is what the server expects
       expect(lastStatus, equals(TaskStatus.notFound));
     });
-
-    testWidgets('DownloadTask expectedFileSize', (widgetTester) async {
-      expect(await task.expectedFileSize(), equals(-1));
-      task = DownloadTask(url: urlWithContentLength);
-      expect(
-          await task.expectedFileSize(), equals(urlWithContentLengthFileSize));
-    });
   });
 
   group('Convenience downloads', () {
@@ -2715,7 +2708,7 @@ void main() {
     });
   });
 
-  group('Range', () {
+  group('Range and Content-Length headers', () {
     testWidgets('parseRange', (widgetTester) async {
       // tested on the native side for Android and iOS
       if (Platform.isLinux || Platform.isMacOS || Platform.isWindows) {
@@ -2786,6 +2779,17 @@ void main() {
       await file.delete();
     });
 
+    testWidgets('DownloadTask expectedFileSize', (widgetTester) async {
+      expect(await task.expectedFileSize(), equals(-1));
+      task = task.copyWith(headers: {'Range': 'bytes=0-10'});
+      expect(await task.expectedFileSize(), equals(11));
+      task = task.copyWith(headers: {'Known-Content-Length': '100'});
+      expect(await task.expectedFileSize(), equals(100));
+      task = DownloadTask(url: urlWithContentLength);
+      expect(
+          await task.expectedFileSize(), equals(urlWithContentLengthFileSize));
+    });
+
     testWidgets('[*] Range or Known-Content-Length in task header',
         (widgetTester) async {
       // Haven't found a url that does not provide content-length, so

example/integration_test/parallel_download_test.dart

@@ -273,13 +273,6 @@ void main() {
       }
     });
 
-    test('[*] retries - must modify transferBytes to fail', () async {
-      FileDownloader().registerCallbacks(taskStatusCallback: statusCallback);
-      expect(await FileDownloader().enqueue(retryTask), isTrue);
-      await statusCallbackCompleter.future;
-      expect(lastStatus, equals(TaskStatus.failed));
-    });
-
     test('cancellation', () async {
       FileDownloader().registerCallbacks(
           taskStatusCallback: statusCallback,
@@ -323,6 +316,54 @@ void main() {
       expect(await file.length(), equals(urlWithLongContentLengthFileSize));
     });
   });
+
+  group('Modifications', () {
+    // Tests in this group require modification of the source code
+    // and may fail without that
+    test('[*] retries - must modify transferBytes to fail', () async {
+      // modify the TransferBytes method to fail, otherwise retries are not
+      // triggered
+      FileDownloader().registerCallbacks(taskStatusCallback: statusCallback);
+      expect(await FileDownloader().enqueue(retryTask), isTrue);
+      await statusCallbackCompleter.future;
+      expect(lastStatus, equals(TaskStatus.failed));
+    });
+
+    test('[*] Range or Known-Content-Length in task header', () async {
+      // modify the getContentLength function to not return a
+      // content length, so that it relies on Range or
+      // Known-Content-Length header to determine content length
+      FileDownloader().registerCallbacks(
+          taskStatusCallback: statusCallback,
+          taskProgressCallback: progressCallback);
+      // try with Range header
+      task = task.copyWith(
+          taskId: "1",
+          url: urlWithContentLength,
+          headers: {'Range': 'bytes=0-${urlWithContentLengthFileSize - 1}'},
+          updates: Updates.statusAndProgress);
+      expect(await FileDownloader().enqueue(task), isTrue);
+      await statusCallbackCompleter.future;
+      expect(lastValidExpectedFileSize, equals(urlWithContentLengthFileSize));
+      // try with Known-Content-Length header
+      statusCallbackCompleter = Completer();
+      lastValidExpectedFileSize = -1;
+      task = task.copyWith(
+          taskId: "2",
+          headers: {'Known-Content-Length': '$urlWithContentLengthFileSize'});
+      expect(await FileDownloader().enqueue(task), isTrue);
+      await statusCallbackCompleter.future;
+      expect(lastValidExpectedFileSize, equals(urlWithContentLengthFileSize));
+      // try without any header, and if the source code mod has been made, this
+      // should fail the task
+      statusCallbackCompleter = Completer();
+      lastValidExpectedFileSize = -1;
+      task = task.copyWith(taskId: "3", headers: {});
+      expect(await FileDownloader().enqueue(task), isTrue);
+      await statusCallbackCompleter.future;
+      expect(lastStatus, equals(TaskStatus.failed));
+    });
+  });
 }
 
 /// Returns true if the supplied file equals the test file

lib/src/desktop/download_isolate.dart

@@ -81,6 +81,8 @@ Future<void> doDownloadTask(
                 unique: true);
         // update the filePath by replacing the last segment with the new filename
         filePath = p.join(p.dirname(filePath), downloadTask.filename);
+        log.finest(
+            'Suggested filename for taskId ${task.taskId}: ${task.filename}');
       }
       if (okResponses.contains(response.statusCode)) {
         resultStatus = await processOkDownloadResponse(

lib/src/desktop/isolate.dart

@@ -408,11 +408,21 @@ int getContentLength(Map<String, String> responseHeaders, Task task) {
   final taskRangeHeader = task.headers['Range'] ?? task.headers['range'] ?? '';
   final taskRange = parseRange(taskRangeHeader);
   if (taskRange.$2 != null) {
-    return taskRange.$2! - taskRange.$1 + 1;
+    var rangeLength = taskRange.$2! - taskRange.$1 + 1;
+    log.finest(
+        'TaskId ${task.taskId} contentLength set to $rangeLength based on Range header');
+    return rangeLength;
   }
   // try extracting it from a special "Known-Content-Length" header
-  return int.tryParse(task.headers['Known-Content-Length'] ??
+  var knownLength = int.tryParse(task.headers['Known-Content-Length'] ??
           task.headers['known-content-length'] ??
           '-1') ??
       -1;
+  if (knownLength != -1) {
+    log.finest(
+        'TaskId ${task.taskId} contentLength set to $knownLength based on Known-Content-Length header');
+  } else {
+    log.finest('TaskId ${task.taskId} contentLength undetermined');
+  }
+  return knownLength;
 }

lib/src/desktop/parallel_download_isolate.dart

@@ -352,7 +352,8 @@ List<Chunk> createChunks(
     final contentLength = getContentLength(headers, task);
     if (contentLength <= 0) {
       throw StateError(
-          'Server does not provide content length - cannot chunk download');
+          'Server does not provide content length - cannot chunk download. '
+          'If you know the length, set Range or Known-Content-Length header');
     }
     parallelDownloadContentLength = contentLength;
     try {
@@ -374,6 +375,7 @@ List<Chunk> createChunks(
     ];
   } on StateError {
     throw StateError(
-        'Server does not provide content length - cannot chunk download');
+        'Server does not provide content length - cannot chunk download. '
+        'If you know the length, set Range or Known-Content-Length header');
   }
 }

lib/src/models.dart

@@ -3,6 +3,7 @@ import 'dart:io';
 import 'dart:math';
 import 'dart:typed_data';
 
+import 'package:background_downloader/src/desktop/isolate.dart';
 import 'package:logging/logging.dart';
 import 'package:mime/mime.dart';
 import 'package:path/path.dart' as path;
@@ -730,10 +731,7 @@ final class DownloadTask extends Task {
       final response = await DesktopDownloader.httpClient
           .head(Uri.parse(url), headers: headers);
       if ([200, 201, 202, 203, 204, 205, 206].contains(response.statusCode)) {
-        return int.parse(response.headers.entries
-            .firstWhere(
-                (element) => element.key.toLowerCase() == 'content-length')
-            .value);
+        return getContentLength(response.headers, this);
       }
     } catch (e) {
       // no content length available