diff --git a/CHANGELOG.md b/CHANGELOG.md index 9288bb40..4b89df60 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,34 @@ +## 7.11.0 + +### Android external storage +Add configuration for Android to use external storage instead of internal storage. Either your app runs in default (internal storage) mode, or in external storage. You cannot switch between internal and external, as the directory structure that - for example - `BaseDirectory.applicationDocuments` refers to is different in each mode. See the [configuration document](https://github.com/781flyingdutchman/background_downloader/blob/main/CONFIG.md) for important details and limitations + +Use `(Config.useExternalStorage, String whenToUse)` with values 'never' or 'always'. Default is `Config.never`. + +### Server suggested filename +If you want the filename to be provided by the server (instead of assigning a value to `filename` yourself), you now have two options. The first is to create a `DownloadTask` that pings the server to determine the suggested filename: +```dart +final task = await DownloadTask(url: 'https://google.com') + .withSuggestedFilename(unique: true); +``` +The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible (e.g. due to a lack of connection). If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt". You can now also supply a `taskWithFilenameBuilder` to suggest the filename yourself, based on response headers. + +The second approach is to set the `filename` field of the `DownloadTask` to `DownloadTask.suggestedFilename`, to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be `DownloadTask.suggestedFilename`. For example: +```dart +final task = await DownloadTask(url: 'https://google.com', filename: DownloadTask.suggestedFilename); +final result = await FileDownloader().download(task); +print('Suggested filename=${result.task.filename}'); // note we don't use 'task', but 'result.task' +print('Wrong use filename=${task.filename}'); // this will print '?' as 'task' hasn't changed +``` + +### Set content length if not provided by server + +To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the download. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes. The downloader will assume this content length when calculating progress. + +### Bug fix + +Partial Downloads, using a Range header, can now be properly paused on all platforms. + ## 7.10.1 Add `displayName` field to `Task` that can be used to store and dsipay a 'human readable' description of the task. It can be displayed in a notification using {displayName}. diff --git a/README.md b/README.md index 01b45f41..0a85474a 100644 --- a/README.md +++ b/README.md @@ -234,7 +234,7 @@ A [DownloadProgressIndicator](https://pub.dev/documentation/background_downloade The widget can be configured to include pause and resume buttons, and to expand to show multiple simultaneous downloads, or to collapse and show a file download counter. -To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the donwload. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes. The downloader will assume this content length when calculating progress. +To provide progress updates (as a percentage of total file size) the downloader needs to know the size of the file when starting the download. Most servers provide this in the "Content-Length" header of their response. If the server does not provide the file size, yet you know the file size (e.g. because you have stored the file on the server yourself), then you can let the downloader know by providing a `{'Range': 'bytes=0-999'}` or a `{'Known-Content-Length': '1000'}` header to the task's `header` field. Both examples are for a content length of 1000 bytes. The downloader will assume this content length when calculating progress. #### Status @@ -299,11 +299,11 @@ If you want the filename to be provided by the server (instead of assigning a va final task = await DownloadTask(url: 'https://google.com') .withSuggestedFilename(unique: true); ``` -The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible. If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt". +The method `withSuggestedFilename` returns a copy of the task it is called on, with the `filename` field modified based on the filename suggested by the server, or the last path segment of the URL, or unchanged if neither is feasible (e.g. due to a lack of connection). If `unique` is true, the filename will be modified such that it does not conflict with an existing filename by adding a sequence. For example "file.txt" would become "file (1).txt". You can also supply a `taskWithFilenameBuilder` to suggest the filename yourself, based on response headers. -The second approach is to set the `filename` field of the `DownloadTask` to '?', to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be '?'. For example: +The second approach is to set the `filename` field of the `DownloadTask` to `DownloadTask.suggestedFilename`, to indicate that you would like the server to suggest the name. In this case, you will receive the name via the task's status and/or progress updates, so you have to be careful _not_ to use the original task's filename, as that will still be `DownloadTask.suggestedFilename`. For example: ```dart -final task = await DownloadTask(url: 'https://google.com', filename: '?'); +final task = await DownloadTask(url: 'https://google.com', filename: DownloadTask.suggestedFilename); final result = await FileDownloader().download(task); print('Suggested filename=${result.task.filename}'); // note we don't use 'task', but 'result.task' print('Wrong use filename=${task.filename}'); // this will print '?' as 'task' hasn't changed diff --git a/example/integration_test/downloader_integration_test.dart b/example/integration_test/downloader_integration_test.dart index c3c5bfce..99ae3184 100644 --- a/example/integration_test/downloader_integration_test.dart +++ b/example/integration_test/downloader_integration_test.dart @@ -2112,7 +2112,7 @@ void main() { taskProgressCallback: progressCallback); task = DownloadTask( url: urlWithContentLength, - filename: '?', + filename: DownloadTask.suggestedFilename, updates: Updates.statusAndProgress, allowPause: true); expect(await FileDownloader().enqueue(task), equals(true)); @@ -2698,7 +2698,8 @@ void main() { try { File(await task.filePath()).deleteSync(); } catch (e) {} - task = DownloadTask(url: urlWithContentLength, filename: '?'); + task = DownloadTask( + url: urlWithContentLength, filename: DownloadTask.suggestedFilename); final result = await FileDownloader().download(task); expect(result.task.filename, equals('5MB-test.ZIP')); final file = File(await result.task.filePath()); @@ -2793,7 +2794,7 @@ void main() { testWidgets('[*] Range or Known-Content-Length in task header', (widgetTester) async { // Haven't found a url that does not provide content-length, so - // can oly be tested by modifying the source code to ignore the + // can only be tested by modifying the source code to ignore the // Content-Length response header and use this one instead FileDownloader().registerCallbacks( taskStatusCallback: statusCallback, diff --git a/example/integration_test/parallel_download_test.dart b/example/integration_test/parallel_download_test.dart index 17fc513c..ce79c6e7 100644 --- a/example/integration_test/parallel_download_test.dart +++ b/example/integration_test/parallel_download_test.dart @@ -158,23 +158,6 @@ void main() { expect(await fileEqualsTestFile(file), isTrue); }); - test('[*] override content length', () async { - // Haven't found a url that does not provide content-length, so - // can oly be tested by modifying the source code to ignore the - // Content-Length response header and use this one instead - FileDownloader().registerCallbacks(taskStatusCallback: statusCallback); - task = task.copyWith( - url: urlWithContentLength, - headers: {'Known-Content-Length': '$urlWithContentLengthFileSize'}); - expect(await FileDownloader().enqueue(task), isTrue); - await statusCallbackCompleter.future; - expect(lastStatus, equals(TaskStatus.complete)); - expect(statusCallbackCounter, equals(3)); - final file = File(await task.filePath()); - expect(file.existsSync(), isTrue); - expect(await fileEqualsTestFile(file), isTrue); - }); - test('simple enqueue, 2 chunks, 2 url', () async { task = ParallelDownloadTask( url: [urlWithContentLength, urlWithContentLength], @@ -359,9 +342,31 @@ void main() { statusCallbackCompleter = Completer(); lastValidExpectedFileSize = -1; task = task.copyWith(taskId: "3", headers: {}); + if (!Platform.isIOS) { + expect(await FileDownloader().enqueue(task), isTrue); + await statusCallbackCompleter.future; + expect(lastStatus, equals(TaskStatus.failed)); + } else { + // on iOS the task fails at enqueue + expect(await FileDownloader().enqueue(task), isFalse); + } + }); + + test('[*] override content length', () async { + // Haven't found a url that does not provide content-length, so + // can only be tested by modifying the source code to ignore the + // Content-Length response header and use this one instead + FileDownloader().registerCallbacks(taskStatusCallback: statusCallback); + task = task.copyWith( + url: urlWithContentLength, + headers: {'Known-Content-Length': '$urlWithContentLengthFileSize'}); expect(await FileDownloader().enqueue(task), isTrue); await statusCallbackCompleter.future; - expect(lastStatus, equals(TaskStatus.failed)); + expect(lastStatus, equals(TaskStatus.complete)); + expect(statusCallbackCounter, equals(3)); + final file = File(await task.filePath()); + expect(file.existsSync(), isTrue); + expect(await fileEqualsTestFile(file), isTrue); }); }); } diff --git a/ios/Classes/Downloader.swift b/ios/Classes/Downloader.swift index 9fff9b0b..e0216301 100644 --- a/ios/Classes/Downloader.swift +++ b/ios/Classes/Downloader.swift @@ -40,6 +40,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession static var localResumeData = [String : String]() // locally stored to enable notification resume static var remainingBytesToDownload = [String : Int64]() // keyed by taskId static var responseBodyData = [String: [Data]]() // list of Data objects received for this UploadTask id + static var tasksWithSuggestedFilename = [String : Task]() // [taskId : Task with suggested filename] + static var tasksWithContentLengthOverride = [String : Int64]() // [taskId : Content length] public static func register(with registrar: FlutterPluginRegistrar) { let channel = FlutterMethodChannel(name: "com.bbflight.background_downloader", binaryMessenger: registrar.messenger()) @@ -111,6 +113,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession methodStoreConfig(key: Downloader.keyConfigCheckAvailableSpace, value: call.arguments, result: result) case "forceFailPostOnBackgroundChannel": methodForceFailPostOnBackgroundChannel(call: call, result: result) + case "testSuggestedFilename": + methodTestSuggestedFilename(call: call, result: result) default: os_log("Invalid method: %@", log: log, type: .error, call.method) result(FlutterMethodNotImplemented) @@ -484,6 +488,22 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession result(nil) } + /// Tests the content-disposition and url translation + /// + /// For testing only + private func methodTestSuggestedFilename(call: FlutterMethodCall, result: @escaping FlutterResult) { + let args = call.arguments as! [Any] + guard let taskJsonString = args[0] as? String, + let contentDisposition = args[1] as? String, + let task = taskFrom(jsonString: taskJsonString) else { + result("") + return + } + let resultTask = suggestedFilenameFromResponseHeaders(task: task, responseHeaders: ["Content-Disposition" : contentDisposition], unique: true) + result(resultTask.filename) + } + + //MARK: Helpers for Task and urlSessionTask @@ -550,6 +570,9 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession os_log("Could not find task related to urlSessionTask %d", log: log, type: .error, task.taskIdentifier) return } + // clear storage related to this task + Downloader.tasksWithSuggestedFilename.removeValue(forKey: task.taskId) + Downloader.tasksWithContentLengthOverride.removeValue(forKey: task.taskId) let responseBody = getResponseBody(taskId: task.taskId) Downloader.responseBodyData.removeValue(forKey: task.taskId) Downloader.taskIdsThatCanResume.remove(task.taskId) @@ -618,9 +641,28 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession /// If the task requires progress updates, provide these at a reasonable interval /// If this is the first update for this file, also emit a 'running' status update public func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didWriteData bytesWritten: Int64, totalBytesWritten: Int64, totalBytesExpectedToWrite: Int64) { - guard let task = getTaskFrom(urlSessionTask: downloadTask) else { return } + // task is var because the filename can be changed on the first 'didWriteData' call + guard var task = getTaskFrom(urlSessionTask: downloadTask) else { return } if Downloader.progressInfo[task.taskId] == nil { // first 'didWriteData' call + let response = downloadTask.response as! HTTPURLResponse + // get suggested filename if needed + if task.filename == "?" { + let newTask = suggestedFilenameFromResponseHeaders(task: task, responseHeaders: response.allHeaderFields) + os_log("Suggested task filename for taskId %@ is %@", log: log, type: .info, newTask.taskId, newTask.filename) + if newTask.filename != task.filename { + // store for future replacement, and replace now + Downloader.tasksWithSuggestedFilename[newTask.taskId] = newTask + task = newTask + } + } + // obtain content length override, if needed and available + if totalBytesExpectedToWrite == -1 { + let contentLength = getContentLength(responseHeaders: response.allHeaderFields, task: task) + if contentLength != -1 { + Downloader.tasksWithContentLengthOverride[task.taskId] = contentLength + } + } // Check if there is enough space if insufficientSpace(contentLength: totalBytesExpectedToWrite) { if !Downloader.taskIdsProgrammaticallyCancelled.contains(task.taskId) { @@ -649,8 +691,9 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession updateNotification(task: task, notificationType: .running, notificationConfig: notificationConfig) } } - Downloader.remainingBytesToDownload[task.taskId] = totalBytesExpectedToWrite - totalBytesWritten - updateProgress(task: task, totalBytesExpected: totalBytesExpectedToWrite, totalBytesDone: totalBytesWritten) + let contentLength = totalBytesExpectedToWrite != -1 ? totalBytesExpectedToWrite : Downloader.tasksWithContentLengthOverride[task.taskId] ?? -1 + Downloader.remainingBytesToDownload[task.taskId] = contentLength - totalBytesWritten + updateProgress(task: task, totalBytesExpected: contentLength, totalBytesDone: totalBytesWritten) } /// Process taskdelegate progress update for upload task @@ -686,6 +729,8 @@ public class Downloader: NSObject, FlutterPlugin, URLSessionDelegate, URLSession else { os_log("Could not find task associated urlSessionTask %d, or did not get HttpResponse", log: log, type: .info, downloadTask.taskIdentifier) return} + Downloader.tasksWithSuggestedFilename.removeValue(forKey: task.taskId) + Downloader.tasksWithContentLengthOverride.removeValue(forKey: task.taskId) let notificationConfig = getNotificationConfigFrom(urlSessionTask: downloadTask) if response.statusCode == 404 { let responseBody = readFile(url: location) diff --git a/ios/Classes/ParallelDownloader.swift b/ios/Classes/ParallelDownloader.swift index 4f9652b4..d129dd10 100644 --- a/ios/Classes/ParallelDownloader.swift +++ b/ios/Classes/ParallelDownloader.swift @@ -21,9 +21,9 @@ func scheduleParallelDownload(task: Task, taskDescription: String, baseRequest: os_log("URL not found for taskId %@", log: log, type: .info, task.taskId) postResult(result: result, value: false) } - else if !parallelDownload.start(contentLength: Int64(httpResponse.value(forHTTPHeaderField: "Content-Length") ?? "0") ?? 0) { - os_log("Cannot chunk or enqueue download", log: log, type: .info) - postResult(result: result, value: false) + else if !parallelDownload.start(contentLengthFromHeader: Int64(httpResponse.value(forHTTPHeaderField: "Content-Length") ?? "-1") ?? -1, responseHeaders: httpResponse.allHeaderFields ) { + os_log("Cannot chunk or enqueue download", log: log, type: .info) + postResult(result: result, value: false) } else { processStatusUpdate(task: task, status: TaskStatus.enqueued) postResult(result: result, value: true) @@ -85,10 +85,12 @@ public class ParallelDownloader: NSObject { /// the /// /// Returns false if start was unsuccessful - public func start(contentLength: Int64) -> Bool { - parallelDownloadContentLength = contentLength + public func start(contentLengthFromHeader: Int64, responseHeaders: [AnyHashable: Any]) -> Bool { + parallelDownloadContentLength = contentLengthFromHeader > 0 ? + contentLengthFromHeader : + getContentLength(responseHeaders: responseHeaders, task: self.parentTask) ParallelDownloader.downloads[parentTask.taskId] = self - chunks = createChunks(task: parentTask, contentLength: contentLength) + chunks = createChunks(task: parentTask, contentLength: parallelDownloadContentLength) let success = !chunks.isEmpty && enqueueChunkTasks() if !success { ParallelDownloader.downloads.removeValue(forKey: parentTask.taskId) diff --git a/ios/Classes/Task.swift b/ios/Classes/Task.swift index 306fa636..ec0ea4af 100644 --- a/ios/Classes/Task.swift +++ b/ios/Classes/Task.swift @@ -35,6 +35,129 @@ struct Task : Codable { var taskType: String } +extension Task { + func copyWith(taskId: String? = nil, + url: String? = nil, + urls: [String]? = nil, + filename: String? = nil, + headers: [String:String]? = nil, + httpRequestMethod: String? = nil, + chunks: Int? = nil, + post: String? = nil, + fileField: String? = nil, + mimeType: String? = nil, + fields: [String:String]? = nil, + directory: String? = nil, + baseDirectory: Int? = nil, + group: String? = nil, + updates: Int? = nil, + requiresWiFi: Bool? = nil, + retries: Int? = nil, + retriesRemaining: Int? = nil, + allowPause: Bool? = nil, + metaData: String? = nil, + displayName: String? = nil, + creationTime: Int64? = nil, + taskType: String? = nil) -> Task { + + var copiedTask = self + + if let taskId = taskId { + copiedTask.taskId = taskId + } + + if let url = url { + copiedTask.url = url + } + + if let urls = urls { + copiedTask.urls = urls + } + + if let filename = filename { + copiedTask.filename = filename + } + + if let headers = headers { + copiedTask.headers = headers + } + + if let httpRequestMethod = httpRequestMethod { + copiedTask.httpRequestMethod = httpRequestMethod + } + + if let chunks = chunks { + copiedTask.chunks = chunks + } + + if let post = post { + copiedTask.post = post + } + + if let fileField = fileField { + copiedTask.fileField = fileField + } + + if let mimeType = mimeType { + copiedTask.mimeType = mimeType + } + + if let fields = fields { + copiedTask.fields = fields + } + + if let directory = directory { + copiedTask.directory = directory + } + + if let baseDirectory = baseDirectory { + copiedTask.baseDirectory = baseDirectory + } + + if let group = group { + copiedTask.group = group + } + + if let updates = updates { + copiedTask.updates = updates + } + + if let requiresWiFi = requiresWiFi { + copiedTask.requiresWiFi = requiresWiFi + } + + if let retries = retries { + copiedTask.retries = retries + } + + if let retriesRemaining = retriesRemaining { + copiedTask.retriesRemaining = retriesRemaining + } + + if let allowPause = allowPause { + copiedTask.allowPause = allowPause + } + + if let metaData = metaData { + copiedTask.metaData = metaData + } + + if let displayName = displayName { + copiedTask.displayName = displayName + } + + if let creationTime = creationTime { + copiedTask.creationTime = creationTime + } + + if let taskType = taskType { + copiedTask.taskType = taskType + } + + return copiedTask + } +} + /// Base directory in which files will be stored, based on their relative /// path. /// diff --git a/ios/Classes/TaskFunctions.swift b/ios/Classes/TaskFunctions.swift index 19f52aa8..1723e4c5 100644 --- a/ios/Classes/TaskFunctions.swift +++ b/ios/Classes/TaskFunctions.swift @@ -78,6 +78,183 @@ func getFilePath(for task: Task, withFilename: String? = nil) -> String? { return directory.appendingPathComponent(withFilename ?? task.filename).path } +//extension StringProtocol { +// func index(of string: S, options: String.CompareOptions = []) -> Index? { +// range(of: string, options: options)?.lowerBound +// } +// func endIndex(of string: S, options: String.CompareOptions = []) -> Index? { +// range(of: string, options: options)?.upperBound +// } +// func indices(of string: S, options: String.CompareOptions = []) -> [Index] { +// ranges(of: string, options: options).map(\.lowerBound) +// } +// func ranges(of string: S, options: String.CompareOptions = []) -> [Range] { +// var result: [Range] = [] +// var startIndex = self.startIndex +// while startIndex < endIndex, +// let range = self[startIndex...] +// .range(of: string, options: options) { +// result.append(range) +// startIndex = range.lowerBound < range.upperBound ? range.upperBound : +// index(range.lowerBound, offsetBy: 1, limitedBy: endIndex) ?? endIndex +// } +// return result +// } +//} + +func stripFileExtension ( _ filename: String ) -> String { + var components = filename.components(separatedBy: ".") + guard components.count > 1 else { return filename } + components.removeLast() + return components.joined(separator: ".") +} + +/** + * Returns a copy of the task with the [Task.filename] property changed + * to the filename suggested by the server, or derived from the url, or + * unchanged. + * + * If [unique] is true, the filename is guaranteed not to already exist. This + * is accomplished by adding a suffix to the suggested filename with a number, + * e.g. "data (2).txt" + * + * The server-suggested filename is obtained from the [responseHeaders] entry + * "Content-Disposition" + */ +func suggestedFilenameFromResponseHeaders( + task: Task, + responseHeaders: [AnyHashable: Any], + unique: Bool = false +) -> Task { + if let disposition = responseHeaders["Content-Disposition"] as? String { + let range = NSMakeRange(0, disposition.utf16.count) + // Try filename="filename" + let plainFilenameRegEx = try! NSRegularExpression(pattern: #"filename=\s*"?([^"]+)"?.*$"#, options: .caseInsensitive) + if let match = plainFilenameRegEx.firstMatch(in: disposition, options: [], range: range) { + let filename = String(disposition[Range(match.range(at: 1), in: disposition)!]) + return uniqueFilename(task: task.copyWith(filename: filename), unique: unique) + } + + // Try filename*=UTF-8'language'"encodedFilename" + let encodedFilenameRegEx = try! NSRegularExpression(pattern: #"filename\*=\s*([^']+)'([^']*)'"?([^"]+)"?"#, options: .caseInsensitive) + if let match = encodedFilenameRegEx.firstMatch(in: disposition, options: [], range: range) { + let encoding = String(disposition[Range(match.range(at: 1), in: disposition)!]).uppercased() + let filename = String(disposition[Range(match.range(at: 3), in: disposition)!]) + if encoding == "UTF-8" { + if let decodedFilename = filename.removingPercentEncoding { + return uniqueFilename(task: task.copyWith(filename: decodedFilename), unique: unique) + } else { + os_log("Could not interpret suggested filename (UTF-8 url encoded)", log: log, type: .debug) + } + } else { + return uniqueFilename(task: task.copyWith(filename: filename), unique: unique) + } + } + } + os_log("Could not determine suggested filename from server", log: log, type: .debug) + // Try filename derived from last path segment of the url + if let uri = URL(string: task.url) { + let suggestedFilename = uri.lastPathComponent + if !suggestedFilename.isEmpty { + return uniqueFilename(task: task.copyWith(filename: suggestedFilename), unique: unique) + } + } + os_log("Could not parse URL pathSegment for suggested filename", log: log, type: .debug) + // if everything fails, return the task with unchanged filename + // except for possibly making it unique + return uniqueFilename(task: task, unique: unique) +} + +/// Returns [Task] with a filename similar to the one +/// supplied, but unused. +/// +/// If [unique], filename will sequence up in "filename (8).txt" format, +/// otherwise returns the [task] +func uniqueFilename(task: Task, unique: Bool) -> Task { + if !unique { + return task + } + let sequenceRegEx = try! NSRegularExpression(pattern: #"\((\d+)\)\.?[^.]*$"#) + let extensionRegEx = try! NSRegularExpression(pattern: #"\.[^.]*$"#) + var newTask = task + guard let filePath = getFilePath(for: task) else { + return task + } + var exists = FileManager.default.fileExists(atPath: filePath) + + while exists { + let range = NSMakeRange(0, newTask.filename.utf16.count) + let extMatch = extensionRegEx.firstMatch(in: newTask.filename, options: [], range: range) + let extString = extMatch != nil ? String(newTask.filename[Range(extMatch!.range, in: newTask.filename)!]) : "" + let seqMatch = sequenceRegEx.firstMatch(in: newTask.filename, options: [], range: range) + let seqString = seqMatch != nil ? String(newTask.filename[Range(seqMatch!.range, in: newTask.filename)!]) : "" + let newSequence = seqString.isEmpty ? 1 : Int(seqString)! + 1 + + let newFilename: String + if seqMatch == nil { + let baseNameWithoutExtension = stripFileExtension(newTask.filename) + newFilename = "\(baseNameWithoutExtension) (\(newSequence))\(extString)" + } else { + let startOfSeq = seqMatch!.range.location + let index = newTask.filename.index(newTask.filename.startIndex, offsetBy: startOfSeq) + newFilename = "\(newTask.filename.prefix(upTo: index)) (\(newSequence))\(extString)" + } + newTask = newTask.copyWith(filename: newFilename) + guard let filePath = getFilePath(for: task) else { + return task + } + exists = FileManager.default.fileExists(atPath: filePath) + } + return newTask +} + +/** + * Parses the range in a Range header, and returns a Pair representing + * the range. The format needs to be "bytes=10-20" + * + * A missing lower range is substituted with 0L, and a missing upper + * range with null. If the string cannot be parsed, returns (0L, null) + */ +func parseRange(rangeStr: String) -> (Int64, Int64?) { + let regex = try! NSRegularExpression(pattern: #"bytes=(\d*)-(\d*)"#) + let range = NSMakeRange(0, rangeStr.utf16.count) + if let match = regex.firstMatch(in: rangeStr, options: [], range: range) { + let start = Int64(String(rangeStr[Range(match.range(at: 1), in: rangeStr)!])) + let end = Int64(String(rangeStr[Range(match.range(at: 2), in: rangeStr)!])) + return (start!, end!) + } else { + return (0, nil) + } +} + +/// Returns the content length extracted from the [responseHeaders], or from +/// the [task] headers +func getContentLength(responseHeaders: [AnyHashable: Any], task: Task) -> Int64 { + // On iOS, the header has already been parsed for Content-Length so we don't need to + // repeat that here + // try extracting it from Range header + let taskRangeHeader = task.headers["Range"] ?? "" + let taskRange = parseRange(rangeStr: taskRangeHeader) + if let end = taskRange.1 { + var rangeLength = end - taskRange.0 + 1 + os_log("TaskId %@ contentLength set to %d based on Range header", log: log, type: .info, task.taskId, rangeLength) + return rangeLength + } + + // try extracting it from a special "Known-Content-Length" header + let knownLength = Int64(task.headers["Known-Content-Length"] ?? "-1") ?? -1 + if knownLength != -1 { + os_log("TaskId %@ contentLength set to %d based on Known-Content-Length header", log: log, type: .info, task.taskId, knownLength) + } else { + os_log("TaskId %@ contentLength undetermined", log: log, type: .info, task.taskId) + } + return knownLength +} + + + + + /// Returns a list of fileData elements, one for each file to upload. /// Each element is a triple containing fileField, full filePath, mimeType /// @@ -318,13 +495,18 @@ func taskFrom(jsonString: String) -> Task? { } /// Return the task corresponding to the URLSessionTask, or nil if it cannot be matched +/// +/// If possible, the task returned contains the suggested filename func getTaskFrom(urlSessionTask: URLSessionTask) -> Task? { guard let jsonData = getTaskJsonStringFrom(urlSessionTask: urlSessionTask)?.data(using: .utf8) else { return nil } let decoder = JSONDecoder() - return try? decoder.decode(Task.self, from: jsonData) + if let task = try? decoder.decode(Task.self, from: jsonData) { + return Downloader.tasksWithSuggestedFilename[task.taskId] ?? task + } + return nil } /// Returns the taskJsonString contained in the urlSessionTask diff --git a/lib/src/desktop/desktop_downloader.dart b/lib/src/desktop/desktop_downloader.dart index f3e7d365..bb48da5b 100644 --- a/lib/src/desktop/desktop_downloader.dart +++ b/lib/src/desktop/desktop_downloader.dart @@ -426,7 +426,7 @@ final class DesktopDownloader extends BaseDownloader { final h = contentDisposition.isNotEmpty ? {'Content-disposition': contentDisposition} : {}; - final t = await task.withSuggestedFilenameFromResponseHeaders(h); + final t = await taskWithSuggestedFilename(task, h, false); return t.filename; } diff --git a/lib/src/desktop/download_isolate.dart b/lib/src/desktop/download_isolate.dart index 76823e67..fa805c6a 100644 --- a/lib/src/desktop/download_isolate.dart +++ b/lib/src/desktop/download_isolate.dart @@ -76,9 +76,8 @@ Future doDownloadTask( throw TaskException('Cannot resume: ETag is not identical, or is weak'); } if (!downloadTask.hasFilename) { - downloadTask = await downloadTask - .withSuggestedFilenameFromResponseHeaders(response.headers, - unique: true); + downloadTask = await taskWithSuggestedFilename( + downloadTask, response.headers, true); // update the filePath by replacing the last segment with the new filename filePath = p.join(p.dirname(filePath), downloadTask.filename); log.finest( diff --git a/lib/src/models.dart b/lib/src/models.dart index dcaedb1a..6942bb8a 100644 --- a/lib/src/models.dart +++ b/lib/src/models.dart @@ -619,107 +619,29 @@ final class DownloadTask extends Task { /// If [unique] is true, the filename is guaranteed not to already exist. This /// is accomplished by adding a suffix to the suggested filename with a number, /// e.g. "data (2).txt" + /// If a [taskWithFilenameBuilder] is supplied, this is the function called to + /// convert the task, response headers and [unique] values into a new [DownloadTask] + /// with the suggested filename. By default, [taskWithSuggestedFilename] is used, + /// which parses the Content-Disposition according to RFC6266, or takes the last + /// path segment of the URL, or leaves the filename unchanged /// /// The suggested filename is obtained by making a HEAD request to the url /// represented by the [DownloadTask], including urlQueryParameters and headers - Future withSuggestedFilename({unique = false}) async { + Future withSuggestedFilename( + {unique = false, + Future Function( + DownloadTask task, Map headers, bool unique) + taskWithFilenameBuilder = taskWithSuggestedFilename}) async { try { final response = await DesktopDownloader.httpClient .head(Uri.parse(url), headers: headers); if ([200, 201, 202, 203, 204, 205, 206].contains(response.statusCode)) { - return withSuggestedFilenameFromResponseHeaders(response.headers, - unique: unique); + return taskWithFilenameBuilder(this, response.headers, unique); } } catch (e) { _log.finer('Error connecting to server'); } - return withSuggestedFilenameFromResponseHeaders({}, unique: unique); - } - - /// Returns a copy of the task with the [Task.filename] property changed - /// to the filename suggested by the server, or derived from the url, or - /// unchanged. - /// - /// If [unique] is true, the filename is guaranteed not to already exist. This - /// is accomplished by adding a suffix to the suggested filename with a number, - /// e.g. "data (2).txt" - /// - /// The server-suggested filename is obtained from the [responseHeaders] entry - /// "Content-Disposition" - Future withSuggestedFilenameFromResponseHeaders( - Map responseHeaders, - {bool unique = false}) { - /// Returns [DownloadTask] with a filename similar to the one - /// supplied, but unused. - /// - /// If [unique], filename will sequence up in "filename (8).txt" format, - /// otherwise returns the [task] - Future uniqueFilename(DownloadTask task, bool unique) async { - if (!unique) { - return task; - } - final sequenceRegEx = RegExp(r'\((\d+)\)\.?[^.]*$'); - final extensionRegEx = RegExp(r'\.[^.]*$'); - var newTask = task; - var filePath = await newTask.filePath(); - var exists = await File(filePath).exists(); - while (exists) { - final extension = - extensionRegEx.firstMatch(newTask.filename)?.group(0) ?? ''; - final match = sequenceRegEx.firstMatch(newTask.filename); - final newSequence = int.parse(match?.group(1) ?? "0") + 1; - final newFilename = match == null - ? '${path.basenameWithoutExtension(newTask.filename)} ($newSequence)$extension' - : '${newTask.filename.substring(0, match.start - 1)} ($newSequence)$extension'; - newTask = newTask.copyWith(filename: newFilename); - filePath = await newTask.filePath(); - exists = await File(filePath).exists(); - } - return newTask; - } - - // start of main method - try { - final disposition = responseHeaders.entries - .firstWhere( - (element) => element.key.toLowerCase() == 'content-disposition') - .value; - // Try filename="filename" - final plainFilenameRegEx = - RegExp(r'filename=\s*"?([^"]+)"?.*$', caseSensitive: false); - var match = plainFilenameRegEx.firstMatch(disposition); - if (match != null && match.group(1)?.isNotEmpty == true) { - return uniqueFilename(copyWith(filename: match.group(1)), unique); - } - // Try filename*=UTF-8'language'"encodedFilename" - final encodedFilenameRegEx = RegExp( - 'filename\\*=\\s*([^\']+)\'([^\']*)\'"?([^"]+)"?', - caseSensitive: false); - match = encodedFilenameRegEx.firstMatch(disposition); - if (match != null && - match.group(1)?.isNotEmpty == true && - match.group(3)?.isNotEmpty == true) { - try { - final suggestedFilename = match.group(1)?.toUpperCase() == 'UTF-8' - ? Uri.decodeComponent(match.group(3)!) - : match.group(3)!; - return uniqueFilename(copyWith(filename: suggestedFilename), true); - } on ArgumentError { - _log.finest( - 'Could not interpret suggested filename (UTF-8 url encoded) ${match.group(3)}'); - } - } - } catch (_) {} - _log.finest('Could not determine suggested filename from server'); - // Try filename derived from last path segment of the url - try { - final suggestedFilename = Uri.parse(url).pathSegments.last; - return uniqueFilename(copyWith(filename: suggestedFilename), unique); - } catch (_) {} - _log.finest('Could not parse URL pathSegment for suggested filename'); - // if everything fails, return the task with unchanged filename - // except for possibly making it unique - return uniqueFilename(this, unique); + return taskWithFilenameBuilder(this, {}, unique); } /// Return the expected file size for this task, or -1 if unknown @@ -739,7 +661,11 @@ final class DownloadTask extends Task { return -1; } - bool get hasFilename => filename != '?'; + /// Constant used with `filename` field to indicate server suggestion requested + static const suggestedFilename = '?'; + + /// True if this task has a filename, or false if set to `suggest` + bool get hasFilename => filename != suggestedFilename; } /// Information related to an upload task @@ -1593,3 +1519,91 @@ final class Config { return value; } } + +// helper function for DownloadTask + +/// Returns a copy of the [task] with the [Task.filename] property changed +/// to the filename suggested by the server, or derived from the url, or +/// unchanged. +/// +/// If [unique] is true, the filename is guaranteed not to already exist. This +/// is accomplished by adding a suffix to the suggested filename with a number, +/// e.g. "data (2).txt" +/// +/// The server-suggested filename is obtained from the [responseHeaders] entry +/// "Content-Disposition" according to RFC6266, or the last path segment of the +/// URL, or leaves the filename unchanged +Future taskWithSuggestedFilename( + DownloadTask task, Map responseHeaders, bool unique) { + /// Returns [DownloadTask] with a filename similar to the one + /// supplied, but unused. + /// + /// If [unique], filename will sequence up in "filename (8).txt" format, + /// otherwise returns the [task] + Future uniqueFilename(DownloadTask task, bool unique) async { + if (!unique) { + return task; + } + final sequenceRegEx = RegExp(r'\((\d+)\)\.?[^.]*$'); + final extensionRegEx = RegExp(r'\.[^.]*$'); + var newTask = task; + var filePath = await newTask.filePath(); + var exists = await File(filePath).exists(); + while (exists) { + final extension = + extensionRegEx.firstMatch(newTask.filename)?.group(0) ?? ''; + final match = sequenceRegEx.firstMatch(newTask.filename); + final newSequence = int.parse(match?.group(1) ?? "0") + 1; + final newFilename = match == null + ? '${path.basenameWithoutExtension(newTask.filename)} ($newSequence)$extension' + : '${newTask.filename.substring(0, match.start - 1)} ($newSequence)$extension'; + newTask = newTask.copyWith(filename: newFilename); + filePath = await newTask.filePath(); + exists = await File(filePath).exists(); + } + return newTask; + } + + // start of main function + try { + final disposition = responseHeaders.entries + .firstWhere( + (element) => element.key.toLowerCase() == 'content-disposition') + .value; + // Try filename="filename" + final plainFilenameRegEx = + RegExp(r'filename=\s*"?([^"]+)"?.*$', caseSensitive: false); + var match = plainFilenameRegEx.firstMatch(disposition); + if (match != null && match.group(1)?.isNotEmpty == true) { + return uniqueFilename(task.copyWith(filename: match.group(1)), unique); + } + // Try filename*=UTF-8'language'"encodedFilename" + final encodedFilenameRegEx = RegExp( + 'filename\\*=\\s*([^\']+)\'([^\']*)\'"?([^"]+)"?', + caseSensitive: false); + match = encodedFilenameRegEx.firstMatch(disposition); + if (match != null && + match.group(1)?.isNotEmpty == true && + match.group(3)?.isNotEmpty == true) { + try { + final suggestedFilename = match.group(1)?.toUpperCase() == 'UTF-8' + ? Uri.decodeComponent(match.group(3)!) + : match.group(3)!; + return uniqueFilename(task.copyWith(filename: suggestedFilename), true); + } on ArgumentError { + _log.finest( + 'Could not interpret suggested filename (UTF-8 url encoded) ${match.group(3)}'); + } + } + } catch (_) {} + _log.finest('Could not determine suggested filename from server'); + // Try filename derived from last path segment of the url + try { + final suggestedFilename = Uri.parse(task.url).pathSegments.last; + return uniqueFilename(task.copyWith(filename: suggestedFilename), unique); + } catch (_) {} + _log.finest('Could not parse URL pathSegment for suggested filename'); + // if everything fails, return the task with unchanged filename + // except for possibly making it unique + return uniqueFilename(task, unique); +} diff --git a/pubspec.yaml b/pubspec.yaml index 721fa599..79dca082 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -1,7 +1,7 @@ name: background_downloader description: A multi-platform background file downloader and uploader. Define the task, enqueue and monitor progress -version: 7.10.1 +version: 7.11.0 repository: https://github.com/781flyingdutchman/background_downloader environment: