Add support for retrying failed connections, including adding a new --tries option to change the default
#6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…with Ruby 3.x, plus added some CGI.escape() to ensure that the requested URL's query string parameters are handled correctly and not accidentally mixed with the WayBack Machine's query string parameters
…p/wait between requests, plus a '--wait-random' option which will randomize the number of wait seconds by a 0.5x-2x. These options are used by the new WaybackMachineDownloader#wait method which is called during subsequent requests. Issue cocoflan#1
… including a new '--tries' option accepting a number of times to retry if a connection fails for a fatal error (not just an HTTP 4XX/5XX error; the default is 20 retries'). Issue cocoflan#1
…r#download_file as the URI should only be escaped when querying the WaybackMachine API, not when requesting files. Issue cocoflan#3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
IMPORTANT: Both PRs #4 & #5 are prerequisites for this PR and should be merged first.
This implements automatic retrying upon a network connection error (not an HTTP 4XX/5XX error), up to 20 times, by default. It also adds a new
--triesoption which accepts a number of times to try (including the first) to override the default of 20 retries.This further improves the reliability of the tool, plus helps if there have been any errors due to rate-limiting (see Issue #1).
Example usage:
./bin/wayback_machine_downloader --to 20120222134837 --tries 10 http://www.folklore.org/