Github rate limits requests

[jaraco]

Github will return a 403 with a header indicating that a request was rate limited. The migration script can uses this information to more robustly respond under these conditions.

[jaraco]

I have some commits in this tree. I'll plan to rebase those after #77 lands.

[jeffwidman]

Yeah, we should definitely be retrying if there's a clear rate-limit header. Also printing a status update to the user so they don't think the script is just hanging. PR welcome.

[jtara]

I get completed 3 of 112 issues, and then it just seems to stall. Is that the symptom?

[jeffwidman]

What are the headers that Github returns when you try to submit the stalled issue?

[jtara]

I'm not sure where to look, sorry. Is there some trace option I can enable?

It shows some JSON on the console, and then just stalls with:

Completed 3 of 112 issues

Here's the last issue it showed on the console. I don't think this gives you anything useful, though:

Issue:  {'title': 'Sketch toolbar', 'body': 'Originally reported by: **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**\n\n----------------------------------------\n\nImplement sketch toolbar with stubbed controller methods or button event handlers.\n\n----------------------------------------\n- Bitbucket: https://bitbucket.org/spatvis/spatvis/issue/3\n', 'closed': True, 'created_at': '2017-05-05T17:55:23Z', 'labels': ['major', 'task']}

Comments:  [{'created_at': '2017-05-18T20:02:34Z', 'body': "*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nMerge remote-tracking branch 'bitbucket/4' closes #3 closes #4\n\n"}, {'created_at': '2017-05-18T16:57:57Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 ditch the toolbar box-shadow for now\n\nToo much trouble to get this to work right.\n\n- need margin for shadow to show\n- z-index issue making it impossible to draw. The obvious solution doesn’t seem to work\n\n'}, {'created_at': '2017-05-17T23:23:52Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 highlight/persist selected erase width on dropdown menu\n\ntodo: also highlight upon selection for consistency\n\n'}, {'created_at': '2017-05-17T19:41:03Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 highlight/persist selected erase width on dropdown menu\n\n'}, {'created_at': '2017-05-17T06:21:46Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 sketch undo/redo flash-active\n\n'}, {'created_at': '2017-05-17T03:40:39Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 drop-down styling, clear all in red\n\n'}, {'created_at': '2017-05-17T03:27:27Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 ref #4 implemented eraser functionality small/medium/large\n\n'}, {'created_at': '2017-05-17T03:27:27Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 Added eraser menu\n\n'}, {'created_at': '2017-05-17T01:14:40Z', 'body': '*Original comment by* **Jonathan Tara (Bitbucket: [watusi](https://bitbucket.org/watusi), GitHub: [watusi](https://github.com/watusi))**:\n\n----------------------------------------\n\nref #3 box shadow on toolbar\n\n'}]

I am going to try again now with MacOS Network Link Conditioner set to 200kbps, 400mSec latency as a crude work-around.

[jtara]

With the Network Link Conditioner enabled, it is progressing. It is very slow in any case, is that normal? But, hey, it made it to 13! ;) The speed seems variable, sometimes it will do a couple in quick succession.

I think I will now chance it without dry run mode...

[jtara]

It copied 67 of 112 issues before dying on a ConnectionResetError [Errno 54].

I will restart with a skip.

This would seem a common issue that should be handled as well.

[magreenblatt]

I'm running a migration of ~4000 issues currently. On average each issue takes ~4s to complete. I have not experienced any rate limit or connection issues so far (> 2000 issues migrated).

[jeffwidman]

Maybe GitHub's infra is a bit more robust in the last 5 years... 😁

I'll leave this open for a bit in case someone else sees the 403, as we should ideally handle it properly with a delayed retry.