-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Godot docs to include v4.2 and fix older version scraping #2201
Conversation
Godot 4.2 was released on Nov 2023. Since v3.5, godot upstream docs have used some new HTML layouts and file structures. Since older versions still need to be scrapable, this introduces another set of versioned filters for godot. This change also makes small changes to the filters to handle the current website markup for the previous godot doc versions. The @GDscript and @globalscope entries can't currently be browsed, because of an encoding mismatch between the frontend and backend; I've identified a possible fix for that but will PR that separately.
Hi. Thanks for doing this! I just tried your changes and getting a 403 when trying to download Godot 4.2 docs.
I tried visiting https://docs.godotengine.org/en/4.2 on a browser and it doesn't look like I've been blacklisted or anything. Regards |
@gordon-boom I haven't done anything around uploading a packaged version to devdocs yet, so I wouldn't expect that to work at this point. If there's a step I need to do as part of this PR to get that working, let me know what it is and I'd be happy to do it! The command I've been using thus far for testing is: |
Ah silly me. I guess I didn't read the difference between download and generate. I will try that next time. |
Just tried it out. Looks good! Let's hope it gets merged. |
This works for me. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
This adds docs for Godot v4.2, and does some fixups for scraping older godot releases from the upstream website.
Godot 4.2 was released on Nov 2023: https://godotengine.org/article/godot-4-2-arrives-in-style/
Since v3.5, godot upstream docs have used some new HTML layouts and file structures. Since older versions still need to be scrapable, this introduces another set of versioned filters for godot.
This change also makes small changes to the filters to handle the current website markup for the previous godot doc versions.
The @GDscript and @globalscope entries can't currently be browsed, because of an encoding mismatch between the frontend and backend; I've identified a possible fix in #2200 .
checklist
SOURCE
file inpublic/icons/your_scraper_name/
are up-to-date if the documentation has a custom iconself.links
contains up-to-date urls ifself.links
is definedQA notes
I generated all of the godot doc versions that the scraper supports, and diffed them against the existing packaged versions to ensure as close of a match as possible.
Command to generate a local copy of the godot 4.2 docs:
bundle exec thor docs:generate [email protected]
Spot checks show that the main differences are:
class_@gdscript
andclass_@globalscope
filenames are escaped with%40
instead of@
. I have no idea how the previous packager got those files to be@
symbols, I suspect the godot website used to encode those differently.