Skip to content

v1.40.6
Compare
Choose a tag to compare
@j-mendez j-mendez released this 11 Sep 00:56
· 611 commits to main since this release
v1.40.6
6bbbd82

Whats Changed

  • feat(chrome): enable chrome rendering page content [experimental]
  • chore(crawl): remove crawl sync method for Sequential crawls

If you need crawls to be sequential use configuration.delay or use website.with_delay(1), set any value greater than 0.

Headless

Use the feature flag chrome for headless and chrome_headed for headful crawling.

Chrome installations are detected automatically on the OS. The current implementation uses chromiumaxide and handles html as raw strings so downloading media will not be ideal since the bytes may be invalid. The chrome feature does not work with the decentralized flag at the moment.

Video below shows 200 plus pages being handled within a couple seconds, headless runs drastically faster.
Try to only use headed for debugging.

Screen.Recording.2023-09-10.at.8.41.44.PM.mov

Full Changelog: v1.37.7...v1.40.6

Assets 2
Loading