v1.46.4

Whats Changed

Crawling all domains found on website now possible with * in external_domains.

1. feat(external): add wildcard handling all domains found #135

Example:

let mut website = Website::new("https://choosealicense.com");

website
    .with_external_domains(Some(Vec::from(["*"].map(|d| d.to_string())).into_iter()));

Use the crawl budget and blacklist features to help prevent infinite crawls:

website
    .with_blacklist_url(Some(Vec::from(["^/blog/".into()])))
    .with_budget(Some(spider::hashbrown::HashMap::from([("*", 300), ("/licenses", 10)])));

Thank you @sebs for the help!

Full Changelog: v1.45.10...v1.46.0