Toimik

All

7 repositories

CommonCrawl
Public
Common Crawl's processing tools
warc wat wet commoncrawl common-crawl warc-files wat-files common-crawl-data wet-files
C#
•
Apache License 2.0
•0•9•0•0•Updated Oct 15, 2024Oct 15, 2024
SitemapsProtocol
Public
Parsers for sitemap / sitemap index (aka Sitemaps Protocol)
sitemap xml-sitemap sitemap-xml sitemaps site-map sitemapindex sitemap-index xml-sitemaps
C#
•
Apache License 2.0
•0•0•0•0•Updated Jul 15, 2024Jul 15, 2024
WarcProtocol
Public
Parser for WARC (aka WebArchive) files
warc webarchive webarchiving warc-files webarchives warc-format warc-reader warc-record
C#
•
Apache License 2.0
•3•11•0•0•Updated Jul 9, 2024Jul 9, 2024
Wikimedia
Public
Wikimedia Downloads' processing tools
wikipedia wikimedia wikimedia-tools
C#
•
Apache License 2.0
•0•0•0•0•Updated May 2, 2024May 2, 2024
UrlNormalization
Public
URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical
url-normalization url-normalizer url-canonicalization url-standardizer url-standardization url-comparer
C#
•
Apache License 2.0
•0•4•0•0•Updated May 2, 2024May 2, 2024
RobotsProtocol
Public
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag
C#
•
Apache License 2.0
•0•0•0•0•Updated May 2, 2024May 2, 2024
IpAddressEnumeration
Public
IP address enumerators
ip-address ipv4-address ip-addresses ip-address-tool ipv4-addresses
C#
•
Apache License 2.0
•0•0•0•0•Updated May 2, 2024May 2, 2024