Skip to content

Releases: VIPnytt/RobotsTxtParser

Version 2.1.0

10 Apr 17:11
Compare
Choose a tag to compare

Version 2.0.2

27 Aug 15:34
2042e4d
Compare
Choose a tag to compare

Bug fixes:

  • Don't throw errors when parsing an empty robots.txt file #5

Version 2.0.1

21 Jul 15:56
Compare
Choose a tag to compare

Bug fixes

  • Rules didn't match when contained both wildcard(s) * and an end anchor $ #5
  • Rules didn't always match when contained special characters like +()[]

Other improvements

  • Normalize rules starting with an wildcard * to always have an prefixed slash /
  • Added more test cases
  • Minor code and test optimizations

Version 2.0.0

18 Feb 13:29
Compare
Choose a tag to compare

Version 2.0.0 stable is finally here, with lots of new abilities, features and improvements.

The essentials:

  • All protocols now supported library-wide
  • robots.txt rendering
  • Import/Export rule sets as arrays
  • Cache robots.txt files to MySQL (optional)
  • Delay handling with precision using MySQL (optional)
  • New directives:
    • Comment
    • NoIndex
    • Request-rate
    • Robot-version
    • Visit-time
  • Some new abilities at directive level
  • Lots of improvements under the hood

Version 1.x incompatible!

Version 2.0.0 rc 2

23 Aug 18:39
Compare
Choose a tag to compare
Version 2.0.0 rc 2 Pre-release
Pre-release
  • Bug fix: DelayManager->debug() didn't return all entries
  • Bug-fix: Wrong user-agent sometimes passed to the MySQL Delay handler
  • No longer force checks the Comment directive for comments
  • Refactored (internal) error handler
  • Minor code optimizions
  • More tests (code coverage at 97,3% 🎉 )

Version 2.0.0 rc 1

10 Aug 19:34
Compare
Choose a tag to compare
Version 2.0.0 rc 1 Pre-release
Pre-release
  • Renamed $delayClient->getQueue() to checkQueue()
  • Import class, for importing exported arrays
  • Updated default byte-limit from 64 to 512 kilobytes
  • Performance optimizions
  • Fixed an bug causing the user-agent string parser to generate warnings
  • Minor refactoring, with performance in mind
  • Refactored documentation

Version 2.0.0 beta 2

02 Aug 22:53
Compare
Choose a tag to compare
Version 2.0.0 beta 2 Pre-release
Pre-release

Version 1.x incompatible.

Changelog since version 2.0-beta.1:

  • PHP 7.1 compatible
  • Added RenderClient class
  • Minor speed improvements
  • Updated vipnytt/useragentparser to v1.0
  • Make sure some cURL options is not overwritten.
  • The confusing isHostListed() func is removed.
  • Refactored UriParser
  • Refactored Clean-param directive classes
  • Refactored Host directive classes
  • HostClient getWithFallback renamed to getWithUriFallback
  • SQL debug methods
  • Documentation

Version 2.0.0 beta 1

13 Jul 04:02
Compare
Choose a tag to compare
Version 2.0.0 beta 1 Pre-release
Pre-release

Version 1.x incompatible.

Changelog since version 2.0-alpha:

  • Support for the NoIndex directive
  • cURL now replaces GuzzleHttp
  • Cache and UriClient classes now supports all available protocols
  • No longer lists empty user-agents.
  • Directive aliases (for simple typos / errors).
  • Retry after HTTP header support.
  • Improved Status-code handler, no longer throws exceptions on strange codes.
  • Improved multi-User-agent directive handler.
  • Host directive now also supports URL redirect host as fallback
  • Tons of code optimizions
  • Refactoring
  • Documentation

Version 2.0.0 alpha

27 Jun 15:13
Compare
Choose a tag to compare
Version 2.0.0 alpha Pre-release
Pre-release

Alpha release, use at your own risk!
Version 1.x incompatible.

Short version change-log (since version 1.0.1):

  • More directives and specifications supported.
  • Increased specification coverage (including bug-fixes and new handlers).
  • Built-in caching system.
  • Built-in delay handling system.
  • Restructured for better maintainability and easier usage.
  • More documentation.

See README for more information.

Version 1.0.1

24 Apr 16:50
Compare
Choose a tag to compare
  • More accurate end anchor $ rule matching.