Prevent duplicates by using URL as uniquie identifier

[schliflo]

Entries that get updated after being crawled for the first time sometimes generate duplicates upon re-crawl. This could be prevented by using the URL as an unique identifier and falling back to updating the existing entry for any given URL on re-crawl.