A node.js powered scrapper 🔥 that iterates trough all the internal links of the specified url.
It works on CSR pages (React, Angular) with dynamic urls.
Once it is done it generates a sitemap.xml file with all the urls found, ready to be uploaded to Google Search Console.
$ sitemap https://vvlog.dev| Parameter | type | default | description |
|---|---|---|---|
| --wait | integer | 1500 | Specify the time (milliseconds) to wait (So the fetches are completed) before starting to parse the page. |
| --limit | integer | 999999 | Specify the limit of urls to parse before stopping the scrapper. |
- Make it a NPM package.
- Make wait time dynamic in response of fetches inside url.
- New params that lets you specify how deep you want to go inside the url.
- Integrate it as part of build process of a create-react-app.
- Clean old code.