Skip to content

Commit

Permalink
chore(bench): fix criterion table count
Browse files Browse the repository at this point in the history
  • Loading branch information
j-mendez committed Dec 6, 2023
1 parent f776841 commit fb6f864
Show file tree
Hide file tree
Showing 6 changed files with 38 additions and 38 deletions.
32 changes: 16 additions & 16 deletions bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,18 @@ Linux
Test url: `https://choosealicense.com` (small)
32 pages

| `libraries` | `speed` |
| :-------------------------------- | :------------------- |
| **`spider-rs: crawl 10 samples`** | `76ms`(✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `1s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `76ms` |
| **`crawlee: crawl 10 samples`** | `1s` |

Test url: `https://rsseau.fr` (medium)
211 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `0.5s` (✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `72s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `0.5s` |
| **`crawlee: crawl 10 samples`** | `72s` |

```sh
----------------------
Expand All @@ -34,17 +34,17 @@ mac Apple M1 Max
Test url: `https://choosealicense.com` (small)
32 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `286ms`(✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `1.7s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `286ms` |
| **`crawlee: crawl 10 samples`** | `1.7s` |

Test url: `https://rsseau.fr` (medium)
211 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `2.5s` (✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `75s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `2.5s` |
| **`crawlee: crawl 10 samples`** | `75s` |

The performance scales the larger the website and if throttling is needed. Linux benchmarks are about 10x faster than macOS for spider-rs.
4 changes: 2 additions & 2 deletions bench/base.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ export const TEST_URL_MEDIUM = "https://rsseau.fr";
export enum BenchSizes {
SMALL = "SMALL",
MEDIUM = "MEDIUM",
LARGE = "LARGE"
}
LARGE = "LARGE",
}
2 changes: 1 addition & 1 deletion bench/compare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ import { bench } from "./case/spider";
// small
bench();
// small/medium
bench(TEST_URL_MEDIUM, BenchSizes.MEDIUM)
bench(TEST_URL_MEDIUM, BenchSizes.MEDIUM);
2 changes: 1 addition & 1 deletion bench/crawlee.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ import { bench } from "./case/crawlee";
// small
bench();
// small/medium
bench(TEST_URL_MEDIUM, BenchSizes.MEDIUM)
bench(TEST_URL_MEDIUM, BenchSizes.MEDIUM);
34 changes: 17 additions & 17 deletions book/src/benchmarks.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,18 @@ Linux
Test url: `https://choosealicense.com` (small)
32 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `76ms`(✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `1s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `76ms` |
| **`crawlee: crawl 10 samples`** | `1s` |

Test url: `https://rsseau.fr` (medium)
211 pages

| `libraries` | `speed` |
| :-------------------------------- | :------------------- |
| **`spider-rs: crawl 10 samples`** | `0.5s` (✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `72s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `0.5s` |
| **`crawlee: crawl 10 samples`** | `72s` |

```sh
----------------------
Expand All @@ -36,17 +36,17 @@ mac Apple M1 Max
Test url: `https://choosealicense.com` (small)
32 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `286ms`(✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `1.7s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `286ms` |
| **`crawlee: crawl 10 samples`** | `1.7s` |

Test url: `https://rsseau.fr` (medium)
211 pages

| `libraries` | `speed` |
| :-------------------------------- | :-------------------- |
| **`spider-rs: crawl 10 samples`** | `2.5s` (✅ **1.00x**) |
| **`crawlee: crawl 10 samples`** | `75s` (✅ **1.00x**) |
| `libraries` | `speed` |
| :-------------------------------- | :------ |
| **`spider-rs: crawl 10 samples`** | `2.5s` |
| **`crawlee: crawl 10 samples`** | `75s` |

The performance scales the larger the website and if throttling is needed. Linux benchmarks are about 10x faster than macOS for spider-rs.
The performance scales the larger the website and if throttling is needed. Linux benchmarks are about 10x faster than macOS for spider-rs.
2 changes: 1 addition & 1 deletion book/src/website.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ To stop a crawl you can use `website.stopCrawl(id)`, pass in the crawl id to sto
const website = new Website("https://choosealicense.com");

const onPageEvent = (_err, page) => {
console.log(page)
console.log(page);
// stop the concurrent crawl when 8 pages are found.
if (website.size >= 8) {
website.stop();
Expand Down

0 comments on commit fb6f864

Please sign in to comment.