Skip to content

Commit

Permalink
chore(docs): add book start
Browse files Browse the repository at this point in the history
  • Loading branch information
j-mendez committed Nov 28, 2023
1 parent 8194012 commit f8e6a61
Show file tree
Hide file tree
Showing 7 changed files with 115 additions and 0 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ env:
DEBUG: napi:*
APP_NAME: spider-rs
MACOSX_DEPLOYMENT_TARGET: '10.13'
concurrency:
group: ${{ github.ref }}
cancel-in-progress: true
permissions:
contents: write
id-token: write
Expand Down
29 changes: 29 additions & 0 deletions .github/workflows/book.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
name: github pages

on:
push:
branches:
- main
pull_request:

jobs:
deploy:
runs-on: ubuntu-20.04
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
steps:
- uses: actions/checkout@v4

- name: Setup mdBook
uses: peaceiris/actions-mdbook@v1
with:
mdbook-version: 'latest'

- run: cd book && mdbook build

- name: Deploy
uses: peaceiris/actions-gh-pages@v3
if: ${{ github.ref == 'refs/heads/main' }}
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./book/book
1 change: 1 addition & 0 deletions book/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
book
6 changes: 6 additions & 0 deletions book/book.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[book]
authors = ["j-mendez"]
language = "en"
multilingual = false
src = "src"
title = "spider-rs"
17 changes: 17 additions & 0 deletions book/src/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Introduction

Spider-RS is the fastest web crawler and indexer written in Rust.

Spider powers some big tools and helps bring the crawling aspect to almost no downtime with the correct setup, view the [spider](https://github.com/spider-rs/spider) project to learn more.

Node.js powerhouse crawling in a few lines!

```ts
import { Website } from "@spider-rs/spider-rs";

const website = new Website("https://rsseau.fr");

await website.crawl();

console.log(website.getLinks());
```
5 changes: 5 additions & 0 deletions book/src/SUMMARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Summary

[Introduction](./README.md)

- [Node.js](./nodejs.md)
54 changes: 54 additions & 0 deletions book/src/nodejs.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Node.js

We use the node-addon to port the Rust project over with napi to target node.js.

There are some performance drawbacks from the addon, even still the crawls are lightning fast and performant.

## Installation

1. `npm i @spider-rs/spider-rs --save`

## Usage

The examples below can help get started with spider.

### Basic

A basic example.

```ts
import { Website } from "@spider-rs/spider-rs";

const website = new Website("https://rsseau.fr")
.withHeaders({
authorization: "somerandomjwt",
})
.withBudget({
// limit up to 200 pages crawled for the entire website
"*": 200,
})
.withBlacklistUrl([new RegExp("/books").source, "/resume"])
.build();

await website.crawl();
console.log(website.getLinks());
```

### Events

You can pass a function that could be async as param to `crawl` and `scrape`.

```ts
import { Website, type NPage } from "@spider-rs/spider-rs";

const website = new Website("https://rsseau.fr");

const links: NPage[] = [];

const onPageEvent = (err: Error | null, value: NPage) => {
links.push(value);
};

await website.crawl(onPageEvent);
console.log(website.getLinks());
```

0 comments on commit f8e6a61

Please sign in to comment.