Skip to content

Latest commit

 

History

History
245 lines (184 loc) · 10.1 KB

README.md

File metadata and controls

245 lines (184 loc) · 10.1 KB
Raw

Rehype URL Inspector

A rehype plugin to inspect, validate, or rewrite URLs anywhere in an HTML document

Cross-Platform Compatibility Build Status

Coverage Status Dependencies

npm License Buy us a tree

Features

  • Inspect every URL on an HTML page and do whatever you want to, such as:

    • Normalize URLs
    • Check for broken links
    • Replace URLs with different URLs
    • Add attributes (like target="blank") to certain links

  • Finds all types of URLs by default, such as:

    • <a href="http://example.com">
    • <img src="img/logo.png">
    • <link rel="stylesheet" href="/css/main.css">
    • <link rel="manifest" href="/site.manifest">
    • <meta rel="canonical" content="https://example.com/some/page/">
    • <meta property="og:image" content="img/logo.png">
    • <script src="//example.com/script.js">
    • <script type="application/ld+json">{"url": "www.example.com"}</script>
    • <style>body { background: url("/img/background.png"); }</style>

  • You can remove the built-in URL rules

  • You can add your own custom URL rules

  • You can abort the URL search at any time

Example

example.html
This HTML file contains many different types of URLs:

<html>
  <head>
    <link rel="canonical" href="http://example.com/some/page/">
    <link rel="manifest" href="/site.webmanifest">
    <link rel="icon" type="image/png" sizes="32x32" href="/img/favicon.png">
    <link rel="stylesheet" type="text/css" href="/css/main.css?v=5">

    <meta name="twitter:url" content="http://example.com/some/page/">
    <meta name="twitter:image" content="http://example.com/img/logo.png">

    <script type="application/ld+json">
      {
        "@context": "http://schema.org",
        "headline": "Hello, World!",
        "url": "http://example.com/some/page/",
        "image": "http://example.com/img/logo.png"
      }
    </script>

    <style>
      body {
        background: #ffffff url("img/background.png") center center no-repeat;
      }
    </style>
  </head>
  <body>
    <h1>
      <a href="/">
        <img src="/img/logo.png"> Hello World
      </a>
    </h1>
    <p>
      <a href="//external.com" target="_blank">Lorem ipsum</a> dolor sit amet,
      non dignissim dolor. Sed diam tellus, <a href="some-page.html">malesuada, dictum nulla</a>.
    </p>

    <script src="//external.com/script.js"></script>
  </body>
</html>

example.js
This script reads the example.html file above and finds all the URLs in it. The script uses unified, rehype-parse, rehype-stringify, and to-vfile.

const unified = require("unified");
const parse = require("rehype-parse");
const inspectUrls = require("@jsdevtools/rehype-url-inspector");
const stringify = require("rehype-stringify");
const toVFile = require("to-vfile");

async function example() {
  // Create a Rehype processor with the inspectUrls plugin
  const processor = unified()
    .use(parse)
    .use(inspectUrls, {
      inspectEach({ url }) {
        // Log each URL
        console.log(url);
      }
    })
    .use(stringify);

  // Read the example HTML file
  let file = await toVFile.read("example.html");

  // Crawl the HTML file and find all the URLs
  await processor.process(file);
}

example();

Running this script produces the following output:

http://example.com/some/page/
/site.webmanifest
/img/favicon.png
/css/main.css?v=5
http://example.com/some/page/
http://example.com/img/logo.png
http://schema.org
http://example.com/some/page/
http://example.com/img/logo.png
img/background.png
/
/img/logo.png
//external.com
some-page.html
//external.com/script.js

Installation

You can install Rehype URL Inspector via npm.

npm install @jsdevtools/rehype-url-inspector

You'll probably want to install unified, rehype-parse, rehype-stringify, and to-vfile as well.

npm install unified rehype-parse rehype-stringify to-vfile

Usage

Using the URL Inspector plugin requires an understanding of how to use Unified and Rehype. Here is an excelleng guide to learn the basics.

The URL Inspector plugin works just like any other Rehype plugin. Pass it to the .use() method with an options object.

const unified = require("unified");
const inspectUrls = require("@jsdevtools/rehype-url-inspector");

// Use the Rehype URL Inspector plugin with custom options
unified().use(inspectUrls, {
  inspect(urls) { ... },      // This function is called once with ALL of the URLs
  inspectEach(url) { ... },   // This function is called for each URL as it's found
  selectors: [
    "a[href]",                // Only search for links, not other types of URLs
    "div[data-image]"         // CSS selectors for custom URL attributes
  ]
});

Options

Rehype URL Inspector supports the following options:

Option Type Default Description
selectors array of strings, objects, and/or functions built-in selectors Selectors indicate where to look for URLs in the document. Each selector can be a CSS attribute selector string, like a[href] or img[src], or a function that accepts a HAST node and returns its URL(s). See extractors.ts for examples.
keepDefaultSelectors boolean false Whether to keep the default selectors in addition to any custom ones.
inspect function no-op A function that is called once and receives an array containing all the URLs in the document
inspectEach function no-op A function that is called for each URL in the document as it's found. Return false to abort the search and skip the rest of the document.

URL Objects

The inspectEach() function receives a UrlMatch object. The inspect() function receves an array of UrlMatch objects. Each object has the following properties:

Property Type Description
url string The URL that was found
propertyName string or undefined The name of the HAST node property where the URL was found, such as "src" or "href". If the URL was found in the text content of the node, then propertyName is undefined.
node object The HAST Element node where the URL was found. You can make changes to this node, such as re-writing the URL, adding additional attributes, etc.
root object The HAST Root node. This gives you access to the whole document if you need it.
file object The File object that gives you information about the HTML file itself, such as the path and file name.

Contributing

Contributions, enhancements, and bug-fixes are welcome! Open an issue on GitHub and submit a pull request.

Building

To build the project locally on your computer:

  1. Clone this repo
    git clone https://github.com/JS-DevTools/rehype-url-inspector.git

  2. Install dependencies
    npm install

  3. Build the code
    npm run build

  4. Run the tests
    npm test

License

Rehype URL Inspector is 100% free and open-source, under the MIT license. Use it however you want.

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Big Thanks To

Thanks to these awesome companies for their support of Open Source developers ❤

Travis CI SauceLabs Coveralls