Skip to content

Commit

Permalink
Refactor “From the ground up”, create new pages off it (scrapinghub#152)
Browse files Browse the repository at this point in the history
  • Loading branch information
Gallaecio authored Mar 31, 2023
1 parent 0953a17 commit 62dbf84
Show file tree
Hide file tree
Showing 23 changed files with 541 additions and 669 deletions.
4 changes: 4 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,7 @@ per-file-ignores =

# the suggestion makes the code worse
tests/test_serialization.py:B028

# Documentation examples may miss docstrings and include undefined, example
# names.
docs/*:D102,F821
7 changes: 3 additions & 4 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,9 @@ Other changes:
0.7.1 (2023-02-01)
------------------

* Updated the :ref:`tutorial <intro-tutorial>` to cover recent features and
focus on best practices. Also, a new module was added,
:mod:`web_poet.example`, that allows using page objects while following the
tutorial.
* Updated the :ref:`tutorial <tutorial>` to cover recent features and focus on
best practices. Also, a new module was added, :mod:`web_poet.example`, that
allows using page objects while following the tutorial.

* :ref:`web-poet-testing` now covers :ref:`Git LFS <git-lfs>` and
:ref:`scrapy-poet <web-poet-testing-scrapy-poet>`, and recommends
Expand Down
22 changes: 20 additions & 2 deletions docs/api-reference.rst
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
.. _api-reference:

=============
API Reference
API reference
=============

.. _input:

Page Inputs
===========

.. module:: web_poet.page_inputs

.. automodule:: web_poet.page_inputs.browser
:members:
:undoc-members:
Expand Down Expand Up @@ -36,6 +38,7 @@ Pages
.. automodule:: web_poet.pages

.. autoclass:: Injectable
:show-inheritance:
:members:
:no-special-members:

Expand Down Expand Up @@ -103,8 +106,23 @@ Fields
.. automodule:: web_poet.fields
:members:

utils

Utils
=====

.. automodule:: web_poet.utils
:members:


Example framework
=================

The :mod:`web_poet.example` module is a simplified, incomplete example of a
web-poet framework, written as support material for the :ref:`tutorial
<tutorial>`.

No part of the :mod:`web_poet.example` module is intended for production use,
and it may change in a backward-incompatible way at any point in the future.

.. automodule:: web_poet.example
:members:
3 changes: 1 addition & 2 deletions docs/frameworks/additional-requests.rst
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
.. _frameworks:
.. _framework-additional-requests:

==============================
Supporting Additional Requests
Supporting additional requests
==============================

To support :ref:`additional requests <additional-requests>`, your framework must
Expand Down
66 changes: 66 additions & 0 deletions docs/frameworks/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
.. _spec:

=======================
Framework specification
=======================

Learn how to build a :ref:`web-poet framework <frameworks>`.

Design principles
=================

:ref:`Page objects <page-objects>` should be flexible enough to be used with:

* synchronous or asynchronous code, callback-based and ``async def / await``
based,
* single-node and distributed systems,
* different underlying HTTP implementations - or without HTTP support
at all, etc.


Minimum requirements
====================

A web-poet framework must support building a :ref:`page object <page-objects>`
given a page object class.

It must be able to build :ref:`input objects <inputs>` for a page object based
on type hints on the page object class, i.e. dependency injection, and
additional input data required by those input objects, such as a target URL or
a dictionary of :ref:`page parameters <page-params>`.

You can implement dependency injection with the andi_ library, which handles
signature inspection, :data:`~typing.Optional` and :data:`~typing.Union`
annotations, as well as indirect dependencies. For practical examples, see the
source code of scrapy-poet_ and of the :mod:`web_poet.example` module.

.. _andi: https://github.com/scrapinghub/andi
.. _scrapy-poet: https://github.com/scrapinghub/scrapy-poet


Additional features
===================

To provide a better experience to your users, consider extending your web-poet
framework further to:

- Support as many input classes from the :mod:`web_poet.page_inputs`
module as possible.

- Support returning a :ref:`page object <page-objects>` given a target URL
and a desired :ref:`output item type <items>`, using
``web_poet.default_registry`` to determine the right :ref:`page
object class <page-object-classes>` to use.

- Allow users to request an :ref:`output item <items>` directly, instead of
requesting a page object just to call its ``to_item`` method.

If you do, consider supporting both synchronous and asynchronous
definitions of the ``to_item`` method, e.g. using
:func:`~.ensure_awaitable`.

- Support :ref:`additional requests <framework-additional-requests>`.

- Support :ref:`retries <framework-retries>`.

- Let users set their own :class:`~web_poet.rules.RulesRegistry` object.
19 changes: 12 additions & 7 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ web-poet

.. toctree::
:caption: Getting started
:hidden:
:maxdepth: 1

intro/overview
intro/install
Expand All @@ -20,25 +20,30 @@ web-poet

.. toctree::
:caption: Writing page objects
:hidden:
:maxdepth: 1

page-objects/additional-requests
page-objects/fields
page-objects/index
page-objects/inputs
page-objects/items
page-objects/rules
page-objects/fields
page-objects/additional-requests
page-objects/input-validation
page-objects/page-params
page-objects/testing
page-objects/frameworks

.. toctree::
:caption: Writing frameworks
:hidden:
:maxdepth: 1

Additional Requests <frameworks/additional-requests>
frameworks/index
Additional requests <frameworks/additional-requests>
Retries <frameworks/retries>

.. toctree::
:caption: Reference
:hidden:
:maxdepth: 1

api-reference
contributing
Expand Down
Loading

0 comments on commit 62dbf84

Please sign in to comment.