Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: improve timing of checks depending on changes since last check #163

Open
wants to merge 85 commits into
base: main
Choose a base branch
from
Open

feat: improve timing of checks depending on changes since last check #163

wants to merge 85 commits into from

Commits on Sep 11, 2024

  1. docs: update changelog

    @bolinocroustibat
    bolinocroustibat committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    739ef30 View commit details
    Browse the repository at this point in the history

  2. feat: select outdated checks with increasing time

    @bolinocroustibat
    bolinocroustibat committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    87d2c52 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    ed5a584 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. fix: fix wrong config var format

    @bolinocroustibat
    bolinocroustibat committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    68f5556 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. tests: fix test

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    d72ffac View commit details
    Browse the repository at this point in the history

  2. docs: fix docstring

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    76ab578 View commit details
    Browse the repository at this point in the history

  3. docs: add comments

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    c72a100 View commit details
    Browse the repository at this point in the history

  4. clean: remove useless imports

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b8c8cac View commit details
    Browse the repository at this point in the history

  5. fix: update the count of outdated checks in get_crawler_status

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    7d07a2b View commit details
    Browse the repository at this point in the history

  6. docs: update docstrings

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    825e793 View commit details
    Browse the repository at this point in the history

  7. refactor: rename some vars

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    463f229 View commit details
    Browse the repository at this point in the history

  8. docs: update changelog

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ab00ea7 View commit details
    Browse the repository at this point in the history

  9. docs: update docstrings

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    0e47263 View commit details
    Browse the repository at this point in the history

  10. tests: add tests (wip)

    @bolinocroustibat
    bolinocroustibat committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1825df6 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. tests: add tests for re-check before/after default delay

    @bolinocroustibat
    bolinocroustibat committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    941e894 View commit details
    Browse the repository at this point in the history

  2. tests: better names and comments

    @bolinocroustibat
    bolinocroustibat committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    2e0fb48 View commit details
    Browse the repository at this point in the history

  3. tests: refactor latest tests as parametrized (test_re_check_depending… 

    …_on_default_delay and test_re_check_depending_on_variable_delays)
    @bolinocroustibat
    bolinocroustibat committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f07bcee View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. feat: manage large resources exceptions differently (#148) 

    Closes #114.
    @bolinocroustibat
    bolinocroustibat committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    5f0c84c View commit details
    Browse the repository at this point in the history

  2. WIP

    @bolinocroustibat
    bolinocroustibat committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    e33f090 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
#	udata_hydra/config_default.toml
    @bolinocroustibat
    bolinocroustibat committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    6adec5c View commit details
    Browse the repository at this point in the history

  4. fix: fix increasing check delay logic

    @bolinocroustibat
    bolinocroustibat committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    c25281c View commit details
    Browse the repository at this point in the history

  5. tests: fix tests

    @bolinocroustibat
    bolinocroustibat committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    35ecc23 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    3491421 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. style: lint code

    @bolinocroustibat
    bolinocroustibat committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    f5eef08 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    852663a View commit details
    Browse the repository at this point in the history

  3. feat: use CHECK_DELAY_DEFAULT

    @bolinocroustibat
    bolinocroustibat committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    6e6ac05 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    431aa7f View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    571a12d View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    ff1fc70 View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    3daf61b View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    524e57c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Oct 2, 2024
    Configuration menu
    Copy the full SHA
    70a2640 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    3e2e6cd View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    a8dc898 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. docs: update changelog

    @bolinocroustibat
    bolinocroustibat committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    9a10b7c View commit details
    Browse the repository at this point in the history

  2. docs: update comment

    @bolinocroustibat
    bolinocroustibat committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    fe9e68c View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    b9e4099 View commit details
    Browse the repository at this point in the history

  2. WIP

    @bolinocroustibat
    bolinocroustibat committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    2f59451 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    6d99e23 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    461cf9a View commit details
    Browse the repository at this point in the history

  2. feat: add migratino to add a next_check column to checks

    @bolinocroustibat
    bolinocroustibat committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    8c6372d View commit details
    Browse the repository at this point in the history

  3. feat: new logic for select_batch

    @bolinocroustibat
    bolinocroustibat committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    5aa528b View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    8f3e121 View commit details
    Browse the repository at this point in the history

  2. clean: remove useless imports

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    96256a8 View commit details
    Browse the repository at this point in the history

  3. docs: add TODO

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    0d4275a View commit details
    Browse the repository at this point in the history

  4. feat: add logic to calculate next_check datetime

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    9661aa5 View commit details
    Browse the repository at this point in the history

  5. fix: better next_check behaviour

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    79a0421 View commit details
    Browse the repository at this point in the history

  6. tests: revert test

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    a7ae5dd View commit details
    Browse the repository at this point in the history

  7. docs: add comment

    @bolinocroustibat
    bolinocroustibat committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    04d324a View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2024

  1. fix: fix wrong var in code

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    9de1c48 View commit details
    Browse the repository at this point in the history

  2. fix: fix type hint

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    23d5ec4 View commit details
    Browse the repository at this point in the history

  3. refactor: refactor crawler status logic

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    80645f5 View commit details
    Browse the repository at this point in the history

  4. refactor: slight refactor of next_check logic

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    ae8ecaa View commit details
    Browse the repository at this point in the history

  5. clean: remove useless imports

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    e809a52 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    3039850 View commit details
    Browse the repository at this point in the history

  7. docs: update changelog

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    a69f5bc View commit details
    Browse the repository at this point in the history

  8. fix: don't use CHECK_DELAY_DEFAULT

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    4c4201e View commit details
    Browse the repository at this point in the history

  9. refactor: change next_check to next_check_at

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    df0f967 View commit details
    Browse the repository at this point in the history

  10. docs: comment fix

    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    e68ab04 View commit details
    Browse the repository at this point in the history

  11. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
#	udata_hydra/crawl/helpers.py
#	udata_hydra/crawl/process_check_data.py
#	udata_hydra/routes/status.py
    @bolinocroustibat
    bolinocroustibat committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    5a95774 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. fix: fix migration

    @bolinocroustibat
    bolinocroustibat committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    b32a703 View commit details
    Browse the repository at this point in the history

  2. fix: fix crawler status bug

    @bolinocroustibat
    bolinocroustibat committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    e67f2a0 View commit details
    Browse the repository at this point in the history

  3. clean: add types

    @bolinocroustibat
    bolinocroustibat committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    bf5a455 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2024

  1. refactor: calculate next_check_at date also depending on resource mod… 

    …ification date, as a standalone function
    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    6285a54 View commit details
    Browse the repository at this point in the history

  2. docs: update docstring

    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    5df8670 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    3a14cd1 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    9adbcff View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    eef5f21 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
#	udata_hydra/analysis/resource.py
#	udata_hydra/crawl/check_resources.py
    @bolinocroustibat
    bolinocroustibat committed Nov 6, 2024
    Configuration menu
    Copy the full SHA
    7f0348f View commit details
    Browse the repository at this point in the history

Commits on Nov 7, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 7, 2024
    Configuration menu
    Copy the full SHA
    0f1e2e1 View commit details
    Browse the repository at this point in the history

Commits on Nov 8, 2024

  1. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Nov 8, 2024
    Configuration menu
    Copy the full SHA
    913913d View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2024

  1. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    a60bff9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    85f0ef3 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    2c07642 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into unavailable-resources-management 

    # Conflicts:
#	CHANGELOG.md
    @bolinocroustibat
    bolinocroustibat committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    4bcb89b View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Nov 15, 2024
    Configuration menu
    Copy the full SHA
    927bb09 View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2024

  1. Merge branch 'main' into unavailable-resources-management

    @bolinocroustibat
    bolinocroustibat committed Nov 20, 2024
    Configuration menu
    Copy the full SHA
    5e651eb View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. fix: fix calculate_next_check type bug issue

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    6a9a7ea View commit details
    Browse the repository at this point in the history

  2. fix tests

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    abc45b6 View commit details
    Browse the repository at this point in the history

  3. fix: fix nasty bug in preprocess_check_data

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    8e7ec6f View commit details
    Browse the repository at this point in the history

  4. fix: fix select resource when last check has no next_check planned

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    0e89128 View commit details
    Browse the repository at this point in the history

  5. fix: fix select batch

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    cc69099 View commit details
    Browse the repository at this point in the history

  6. tests: add tests

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    d1590cb View commit details
    Browse the repository at this point in the history

  7. tests: improve tests

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    d58d4af View commit details
    Browse the repository at this point in the history

  8. feat: check get_all and get_latest methods return next_check_at

    @bolinocroustibat
    bolinocroustibat committed Nov 22, 2024
    Configuration menu
    Copy the full SHA
    9e0f626 View commit details
    Browse the repository at this point in the history