diff --git a/docs/api/cli/cli.finding_archives.rst b/docs/api/cli/cli.finding_archives.rst new file mode 100644 index 0000000..490bb36 --- /dev/null +++ b/docs/api/cli/cli.finding_archives.rst @@ -0,0 +1,103 @@ +.. _cli-finding-archives: + +================ +Finding Archives +================ + + + + +In this section we'll take a look at finding archives via the command line. + +You can find archives from the command line interface or from :ref:`python `. This documentation mirrors the python documentation. + + +Using ``listdir`` +~~~~~~~~~~~~~~~~~ + +In our database we have many archives. We know that ``impactlab`` is a top-level directory-like namespace in our database. Let's have a look. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-1-START + :end-before: .. EXAMPLE-BLOCK-1-END + + +Ok. We see that ``labor``, ``climate``, ``mortality`` and ``conflict`` are all directory-like namespaces groupings below ``impactlab``. Lets have a look at ``conflict``. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-2-START + :end-before: .. EXAMPLE-BLOCK-2-END + +Let's see what is in ``impactlab/conflict/global``. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-3-START + :end-before: .. EXAMPLE-BLOCK-3-END + +We can see that there is currently only version ``0.0.1`` of ``conflict_global_daily.csv`` + + +Using ``filter`` +~~~~~~~~~~~~~~~~~ + +DataFS lets you filter so you can limit the search space on archive names. At the command line, you can use the ``prefix``, ``path``, ``str``, and ``regex`` pattern options to filter archives. +Let's look at using the ``prefix`` ``project1_variable1_`` which corresponds to the ``prefix`` option, the beginning string of a set of archive names. + + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-4-START + :end-before: .. EXAMPLE-BLOCK-4-END + +We can also filter on ``path``. In this case we want to filter all NetCDF files that match a specific pattern. We need to set our ``engine`` value to ``path`` and put in our search pattern. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-5-START + :end-before: .. EXAMPLE-BLOCK-5-END + +We can also filter archives with archive names containing a specific string by setting ``engine`` to ``str``. In this example we want all archives with the string ``variable2``. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-6-START + :end-before: .. EXAMPLE-BLOCK-6-END + + + + +Using ``search`` +~~~~~~~~~~~~~~~~ + +DataFS ``search`` capabilites are enabled via tagging of archives. The arguments of the ``search`` command are tags associated with a given archive. If archives are not tagged, they cannot be searched. Please see :ref:`this ` for a reference on how to tag archives. + +Our archives have been tagged with ``team1``, ``team2``, or ``team3`` Let's search for some archives with tag ``team3``. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-7-START + :end-before: .. EXAMPLE-BLOCK-7-END + + +Let's use ``get_tags`` to have a look at one of our archives' tags. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-8-START + :end-before: .. EXAMPLE-BLOCK-8-END + +We can see that indeed it has been tagged with ``team3``. + + +For completeness, let's have a look at archives with tag of ``team1``. + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-9-START + :end-before: .. EXAMPLE-BLOCK-9-END + +And now let's have a look at one of them to see what tags are associated with it. + + +.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-10-START + :end-before: .. EXAMPLE-BLOCK-10-END + +We can see clearly that our archive has been tagged with ``team1``. + + +We want your feedback. If you find bugs or have suggestions to improve this documentation, please consider contributing. diff --git a/docs/api/cli/cli.io.rst b/docs/api/cli/cli.io.rst index 568df74..9e324e9 100644 --- a/docs/api/cli/cli.io.rst +++ b/docs/api/cli/cli.io.rst @@ -62,7 +62,7 @@ Now let's read this to make sure we got what we want Writing to Archives with Filepaths ---------------------------------- -Let's say we made some major edits to our sample_archive locally and we want to update them in the manager and at our authority. We can update the same as before but this time we'll add the filepath that points to our file. +Let's say we made some major edits to `my_archive` locally and we want to update them in the manager and at our authority. We can update the same as before but this time we'll add the filepath that points to our file. .. include:: ../../../tests/cli_snippets/test_cli_io.py :start-after: .. EXAMPLE-BLOCK-7-START diff --git a/docs/api/cli/cli.metadata.rst b/docs/api/cli/cli.metadata.rst index 15ddf6f..7a9e69e 100644 --- a/docs/api/cli/cli.metadata.rst +++ b/docs/api/cli/cli.metadata.rst @@ -13,7 +13,7 @@ Viewing Metadata ---------------- -We'll keep working with our ``sample_archive`` that we created earlier. Right now we'll take a look at our metadata. +We'll keep working with ``my_archive`` that we created earlier. Right now we'll take a look at our metadata. .. include:: ../../../tests/cli_snippets/test_cli_metadata.py :start-after: .. EXAMPLE-BLOCK-1-START diff --git a/docs/api/cli/cli.rst b/docs/api/cli/cli.rst index c1c7f13..eb3285b 100644 --- a/docs/api/cli/cli.rst +++ b/docs/api/cli/cli.rst @@ -11,4 +11,5 @@ Using the Command-Line Interface cli.io cli.metadata cli.versioning - cli.dependencies \ No newline at end of file + cli.dependencies + cli.finding_archives \ No newline at end of file diff --git a/docs/api/python/pythonapi.finding_archives.rst b/docs/api/python/pythonapi.finding_archives.rst new file mode 100644 index 0000000..2572f4c --- /dev/null +++ b/docs/api/python/pythonapi.finding_archives.rst @@ -0,0 +1,127 @@ +.. _pythonapi-finding-archives: + + +============================== +Searching and Finding Archives +============================== + + +DataFS allows you to search and locate archives with the following methods: +:py:meth:`~datafs.DataAPI.listdir`, :py:meth:`~datafs.DataAPI.filter`, and :py:meth:`~datafs.DataAPI.search`. Let's look at each method to see how they work. + +Using :py:meth:`~datafs.DataAPI.listdir` +---------------------------------------- + +:py:meth:`~datafs.DataAPI.listdir` works just like typlical unix style ``ls`` in the sense that it returns all objects subordinate to the specified directory. If your team has used ``/`` to organize archive naming then you can explore the archive namespace just as you would explore a directory in a filesystem. + +For example if we provide ``impactlab/conflict/global`` as an argument to ``listdir`` we get the following: + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-1-START + :end-before: .. EXAMPLE-BLOCK-1-END + +It looks like we only have one file ``conflict_global_daily.csv`` in our directory. + +Let's see what kind of archives we have in our system. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-2-START + :end-before: .. EXAMPLE-BLOCK-2-END + +It looks like our top level directory is ``impactlab``. + +Then if we use ``impactlab`` as an argument we see that we have several directory-like groupings below this. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-3-START + :end-before: .. EXAMPLE-BLOCK-3-END + +Let's explore ``conflict`` to see what kind of namespace groupings we have in there. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-4-START + :end-before: .. EXAMPLE-BLOCK-4-END + +OK. Just one. Now let's have a look inside the ``impactlab/conflict/global`` namespace. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-5-START + :end-before: .. EXAMPLE-BLOCK-5-END + +We see that if we give a full path with a file extension that we get version numbers of our archives. + +Using :py:meth:`~datafs.DataAPI.filter` +--------------------------------------- + +DataFS also lets you filter so you can limit the search space on archive names. With :py:meth:`~datafs.DataAPI.filter` you can use the ``prefix``, ``path``, ``str``, and ``regex`` pattern options to filter archives. +Let's look at using the prefix ``project1_variable1_`` which corresponds to the ``prefix`` option, the beginning string of a set of archive names. Let's also see how many archives we have in total by filtering without arguments. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-6-START + :end-before: .. EXAMPLE-BLOCK-6-END + +We see there are 125. By filtering with our prefix we can significantly reduce the number of archives we are looking at. + +We can also filter on ``path``. In this case we want to filter all NetCDF files that match a specific pattern. We need to set our ``engine`` value to ``path`` and put in our search pattern. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-7-START + :end-before: .. EXAMPLE-BLOCK-7-END + + +We can also filter archives with archive names containing a specific string by setting ``engine`` to ``str``. In this +example we want all archives with the string ``variable2``. The filtering query returns 25 items. Let's look at the first few. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-8-START + :end-before: .. EXAMPLE-BLOCK-8-END + + +Using :py:meth:`~datafs.DataAPI.search` +--------------------------------------- + + +DataFS :py:meth:`~datafs.DataAPI.search` capabilites are enabled via tagging of archives. The arguments of the :py:meth:`~datafs.DataAPI.search` method are tags associated with a given archive. If archives are not tagged, they cannot be searched with the :py:meth:`~datafs.DataAPI.search` method. See :ref:`pythonapi-tagging` for info on how to tag archives. + +If we use :py:meth:`~datafs.DataAPI.search` without arguments, it is the same implementation as :py:meth:`~datafs.DataAPI.filter` without arguments. + +Let's see this in action. + + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-9-START + :end-before: .. EXAMPLE-BLOCK-9-END + +Our archives have been tagged with ``team1``, ``team2``, or ``team3`` Let's search for some archives with tag ``team3``. It brings back 41 archives. So we'll just look at a few. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-10-START + :end-before: .. EXAMPLE-BLOCK-10-END + +And lets look at the some of these archives to see what their tags are. We'll use +:py:meth:`~datafs.core.data_archive.DataArchive.get_tags` + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-11-START + :end-before: .. EXAMPLE-BLOCK-11-END + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-12-START + :end-before: .. EXAMPLE-BLOCK-12-END + + +And how about with tag ``team1``. We see that there are 42 archives with ``team1`` tag. + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-12-START + :end-before: .. EXAMPLE-BLOCK-12-END + +And let's use :py:meth:`~datafs.core.data_archive.DataArchive.get_tags` to confirm the tags are ``team1`` + +.. include:: ../../../examples/snippets/pythonapi_finding_archives.py + :start-after: .. EXAMPLE-BLOCK-13-START + :end-before: .. EXAMPLE-BLOCK-13-END + + + +We want your feedback. If you have improvements or suggestions for the documentation please consider making contributions. diff --git a/docs/api/python/pythonapi.rst b/docs/api/python/pythonapi.rst index e01767e..5c1cfae 100644 --- a/docs/api/python/pythonapi.rst +++ b/docs/api/python/pythonapi.rst @@ -12,4 +12,5 @@ Using the Python API pythonapi.io pythonapi.metadata pythonapi.versioning - pythonapi.dependencies \ No newline at end of file + pythonapi.dependencies + pythonapi.finding_archives \ No newline at end of file diff --git a/docs/samples/samples.pythonapi.finding.archives.rst b/docs/samples/samples.pythonapi.finding.archives.rst new file mode 100644 index 0000000..fb83623 --- /dev/null +++ b/docs/samples/samples.pythonapi.finding.archives.rst @@ -0,0 +1,3 @@ +.. include:: ../../examples/snippets/pythonapi_finding_archives.py + :start-after: ''' + :end-before: ''' \ No newline at end of file diff --git a/docs/samples/samples.rst b/docs/samples/samples.rst index ff64ee5..4edd571 100644 --- a/docs/samples/samples.rst +++ b/docs/samples/samples.rst @@ -12,4 +12,5 @@ Documentation Code Source samples.pythonapi.io samples.pythonapi.metadata samples.pythonapi.versioning - samples.pythonapi.dependencies \ No newline at end of file + samples.pythonapi.dependencies + samples.pythonapi.finding.archives \ No newline at end of file diff --git a/docs/whatsnew/v0.7.1.txt b/docs/whatsnew/v0.7.1.txt index 23ecf5a..701a940 100644 --- a/docs/whatsnew/v0.7.1.txt +++ b/docs/whatsnew/v0.7.1.txt @@ -149,10 +149,18 @@ Bug Fixes Under the hood ~~~~~~~~~~~~~~ - - Use ``:issue:`` and ``:pull:`` directives to reference a github issue or pull request (:issue:`209`) - - The sphinx build is now tested on travis. Run the tests locally with the command ``sphinx-build -W -b html -d docs/_build/doctrees docs/. docs/_build/html`` (:issue:`211`) - - The docs structure has been reorganized + - Use ``:issue:`` and ``:pull:`` directives to reference a github issue or pull request (:issue:`209`) + + - The sphinx build is now tested on travis. Run the tests locally with the command ``sphinx-build -W -b html -d docs/_build/doctrees docs/. docs/_build/html`` (:issue:`211`) + + - The docs structure has been reorganized + - Conda dependencies pinned in ``requirements_conda.txt``, and the channel ``conda-forge`` was added to the travis conda environment so we have access to the latest conda builds. (:issue:`247`) + - Running the ``configure`` command not creates an empty 'default' profile if no configuration file exists + - Additional documentation on :ref:`tagging files ` and :ref:`searching and finding files ` + + + See the issue tracker on GitHub for a complete list. diff --git a/examples/snippets/pythonapi_finding_archives.py b/examples/snippets/pythonapi_finding_archives.py new file mode 100644 index 0000000..f0cce82 --- /dev/null +++ b/examples/snippets/pythonapi_finding_archives.py @@ -0,0 +1,348 @@ +''' +.. _snippets-pythonapi-finding-archives: + +Python API: Searching and Finding Archives +========================================== + +This is the tested source code for the snippets used in +:ref:`pythonapi-finding-archives`. The config file we're using in this example +can be downloaded +:download:`here <../../examples/snippets/resources/datafs_mongo.yml>`. + + +Setup +----- +.. code-block:: python + + >>> import datafs + >>> from fs.tempfs import TempFS + >>> import os + >>> import itertools + +We test with the following setup: + +.. code-block:: python + + >>> api = datafs.get_api( + ... config_file='examples/snippets/resources/datafs_mongo.yml') + ... + +This assumes that you have a config file at the above location. The config file +we're using in this example can be downloaded +:download:`here <../../examples/snippets/resources/datafs_mongo.yml>`. + + +clean up any previous test failures + +.. code-block:: python + + >>> try: + ... api.delete_archive('my_archive') + ... api.delete_archive('streaming_archive') + ... api.delete_archive('sample_archive') + ... except (KeyError, OSError): + ... pass + ... + >>> try: + ... api.manager.delete_table('DataFiles') + ... except KeyError: + ... pass + ... + +Add a fresh manager table: + +.. code-block:: python + + >>> api.manager.create_archive_table('DataFiles') + +Set up some archives to search + +.. code-block:: python + + >>> with open('test.txt', 'w') as f: + ... f.write('test test') + ... + >>> tas_archive = api.create('impactlab/climate/tas/tas_day_us.csv') + >>> tas_archive.update('test.txt') + >>> precip_archive = api.create('impactlab/climate/pr/pr_day_us.csv') + >>> precip_archive.update('test.txt') + >>> socio = api.create('impactlab/mortality/global/mortality_glob_day.csv') + >>> socio.update('test.txt') + >>> socio1 = api.create('impactlab/conflict/global/conflict_glob_day.csv') + >>> socio1.update('test.txt') + >>> socio2 = api.create('impactlab/labor/global/labor_glob_day.csv') + >>> socio2.update('test.txt') + +Example 1 +--------- + +Displayed example 1 code + +.. EXAMPLE-BLOCK-1-START + +.. code-block:: python + + >>> api.listdir('impactlab/conflict/global') + [u'conflict_glob_day.csv'] + +.. EXAMPLE-BLOCK-1-END + +Example 2 +--------- + +Displayed example 2 code + +.. EXAMPLE-BLOCK-2-START + +.. code-block:: python + + >>> api.listdir('') + [u'impactlab'] + +.. EXAMPLE-BLOCK-2-END + +Example 3 +--------- + +Displayed example 3 code + +.. EXAMPLE-BLOCK-3-START + +.. code-block:: python + + >>> api.listdir('impactlab') + [u'labor', u'climate', u'conflict', u'mortality'] + +.. EXAMPLE-BLOCK-3-END + +Example 4 +--------- + +Displayed example 4 code + +.. EXAMPLE-BLOCK-4-START + +.. code-block:: python + + >>> api.listdir('impactlab/conflict') + [u'global'] + +.. EXAMPLE-BLOCK-4-END + +Example 5 +--------- + +Displayed example 5 code + +.. EXAMPLE-BLOCK-5-START + +.. code-block:: python + + >>> api.listdir('impactlab/conflict/global') + [u'conflict_glob_day.csv'] + >>> api.listdir('impactlab/conflict/global/conflict_glob_day.csv') + [u'0.0.1'] + +.. EXAMPLE-BLOCK-5-END + +Teardown +-------- + +.. code-block:: python + + >>> try: + ... tas_archive.delete() + ... precip_archive.delete() + ... socio.delete() + ... socio1.delete() + ... socio2.delete() + ... os.remove('test.txt') + ... except KeyError: + ... pass + >>> try: + ... api.manager.delete_table('DataFiles') + ... except KeyError: + ... pass + +Setup + +.. code-block:: python + +>>> api.manager.create_archive_table('DataFiles') + +Filter example setup + +.. code-block:: python + + >>> archive_names = [] # doctest: +ELLIPSIS + >>> for indices in itertools.product(*(range(1, 6) for _ in range(3))): + ... archive_name = ( + ... 'project{}_variable{}_scenario{}.nc'.format(*indices)) + ... archive_names.append(archive_name) + >>> + >>> for i, name in enumerate(archive_names): + ... if i % 3 == 0: + ... api.create(name, tags=['team1']) + ... elif i % 2 == 0: + ... api.create(name, tags=['team2']) + ... else: + ... api.create(name, tags=['team3']) # doctest: +ELLIPSIS + + + + ... + + + + +Example 6 +--------- + +Displayed example 6 code + +.. EXAMPLE-BLOCK-6-START + +.. code-block:: python + + >>> len(list(api.filter())) + 125 + >>> filtered_list1 = api.filter(prefix='project1_variable1_') + >>> list(filtered_list1) # doctest: +NORMALIZE_WHITESPACE + [u'project1_variable1_scenario1.nc', u'project1_variable1_scenario2.nc', + u'project1_variable1_scenario3.nc', u'project1_variable1_scenario4.nc', + u'project1_variable1_scenario5.nc'] + +.. EXAMPLE-BLOCK-6-END + +Example 7 +--------- + +Displayed example 7 code + +.. EXAMPLE-BLOCK-7-START + +.. code-block:: python + + >>> filtered_list2 = api.filter(pattern='*_variable4_scenario4.nc', + ... engine='path') + >>> list(filtered_list2) # doctest: +NORMALIZE_WHITESPACE + [u'project1_variable4_scenario4.nc', u'project2_variable4_scenario4.nc', + u'project3_variable4_scenario4.nc', u'project4_variable4_scenario4.nc', + u'project5_variable4_scenario4.nc'] + +.. EXAMPLE-BLOCK-7-END + +Example 8 +--------- + +Displayed example 8 code + +.. EXAMPLE-BLOCK-8-START + +.. code-block:: python + + >>> filtered_list3 = list(api.filter(pattern='variable2', engine='str')) + >>> len(filtered_list3) + 25 + >>> filtered_list3[:4] # doctest: +NORMALIZE_WHITESPACE + [u'project1_variable2_scenario1.nc', u'project1_variable2_scenario2.nc', + u'project1_variable2_scenario3.nc', u'project1_variable2_scenario4.nc'] + +.. EXAMPLE-BLOCK-8-END + +Example 9 +--------- + +Displayed example 9 code + +.. EXAMPLE-BLOCK-9-START + +.. code-block:: python + + >>> archives_search = list(api.search()) + >>> archives_filter = list(api.filter()) + >>> len(archives_search) + 125 + >>> len(archives_filter) + 125 + +.. EXAMPLE-BLOCK-9-END + +Example 10 +---------- + +Displayed example 10 code + +.. EXAMPLE-BLOCK-10-START + +.. code-block:: python + + >>> tagged_search = list(api.search('team3')) + >>> len(tagged_search) + 41 + >>> tagged_search[:4] #doctest: +NORMALIZE_WHITESPACE + [u'project1_variable1_scenario2.nc', u'project1_variable2_scenario1.nc', + u'project1_variable2_scenario3.nc', u'project1_variable3_scenario2.nc'] + +.. EXAMPLE-BLOCK-10-END + +Example 11 +---------- + +Displayed example 11 code + +.. EXAMPLE-BLOCK-11-START + +.. code-block:: python + + >>> tags = [] + >>> for arch in tagged_search[:4]: + ... tags.append(api.manager.get_tags(arch)[0]) + >>> tags + [u'team3', u'team3', u'team3', u'team3'] + +.. EXAMPLE-BLOCK-11-END + +Example 12 +---------- + +Displayed example 12 code + +.. EXAMPLE-BLOCK-12-START + +.. code-block:: python + + >>> tagged_search_team1 = list(api.search('team1')) + >>> len(tagged_search_team1) + 42 + >>> tagged_search_team1[:4] #doctest: +NORMALIZE_WHITESPACE + [u'project1_variable1_scenario1.nc', u'project1_variable1_scenario4.nc', + u'project1_variable2_scenario2.nc', u'project1_variable2_scenario5.nc'] + +.. EXAMPLE-BLOCK-12-END + +Example 13 +---------- + +Displayed example 13 code + +.. EXAMPLE-BLOCK-13-START + +.. code-block:: python + + >>> tags = [] + >>> for arch in tagged_search_team1[:4]: + ... tags.append(api.manager.get_tags(arch)[0]) + >>> tags + [u'team1', u'team1', u'team1', u'team1'] + +.. EXAMPLE-BLOCK-13-END + +Teardown +-------- + +.. code-block:: python + + >>> api.manager.delete_table('DataFiles') + +''' diff --git a/examples/snippets/pythonapi_io.py b/examples/snippets/pythonapi_io.py index 2bde36f..e767594 100644 --- a/examples/snippets/pythonapi_io.py +++ b/examples/snippets/pythonapi_io.py @@ -267,4 +267,11 @@ .. EXAMPLE-BLOCK-11-END +Teardown +-------- + +.. code-block:: python + + >>> streaming_archive.delete() + ''' diff --git a/tests/cli_snippets/conftest.py b/tests/cli_snippets/conftest.py index 8a80713..74030f2 100644 --- a/tests/cli_snippets/conftest.py +++ b/tests/cli_snippets/conftest.py @@ -162,6 +162,7 @@ def cli_validator_listdir(cli_setup, validator): socio1.update('test.txt') socio2 = api.create('impactlab/labor/global/labor_global_daily.csv') socio2.update('test.txt') + validator.call_engines['datafs'] = ClickValidator(app=cli, prefix=prefix) yield validator.teststring diff --git a/tests/cli_snippets/test_cli_finding_archives.py b/tests/cli_snippets/test_cli_finding_archives.py new file mode 100644 index 0000000..3d0c18d --- /dev/null +++ b/tests/cli_snippets/test_cli_finding_archives.py @@ -0,0 +1,180 @@ +import pytest + + +@pytest.mark.examples +@pytest.mark.cli_snippets +def test_cli_listdir(cli_validator_listdir): + cli_validator_listdir(r''' + + +Snippet 1 + +.. EXAMPLE-BLOCK-1-START + +.. code-block:: bash + + $ datafs listdir impactlab + labor + climate + conflict + mortality + +.. EXAMPLE-BLOCK-1-END + +Snippet 2 + +.. EXAMPLE-BLOCK-2-START + +.. code-block:: bash + + $ datafs listdir impactlab/conflict + global + +.. EXAMPLE-BLOCK-2-END + +Snippet 3 + +.. EXAMPLE-BLOCK-3-START + +.. code-block:: bash + + $ datafs listdir impactlab/conflict/global + conflict_global_daily.csv + $ datafs listdir impactlab/conflict/global/conflict_global_daily.csv + 0.0.1 + +.. EXAMPLE-BLOCK-3-END + +''') + + +@pytest.mark.examples +@pytest.mark.cli_snippets +def test_cli_filter(cli_validator_manager_various): + cli_validator_manager_various(r''' + + +Snippet 4 + +.. EXAMPLE-BLOCK-4-START + +.. code-block:: bash + + $ datafs filter --prefix project1_variable1_ # doctest: +SKIP + project1_variable1_scenario5.nc + project1_variable1_scenario1.nc + project1_variable1_scenario4.nc + project1_variable1_scenario2.nc + project1_variable1_scenario3.nc + +.. EXAMPLE-BLOCK-4-END + +Snippet 5 + +.. EXAMPLE-BLOCK-5-START + +.. code-block:: bash + + $ datafs filter --pattern *_variable4_scenario4.nc --engine path \ + # doctest: +SKIP + project1_variable4_scenario4.nc + project2_variable4_scenario4.nc + project3_variable4_scenario4.nc + project5_variable4_scenario4.nc + project4_variable4_scenario4.nc + +.. EXAMPLE-BLOCK-5-END + +Snippet 6 + +.. EXAMPLE-BLOCK-6-START + +.. code-block:: bash + + $ datafs filter --pattern variable2 --engine str # doctest: +ELLIPSIS +SKIP + project1_variable2_scenario1.nc + project1_variable2_scenario2.nc + project1_variable2_scenario3.nc + ... + project5_variable2_scenario3.nc + project5_variable2_scenario4.nc + project5_variable2_scenario5.nc + +.. EXAMPLE-BLOCK-6-END + +''') + + +@pytest.mark.examples +@pytest.mark.cli_snippets +def test_cli_search(cli_validator_manager_various): + + cli_validator_manager_various(r''' + + +Snippet 7 + +.. EXAMPLE-BLOCK-7-START + +.. code-block:: bash + + $ datafs search team3 # doctest: +ELLIPSIS +SKIP + project2_variable2_scenario2.nc + project5_variable4_scenario1.nc + project1_variable5_scenario4.nc + project3_variable2_scenario1.nc + project2_variable1_scenario1.nc + ... + project5_variable1_scenario2.nc + project2_variable5_scenario5.nc + project5_variable2_scenario5.nc + project3_variable2_scenario5.nc + +.. EXAMPLE-BLOCK-7-END + + +Snippet 8 + +.. EXAMPLE-BLOCK-8-START + +.. code-block:: bash + + $ datafs get_tags project2_variable2_scenario2.nc + team3 +.. EXAMPLE-BLOCK-8-END + + +Snippet 9 + +.. EXAMPLE-BLOCK-9-START + +.. code-block:: bash + + $ datafs search team1 # doctest: +ELLIPSIS +SKIP + project1_variable1_scenario4.nc + project1_variable2_scenario2.nc + project1_variable2_scenario5.nc + project1_variable3_scenario3.nc + project1_variable4_scenario1.nc + project1_variable4_scenario4.nc + ... + project5_variable3_scenario2.nc + project5_variable3_scenario5.nc + project5_variable4_scenario3.nc + project5_variable5_scenario1.nc + project5_variable5_scenario4.nc + + +.. EXAMPLE-BLOCK-9-END + +Snippet 10 + +.. EXAMPLE-BLOCK-10-START + +.. code-block:: bash + + $ datafs get_tags project2_variable5_scenario1.nc + team1 + +.. EXAMPLE-BLOCK-10-END +''') diff --git a/tests/test_examples.py b/tests/test_examples.py index 4982201..8507490 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -13,7 +13,8 @@ pythonapi_dependencies, pythonapi_io, pythonapi_metadata, - pythonapi_versioning) + pythonapi_versioning, + pythonapi_finding_archives) from datafs.managers.manager_dynamo import DynamoDBManager from tests.resources import has_special_dependencies @@ -134,3 +135,10 @@ def test_docs_pythonapi_metadata(example_snippet_working_dirs): def test_docs_pythonapi_versioning(example_snippet_working_dirs): failures, _ = doctest.testmod(pythonapi_versioning, report=True) assert failures == 0 + + +@pytest.mark.examples +@pytest.mark.python_snippets +def test_docs_pythonapi_finding_archives(example_snippet_working_dirs): + failures, _ = doctest.testmod(pythonapi_finding_archives, report=True) + assert failures == 0