Docs2

docs/api/cli/cli.finding_archives.rst

@@ -0,0 +1,103 @@
+.. _cli-finding-archives:
+
+================
+Finding Archives
+================
+
+
+
+
+In this section we'll take a look at finding archives via the command line. 
+
+You can find archives from the command line interface or from :ref:`python <pythonapi-finding-archives>`. This documentation mirrors the python documentation. 
+
+
+Using ``listdir``
+~~~~~~~~~~~~~~~~~
+
+In our database we have many archives. We know that ``impactlab`` is a top-level directory-like namespace in our database. Let's have a look. 
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-1-START
+    :end-before: .. EXAMPLE-BLOCK-1-END
+
+
+Ok. We see that ``labor``, ``climate``, ``mortality`` and ``conflict`` are all directory-like namespaces groupings below ``impactlab``. Lets have a look at ``conflict``. 
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-2-START
+    :end-before: .. EXAMPLE-BLOCK-2-END
+
+Let's see what is in ``impactlab/conflict/global``. 
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-3-START
+    :end-before: .. EXAMPLE-BLOCK-3-END
+
+We can see that there is currently only version ``0.0.1`` of ``conflict_global_daily.csv``
+
+
+Using ``filter``
+~~~~~~~~~~~~~~~~~
+
+DataFS lets you filter so you can limit the search space on archive names. At the command line, you can use the ``prefix``, ``path``, ``str``, and ``regex`` pattern options to filter archives.
+Let's look at using the ``prefix`` ``project1_variable1_`` which corresponds to the ``prefix`` option, the beginning string of a set of archive names. 
+
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-4-START
+    :end-before: .. EXAMPLE-BLOCK-4-END
+
+We can also filter on ``path``. In this case we want to filter all NetCDF files that match a specific pattern. We need to set our ``engine`` value to ``path`` and put in our search pattern.
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-5-START
+    :end-before: .. EXAMPLE-BLOCK-5-END
+
+We can also filter archives with archive names containing a specific string by setting ``engine`` to ``str``. In this example we want all archives with the string ``variable2``. 
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-6-START
+    :end-before: .. EXAMPLE-BLOCK-6-END
+
+
+
+
+Using ``search``
+~~~~~~~~~~~~~~~~
+
+DataFS ``search`` capabilites are enabled via tagging of archives. The arguments of the ``search`` command are tags associated with a given archive. If archives are not tagged, they cannot be searched. Please see :ref:`this <cli-tagging>` for a reference on how to tag archives.
+
+Our archives have been tagged with ``team1``, ``team2``, or ``team3`` Let's search for some archives with tag ``team3``.
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-7-START
+    :end-before: .. EXAMPLE-BLOCK-7-END
+
+
+Let's use ``get_tags`` to have a look at one of our archives' tags.
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-8-START
+    :end-before: .. EXAMPLE-BLOCK-8-END
+
+We can see that indeed it has been tagged with ``team3``. 
+
+
+For completeness, let's have a look at archives with tag of ``team1``.   
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-9-START
+    :end-before: .. EXAMPLE-BLOCK-9-END
+
+And now let's have a look at one of them to see what tags are associated with it.
+
+
+.. include:: ../../../tests/cli_snippets/test_cli_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-10-START
+    :end-before: .. EXAMPLE-BLOCK-10-END
+
+We can see clearly that our archive has been tagged with ``team1``. 
+
+
+We want your feedback. If you find bugs or have suggestions to improve this documentation, please consider contributing. 

docs/api/cli/cli.io.rst

@@ -62,7 +62,7 @@ Now let's read this to make sure we got what we want
 Writing to Archives with Filepaths
 ----------------------------------
 
-Let's say we made some major edits to our sample_archive locally and we want to update them in the manager and at our authority. We can update the same as before but this time we'll add the filepath that points to our file.
+Let's say we made some major edits to `my_archive` locally and we want to update them in the manager and at our authority. We can update the same as before but this time we'll add the filepath that points to our file.
 
 .. include:: ../../../tests/cli_snippets/test_cli_io.py
     :start-after: .. EXAMPLE-BLOCK-7-START

docs/api/cli/cli.metadata.rst

@@ -13,7 +13,7 @@ Viewing Metadata
 ----------------
 
 
-We'll keep working with our ``sample_archive`` that we created earlier. Right now we'll take a look at our metadata. 
+We'll keep working with ``my_archive`` that we created earlier. Right now we'll take a look at our metadata. 
 
 .. include:: ../../../tests/cli_snippets/test_cli_metadata.py
     :start-after: .. EXAMPLE-BLOCK-1-START

docs/api/cli/cli.rst

@@ -11,4 +11,5 @@ Using the Command-Line Interface
     cli.io
     cli.metadata
     cli.versioning
-    cli.dependencies
+    cli.dependencies
+    cli.finding_archives

docs/api/python/pythonapi.finding_archives.rst

@@ -0,0 +1,127 @@
+.. _pythonapi-finding-archives:
+
+
+==============================
+Searching and Finding Archives
+==============================
+
+
+DataFS allows you to search and locate archives with the following methods: 
+:py:meth:`~datafs.DataAPI.listdir`, :py:meth:`~datafs.DataAPI.filter`, and :py:meth:`~datafs.DataAPI.search`. Let's look at each method to see how they work. 
+
+Using :py:meth:`~datafs.DataAPI.listdir`
+----------------------------------------
+
+:py:meth:`~datafs.DataAPI.listdir` works just like typlical unix style ``ls`` in the sense that it returns all objects subordinate to the specified directory. If your team has used ``/`` to organize archive naming then you can explore the archive namespace just as you would explore a directory in a filesystem.
+
+For example if we provide ``impactlab/conflict/global`` as an argument to ``listdir`` we get the following:  
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-1-START
+    :end-before: .. EXAMPLE-BLOCK-1-END
+
+It looks like we only have one file ``conflict_global_daily.csv`` in our directory.
+
+Let's see what kind of archives we have in our system. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-2-START
+    :end-before: .. EXAMPLE-BLOCK-2-END
+
+It looks like our top level directory is ``impactlab``. 
+
+Then if we use ``impactlab`` as an argument we see that we have several directory-like groupings below this. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-3-START
+    :end-before: .. EXAMPLE-BLOCK-3-END
+
+Let's explore ``conflict`` to see what kind of namespace groupings we have in there. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+	:start-after: .. EXAMPLE-BLOCK-4-START
+	:end-before: .. EXAMPLE-BLOCK-4-END
+
+OK. Just one. Now let's have a look inside the ``impactlab/conflict/global`` namespace. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+	:start-after: .. EXAMPLE-BLOCK-5-START
+	:end-before: .. EXAMPLE-BLOCK-5-END
+
+We see that if we give a full path with a file extension that we get version numbers of our archives. 
+
+Using :py:meth:`~datafs.DataAPI.filter`
+---------------------------------------
+
+DataFS also lets you filter so you can limit the search space on archive names. With :py:meth:`~datafs.DataAPI.filter` you can use the ``prefix``, ``path``, ``str``, and ``regex`` pattern options to filter archives.
+Let's look at using the prefix ``project1_variable1_`` which corresponds to the ``prefix`` option, the beginning string of a set of archive names. Let's also see how many archives we have in total by filtering without arguments. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-6-START
+    :end-before: .. EXAMPLE-BLOCK-6-END
+
+We see there are 125. By filtering with our prefix we can significantly reduce the number of archives we are looking at. 
+
+We can also filter on ``path``. In this case we want to filter all NetCDF files that match a specific pattern. We need to set our ``engine`` value to ``path`` and put in our search pattern. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-7-START
+    :end-before: .. EXAMPLE-BLOCK-7-END
+
+
+We can also filter archives with archive names containing a specific string by setting ``engine`` to ``str``. In this
+example we want all archives with the string ``variable2``. The filtering query returns 25 items. Let's look at the first few. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-8-START
+    :end-before: .. EXAMPLE-BLOCK-8-END
+
+
+Using :py:meth:`~datafs.DataAPI.search`
+---------------------------------------
+
+
+DataFS :py:meth:`~datafs.DataAPI.search` capabilites are enabled via tagging of archives. The arguments of the :py:meth:`~datafs.DataAPI.search` method are tags associated with a given archive. If archives are not tagged, they cannot be searched with the :py:meth:`~datafs.DataAPI.search` method. See :ref:`pythonapi-tagging` for info on how to tag archives.
+
+If we use :py:meth:`~datafs.DataAPI.search` without arguments, it is the same implementation as :py:meth:`~datafs.DataAPI.filter` without arguments. 
+
+Let's see this in action. 
+
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-9-START
+    :end-before: .. EXAMPLE-BLOCK-9-END
+
+Our archives have been tagged with ``team1``, ``team2``, or ``team3`` Let's search for some archives with tag ``team3``. It brings back 41 archives. So we'll just look at a few. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-10-START
+    :end-before: .. EXAMPLE-BLOCK-10-END
+
+And lets look at the some of these archives to see what their tags are. We'll use
+:py:meth:`~datafs.core.data_archive.DataArchive.get_tags`
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-11-START
+    :end-before: .. EXAMPLE-BLOCK-11-END
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-12-START
+    :end-before: .. EXAMPLE-BLOCK-12-END
+
+
+And how about with tag ``team1``. We see that there are 42 archives with ``team1`` tag. 
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-12-START
+    :end-before: .. EXAMPLE-BLOCK-12-END
+
+And let's use :py:meth:`~datafs.core.data_archive.DataArchive.get_tags` to confirm the tags are ``team1``
+
+.. include:: ../../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: .. EXAMPLE-BLOCK-13-START
+    :end-before: .. EXAMPLE-BLOCK-13-END
+
+
+
+We want your feedback. If you have improvements or suggestions for the documentation please consider making contributions. 

docs/api/python/pythonapi.rst

@@ -12,4 +12,5 @@ Using the Python API
     pythonapi.io
     pythonapi.metadata
     pythonapi.versioning
-    pythonapi.dependencies
+    pythonapi.dependencies
+    pythonapi.finding_archives

docs/samples/samples.pythonapi.finding.archives.rst

@@ -0,0 +1,3 @@
+.. include:: ../../examples/snippets/pythonapi_finding_archives.py
+    :start-after: '''
+    :end-before: '''

docs/samples/samples.rst

@@ -12,4 +12,5 @@ Documentation Code Source
     samples.pythonapi.io
     samples.pythonapi.metadata
     samples.pythonapi.versioning
-    samples.pythonapi.dependencies
+    samples.pythonapi.dependencies
+    samples.pythonapi.finding.archives

docs/whatsnew/v0.7.1.txt

@@ -149,10 +149,18 @@ Bug Fixes
 Under the hood
 ~~~~~~~~~~~~~~
 
-  - Use ``:issue:`` and ``:pull:`` directives to reference a github issue or pull request (:issue:`209`)
-  - The sphinx build is now tested on travis. Run the tests locally with the command ``sphinx-build -W -b html -d docs/_build/doctrees docs/. docs/_build/html`` (:issue:`211`)
-  - The docs structure has been reorganized
+	- Use ``:issue:`` and ``:pull:`` directives to reference a github issue or pull request (:issue:`209`)
+
+	- The sphinx build is now tested on travis. Run the tests locally with the command ``sphinx-build -W -b html -d docs/_build/doctrees docs/. docs/_build/html`` (:issue:`211`)
+
+	- The docs structure has been reorganized
+
   - Conda dependencies pinned in ``requirements_conda.txt``, and the channel ``conda-forge`` was added to the travis conda environment so we have access to the latest conda builds. (:issue:`247`)
+
   - Running the ``configure`` command not creates an empty 'default' profile if no configuration file exists
 
+  - Additional documentation on :ref:`tagging files <pythonapi-tagging>`  and :ref:`searching and finding files <pythonapi-finding-archives>`
+
+
+
 See the issue tracker on GitHub for a complete list.