Skip to content

Commit

Permalink
add configuration example in docs and improve how-to section
Browse files Browse the repository at this point in the history
  • Loading branch information
JessyBarrette committed Aug 15, 2023
1 parent 1f65be6 commit b211e25
Show file tree
Hide file tree
Showing 3 changed files with 86 additions and 32 deletions.
23 changes: 20 additions & 3 deletions docs/cli.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
## Command line interface
The ocean-data-parser provides a command line interface (CLI) that can be use to run a number of different data transformations.

::: mkdocs-click
:module: ocean_data_parser.cli
:command: main

!!! Tip "Environment Variables"
All the inputs available within the `odpy` command can be defined through environment variables: `ODPY_*`, `ODPY_CONVERT_*` and `ODPY_COMPILE_*` respectively.

Expand All @@ -11,7 +15,20 @@ The ocean-data-parser provides a command line interface (CLI) that can be use to
!!! Warning
An argument take priority over an environment variable.

::: mkdocs-click
:module: ocean_data_parser.cli
:command: main
## Convert Configuration
`odpy convert` can be used via a configuration file like the following:

``` {.yaml title="odpy-convert-config.yaml" .annotate}
--8<-- "ocean_data_parser/batch/default-batch-config.yaml"
```

!!! info
A new configuration file can be generated via `odpy convert`:

```console
odpy convert --new-config path/to/config.yaml
```




75 changes: 47 additions & 28 deletions docs/how-to.md
Original file line number Diff line number Diff line change
@@ -1,41 +1,60 @@
## Command line
---
hide:
- navigation
---
### Conversion

You can use the odpy command within a terminal to access the different functions of the `ocean-data-parser`.
??? question "Convert files to netcdf"

For more info use the command:
```
odpy --help
```
You can use the odpy command within a terminal to access the different functions of the `ocean-data-parser`.

!!! Info
For more details on how to use the command line interface see [commmand line section](cli.md).
Convert all cnv files in subdiretories by using the seabird.cnv parser and save to output directory:
```console
odpy convert --input-path input/**/*.cnv --parser=seabird.cnv --output_path=output
```

See [commmand line section](cli.md) for more detail or use the command: `odpy convert --help`

## Import a specific parser
??? question "Avoid reconverting over again the same files"

As an example, to load a compatible file you can use the automated parser detection method:
The `ocean-data-parser` provides a file retristry which can be used to:
- track which files were converted and outputted where
- error associated with each files
- file modified time
- file hash

```python
from ocean_data_parser import read
If activated, a registry file (*.csv/*.parquet) will be saved and any time a conversion is rerun. odpy will first compare the available files to the already parsed files available within the registry and will only convert the ones which have changes. Those changes are primarily based on the file modified time, each modified file is then rehashed and if that hash is different the file will be reconverted and the output overwritten (default)

# Load a file to an xarray object
ds = read.file('Path to file')
### Parser handling

# Save to netcdf
ds.to_netcdf('save-path.nc')
```
??? question "Load any compatible files in my own project"

!!! warning
The parser detection method relies on the file extension and the first few lines present within the given file.

Or specify the specific parser to use for this file format:
``` python
from ocean_data_parser.parsers import seabird
To load a compatible file you can use the automated parser detection method:

# Load a seabird cnv file as an xarray dataset
ds = seabird.cnv('Path to seabird cnv file')
```python
from ocean_data_parser import read

# Save to netcdf
ds.to_netcdf('save-path.nc')
```
The `ocean-data-parser` can then be used within either a python package, script or jupyter notebook. See [documentation Notebook section](https://cioos-siooc.github.io/ocean-data-parser) for examples on how to use the package within a jupyter notebook.
# Load a file to an xarray object
ds = read.file('Path to file')

# Save to netcdf
ds.to_netcdf('save-path.nc')
```

:warning: The parser detection method relies on the file extension and the first few lines present within the given file. It is preferable to define a specific parser when a tool is used in production.



??? question "Load a file with a specific parser in my own project"
You can import a specific parser via the `ocean_data_parser.parser`
``` python
from ocean_data_parser.parsers import seabird

# Load a seabird cnv file as an xarray dataset
ds = seabird.cnv('Path to seabird cnv file')

# Save to netcdf
ds.to_netcdf('save-path.nc')
```
The `ocean-data-parser` can then be used within either a python package, script or jupyter notebook. See [documentation Notebook section](https://cioos-siooc.github.io/ocean-data-parser) for examples on how to use the package within a jupyter notebook.
20 changes: 19 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ theme:
- toc.follow
- navigation.tabs
- navigation.tabs.sticky
- content.code.copy
- content.code.select



plugins:
- mkdocstrings
Expand All @@ -53,7 +57,21 @@ markdown_extensions:
- def_list
- pymdownx.tasklist:
custom_checkbox: true

- pymdownx.highlight:
anchor_linenums: true
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.superfences
- admonition
- pymdownx.details
- pymdownx.superfences
- attr_list
- pymdownx.emoji:
emoji_index: !!python/name:materialx.emoji.twemoji
emoji_generator: !!python/name:materialx.emoji.to_svg

nav:
- Welcome: index.md
- Install: install.md
Expand Down

0 comments on commit b211e25

Please sign in to comment.