Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Add docs with mkdocs #202

Merged
merged 32 commits into from
Oct 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
49dc26a
Merge pull request #204 from peytondmurray/17-lint-workflow
XinranTang Oct 9, 2024
79eb203
Add mkdocs as optional dependency
smokestacklightnin Aug 29, 2024
34d7b80
Add `mkdocs.yml` along with all necessary files
smokestacklightnin Aug 29, 2024
0d9e6d6
Move icon to `images` folder
smokestacklightnin Aug 29, 2024
15a7747
Add workflow to deploy docs
smokestacklightnin Aug 29, 2024
d6a4e36
Ignore test modules
smokestacklightnin Aug 29, 2024
874787d
Fix newline issues in "Raises" and "Returns" sections
smokestacklightnin Aug 29, 2024
f24501d
Remove `.md`
smokestacklightnin Aug 30, 2024
476ee22
Add empty `index.md`
smokestacklightnin Aug 30, 2024
e0c3c08
Change favicon
smokestacklightnin Aug 30, 2024
2cd9e51
Add attributes to docstring
smokestacklightnin Aug 30, 2024
c3131ac
Temporary fix for unused import
smokestacklightnin Aug 30, 2024
27905ca
Fill out Overview section
smokestacklightnin Aug 30, 2024
941dfe0
Use indentation instead of slash character `\`
smokestacklightnin Sep 1, 2024
a510016
Add overview from `mlmd.errors` api docs
smokestacklightnin Sep 1, 2024
2649a31
Add `show_if_no_docstring` option to mlmd.errors
smokestacklightnin Sep 1, 2024
f5b18b8
Use spaces instead of tabs
smokestacklightnin Sep 1, 2024
4802bb2
Add overview with links.
smokestacklightnin Sep 1, 2024
47960a4
Add link to tutorial
smokestacklightnin Sep 23, 2024
ead6757
Fix repo name
smokestacklightnin Sep 23, 2024
18875d8
Rename `root.md` to `index.md`
smokestacklightnin Sep 23, 2024
f5de8a5
Change links to point internally
smokestacklightnin Sep 23, 2024
61b75e9
Remove logo
smokestacklightnin Sep 23, 2024
946ef60
Fix broken link
smokestacklightnin Sep 23, 2024
50ed6c1
Fix admonitions
smokestacklightnin Sep 24, 2024
3844057
Add white background to png files
smokestacklightnin Sep 24, 2024
a3adca3
Add build docs check on pull request
smokestacklightnin Sep 29, 2024
7f63761
Remove `--verbose` flag
smokestacklightnin Oct 2, 2024
4c2ead1
Fix triggers
smokestacklightnin Oct 2, 2024
040369b
Fix link
smokestacklightnin Oct 2, 2024
4525700
Only trigger on push to master
smokestacklightnin Oct 2, 2024
aea4660
Fix linting errors
smokestacklightnin Oct 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .github/workflows/cd-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: deploy-docs
on:
workflow_dispatch:
push:
branches:
- 'master'
pull_request:
permissions:
contents: write
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
cache: 'pip'
cache-dependency-path: |
setup.py

- name: Save time for cache for mkdocs
run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV

- name: Caching
uses: actions/cache@v4
with:
key: mkdocs-material-${{ env.cache_id }}
path: .cache
restore-keys: |
mkdocs-material-

- name: Install Dependencies
run: pip install -r requirements-docs.txt

- name: Deploy to GitHub Pages
if: (github.event_name != 'pull_request')
run: |
git config user.name github-actions[bot]
git config user.email 41898282+github-actions[bot]@users.noreply.github.com
mkdocs gh-deploy --force
smokestacklightnin marked this conversation as resolved.
Show resolved Hide resolved

- name: Build docs to check for errors
run: mkdocs build
if: (github.event_name == 'pull_request')
45 changes: 43 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ __pycache__/

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
Expand Down Expand Up @@ -47,8 +48,10 @@ htmlcov/
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
Expand All @@ -58,6 +61,7 @@ coverage.xml
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
Expand All @@ -70,6 +74,7 @@ instance/
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
Expand All @@ -85,12 +90,33 @@ ipython_config.py
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that dont work, or not
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# celery beat schedule file
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py
Expand Down Expand Up @@ -136,3 +162,18 @@ bazel-*

MODULE.bazel
MODULE.bazel.lock
# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

# Bazel build files
bazel-*
26 changes: 14 additions & 12 deletions g3doc/get_started.md → docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ following list provides a non-exhaustive overview of some of the major benefits.
within a range; find previous executions in a context with the same inputs.

See the
[MLMD tutorial](https://www.tensorflow.org/tfx/tutorials/mlmd/mlmd_tutorial) for
[MLMD tutorial](https://tensorflow.github.io/tfx/tutorials/mlmd/mlmd_tutorial/) for
an example that shows you how to use the MLMD API and the metadata store to
retrieve lineage information.

Expand Down Expand Up @@ -484,10 +484,11 @@ the steps together within a single transaction, and the transaction is
rolled-back when an error occurs. The migration script is provided together with
any schema-change commit and verified through testing.

Note: The migration DDLs in MySQL are not transactional. When using MySQL, there
should only be a single connection with the upgrade migration enabled to use the
old database. Take a backup of the database before upgrading to prevent
potential data losses.
!!! Note
The migration DDLs in MySQL are not transactional. When using MySQL, there
should only be a single connection with the upgrade migration enabled to use the
old database. Take a backup of the database before upgrading to prevent
potential data losses.

### Downgrade the database schema

Expand All @@ -509,11 +510,12 @@ metadata_store.downgrade_schema(connection_config,
downgrade_to_schema_version = 0)
```

Note: When downgrading, MLMD prevents data loss as much as possible. However,
newer schema versions might be inherently more expressive and hence a downgrade
can introduce data loss. When using backends that do not support DDL
transactions (e.g., MySQL), the database should be backed up before downgrading
and the downgrade script should be the only MLMD connection to the database.
!!! Note
When downgrading, MLMD prevents data loss as much as possible. However,
newer schema versions might be inherently more expressive and hence a downgrade
can introduce data loss. When using backends that do not support DDL
transactions (e.g., MySQL), the database should be backed up before downgrading
and the downgrade script should be the only MLMD connection to the database.

The list of `schema_version` used in MLMD releases are:

Expand Down Expand Up @@ -555,7 +557,7 @@ ml-metadata (MLMD) | schema_version

The MLMD library has a high-level API that you can readily use with your ML
pipelines. See the
[MLMD API documentation](https://www.tensorflow.org/tfx/ml_metadata/api_docs/python/mlmd)
[MLMD API documentation](api/mlmd/)
for more details.

Check out
Expand All @@ -564,5 +566,5 @@ to learn how to use MLMD declarative nodes filtering capabilities on properties
and 1-hop neighborhood nodes.

Also check out the
[MLMD tutorial](https://www.tensorflow.org/tfx/tutorials/mlmd/mlmd_tutorial) to
[MLMD tutorial](https://tensorflow.github.io/tfx/tutorials/mlmd/mlmd_tutorial/) to
learn how to use MLMD to trace the lineage of your pipeline components.
Binary file removed g3doc/images/mlmd_flow.png
Binary file not shown.
Binary file removed g3doc/images/mlmd_overview.png
Binary file not shown.
118 changes: 118 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
site_name: "ML Metadata"
repo_name: "ML Metadata"
repo_url: https://github.com/google/ml-metadata

theme:
name: material
palette:
# Palette toggle for automatic mode
- media: "(prefers-color-scheme)"
primary: custom
accent: custom
toggle:
icon: material/brightness-auto
name: Switch to light mode

# Palette toggle for light mode
- media: "(prefers-color-scheme: light)"
primary: custom
accent: custom
scheme: default
toggle:
icon: material/brightness-7
name: Switch to dark mode

# Palette toggle for dark mode
- media: "(prefers-color-scheme: dark)"
primary: custom
accent: custom
scheme: slate
toggle:
icon: material/brightness-4
name: Switch to system preference
favicon: images/favicon.png

features:
- content.code.copy
- content.code.select
plugins:
- search
- autorefs
- mkdocstrings:
default_handler: python
handlers:
python:
options:
show_source: true
show_root_heading: true
unwrap_annotated: true
show_symbol_type_toc: true
show_symbol_type_heading: true
merge_init_into_class: true
show_signature_annotations: true
separate_signature: true
signature_crossrefs: true
group_by_category: true
show_category_heading: true
inherited_members: true
show_submodules: true
show_object_full_path: false
show_root_full_path: true
docstring_section_style: "spacy"
summary: true
filters:
- "!^_"
- "^__init__$"
- "^__call__$"
- "!^logger"
- "!_test$"
extensions:
- griffe_inherited_docstrings
import:
- https://docs.python.org/3/objects.inv
- caption:
figure:
ignore_alt: true

markdown_extensions:
- admonition
- attr_list
- toc:
permalink: true
- pymdownx.highlight:
anchor_linenums: true
linenums: false
line_spans: __span
pygments_lang_class: true
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.superfences
- pymdownx.arithmatex:
generic: true
- md_in_html
- pymdownx.emoji:
emoji_index: !!python/name:material.extensions.emoji.twemoji
emoji_generator: !!python/name:material.extensions.emoji.to_svg

extra_css:
- stylesheets/extra.css

extra_javascript:
- javascripts/mathjax.js
- https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js

watch:
- ml_metadata
nav:
- Guide: index.md
- Tutorial: https://tensorflow.github.io/tfx/tutorials/mlmd/mlmd_tutorial/
- API:
- mlmd:
- Overview: api/mlmd/index.md
- mlmd: api/mlmd/mlmd.md
- mlmd.errors:
- Overview: api/mlmd.errors/index.md
- mlmd.errors: api/mlmd.errors/mlmd.errors.md
- mlmd.proto:
- Overview: api/mlmd.proto/index.md
- mlmd.proto: api/mlmd.proto/mlmd.proto.md
35 changes: 21 additions & 14 deletions ml_metadata/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,23 +13,30 @@
# limitations under the License.
"""Init module for ML Metadata."""


# pylint: disable=g-import-not-at-top
try:
from ml_metadata import proto
from ml_metadata import proto

# Import metadata_store API.
from ml_metadata.metadata_store import (
ListOptions,
MetadataStore,
OrderByField,
downgrade_schema,
)
# Import metadata_store API.
from ml_metadata.metadata_store import (
ListOptions,
MetadataStore,
OrderByField,
downgrade_schema,
)

# Import version string.
from ml_metadata.version import __version__
# Import version string.
from ml_metadata.version import __version__

except ImportError as err:
import sys
sys.stderr.write(f'Error importing: {err}')
# pylint: enable=g-import-not-at-top
import sys # pylint: enable=g-import-not-at-top

sys.stderr.write(f"Error importing: {err}")


__all__ = [
"downgrade_schema",
"ListOptions",
"MetadataStore",
"OrderByField",
]
7 changes: 7 additions & 0 deletions ml_metadata/metadata_store/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,10 @@
OrderByField,
downgrade_schema,
)

__all__ = [
"downgrade_schema",
"ListOptions",
"MetadataStore",
"OrderByField",
]
12 changes: 9 additions & 3 deletions ml_metadata/metadata_store/metadata_store.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,13 @@

@enum.unique
class OrderByField(enum.Enum):
"""Defines the available fields to order results in ListOperations."""
"""Defines the available fields to order results in ListOperations.

Attributes:
CREATE_TIME: `<OrderByField.CREATE_TIME: 1>`
ID: `<OrderByField.ID: 3>`
UPDATE_TIME: `<OrderByField.UPDATE_TIME: 2>`
"""

CREATE_TIME = (
metadata_store_pb2.ListOperationOptions.OrderByField.Field.CREATE_TIME)
Expand Down Expand Up @@ -1041,8 +1047,8 @@ def get_artifacts_and_types_by_artifact_ids(

Returns:
Artifacts with matching ids and ArtifactTypes which can be matched by
type_ids from Artifacts. Each ArtifactType contains id, name,
properties and custom_properties fields.
type_ids from Artifacts. Each ArtifactType contains id, name,
properties and custom_properties fields.
"""
del extra_options
request = metadata_store_service_pb2.GetArtifactsByIDRequest(
Expand Down
Loading
Loading