Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created a new page in Integrations section for Pydantic plugin in Flyte documentation #1757

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions examples/pydantic_plugin/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM python:3.11-slim-buster
LABEL org.opencontainers.image.source=https://github.com/flyteorg/flytesnacks

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

# Install Python dependencies
COPY requirements.in /root
RUN pip install -r /root/requirements.in

# Copy the actual code
COPY . /root/

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
28 changes: 28 additions & 0 deletions examples/pydantic_plugin/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
(pydantic_integration)=

# Pydantic

```{eval-rst}
.. tags:: Pydantic, Flytekit
```
[Pydantic](https://pydantic.dev/) is a data validation and settings management library that leverages Python type annotations to enforce type hints at runtime. It provides user-friendly errors when data is invalid, making it easier to ensure data integrity.

The Flytekit Pydantic plugin adds type support for Pydantic models, enabling seamless integration of data validation and settings management within Flyte tasks and workflows. This documentation demonstrates how to integrate Pydantic with Flytekit using the Flytekit Pydantic plugin.
sumana-2705 marked this conversation as resolved.
Show resolved Hide resolved

## Installation

To install the Flytekit Pydantic plugin, run the following command:

```
pip install flytekitplugins-pydantic
```
## Example usage

For a usage example, see {doc}`Pydantic example usage <pydantic_integration_example>`.

```{toctree}
:maxdepth: -1
:hidden:

example
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# %% [markdown]
# (pydantic_integration_example)=
#
# # Pydantic Integration Example
#
# Pydantic is a data validation and settings management library for Python, enabling the creation of data models with type annotations.
#
# Flyte leverages Pydantic for robust input validation and serialization, ensuring that task inputs are correctly structured.
Comment on lines +6 to +8
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems repetitive. also, can you come up with a better example? something that solves a use case. needn't be complex, but must solve a problem. does that make sense?


# %%
from typing import List

from flytekit import task, workflow
from flytekit.types.file import FlyteFile
from pydantic.v1 import BaseModel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pydantic v2 should work: flyteorg/flytekit#2217

could you run this workflow on Union serverless/flyte sandbox and share with us the result of the execution?

Copy link
Contributor Author

@sumana-2705 sumana-2705 Oct 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@samhita-alla I am currently experiencing difficulties while executing my code on the Union serverless/Flyte sandbox. Specifically, I have encountered errors related to importing the BaseModel from the Pydantic library.

  1. When attempting to execute the following code:
    from pydantic.v1 import BaseModel

I got this error:
Screenshot 2024-10-26 114446

  1. When I tried this:
    from pydantic import BaseModel

I got this error:
Screenshot 2024-10-26 114642

I am not able to understand how to avoid these errors. How to resolve these errors?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

@sumana-2705 sumana-2705 Oct 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.



# %% [markdown]
# Let's first define a Pydantic model for training configuration.
# %%
class TrainConfig(BaseModel):
lr: float = 1e-3 # Learning rate
batch_size: int = 32 # Batch size for training
files: List[FlyteFile] # List of file inputs for training


# %% [markdown]
# Next, we use the Pydantic model in a Flyte task to train a model.
# %%
@task
def train(cfg: TrainConfig):
print(f"Training with learning rate: {cfg.lr} and batch size: {cfg.batch_size}")
for file in cfg.files:
print(f"Processing file: {file}")


# %% [markdown]
# Now we define a Flyte workflow that utilizes the training task.
# %%
@workflow
def training_workflow(lr: float = 1e-3, batch_size: int = 32, files: List[FlyteFile] = []):
cfg = TrainConfig(lr=lr, batch_size=batch_size, files=files)
train(cfg=cfg)


# %% [markdown]
# Finally, we execute the workflow with sample parameters.
# %%
if __name__ == "__main__":
training_workflow(lr=1e-3, batch_size=32, files=[FlyteFile(path="path/to/your/file")])
1 change: 1 addition & 0 deletions examples/pydantic_plugin/requirements.in
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
flytekitplugins-pydantic
Loading