Pydantic Forms v2

[pboers1988]

Background

Pydantic forms is a form library that facilitates a backend, which is responsible for defining the form fields and validating the submitted information + a frontend that is able to render a collection of form fields. It was an integral component of the WFO, but when SURF began developing CIM, the backend elements were extracted into a separate repository. This separate project, named Pydantic Forms, was subsequently released as a PyPi package.

The frontend relies on JSON scheme and the Uniforms library to render the form fields. In the SURF Orchestrator V1 GUI there was a lot of (SURF)specific JS to handle fetching data, interacting with the App Context and to implement some complex fields like the port selectors. This makes it very hard to extract the functionality into a NPMJS package that can be used in other React applications as the ways of fetching data and handling state are too opinionated.

SURF Features

Before we implemented the dynamic forms we already had forms that we had to keep in sync between backend and frontend; which had a mix of validation problems and strategies. Most of the current features “evolved” from our use cases.

• Labels and help texts are served from the backend and can be multilingual
• Prefilled form rely on specifying defaults on the backend
• Validation is done 100% in backend; you can always submit the form. Validation errors are displayed in the GUI on field or form level.
• Validation use Pydantic logic: and can also execute python logic to validate complex backend logic (like verifying if a value already exists in an external system).
• Basic python types are mapped to “logical” form field types: str, int, datetime, date can be used in a pydantic model to signal the frontend that we need some text, a number, a date and time or only a date. Form fields are determined by their python type.
• Complex form fields can be defined by adding custom types: in backend and frontend.
• Backend validators are covered by unit tests
• Frontend (business)logic is partially covered by unit tests
• Validators and form fields can access the data of other form on the same FormPage
• A form wizard, that serves multiple FormPage as a sequence, is supported. These form pages can depend on earlier choices.
• Around 8 simple Form Fields and 12 complexer variants.
• There are Forms that are part of a WFO Process and Forms that handle cases outside of the WFO context (e.g. CIM)
• Error messages are in one language (EN/Python ValueErrors)
• All form fields can be used for a single value, but also for a list of values.

Community Features

There are at least 2 places where the forms are used in other web-applications. As the SURF frontend relies on uniforms with some hooks and uniforms doesn’t have any real progress in the newer version (mostly due to commercial plans), within one of the other projects a decision was made to rewrite the form on the client side removing the uniforms dependency in favor of React Hooks Form and ZOD. This form implementation was shared with SURF and could be the basis of the new Forms functionality in the V2 GUI of the orchestrator.

• Hybrid validation. The frontend can validate most simple fields, like a string that conforms to a regex or a required field, before submit.
• 3 ways of loading data: via default values on the form, via backend when retrieving the form labels or from the JS side by communicating with the DynamicForm ContextProvider.
• Debug mode: that makes it easy to debug the new parser.
• An alternative way to implement a form wizard by utilizing local storage: to store the state of each FormPage. Users can continue their forms where they left it. The form state could also be stored in the backend so users can switch to another browser and continue.
• Future
• The Forms are a crucial part of the WFO, but they also shine in other places. Whilst working on the V2 WFO GUI, we focused on getting the forms working as is: most of the functionality was copied from the V1 with only some small changes regarding how data is fetched/handled. Now a traject will be started to refactor the forms whilst also making it easier to use in other places.

Goals

• Release an un-opinionated NPM package that can be used to implement forms in your own GUI based on pydantic forms, which can handle current use cases and is flexible enough so it can be extended. Still unclear if that will mostly contain an Open Sourced parser or a complete DynamicForms Provider.
• Release a EUI Formfields package (as a reference): hoping on the community to provide packages for other React Component Libraries.
• Refactor the forms for WFO GUI V2; without uniforms.
• Document some of the advanced use-cases so users know how to implement their own custom fields (autocomplete, multi choice, file-upload, etc.)
• Multilingual labels and error messages

Challenges

• The Forms need a way to fetch data and to communicate with the pydantic-form powered backend. All current implementations use a hardcoded api client. Currently: Axios, Fetch, SWR, Redux toolkit based solutions are used. As the form logic relies on some HTTP Errors (404, 422, 510 status codes) to be handled by the form itself; this leads to some complexer setup because most API clients in JS projects, by default, will handle these error codes before the data reaches the component. So data fetching should be pluggable so it doesn’t dictate an API client.
• The React-Hooks implementation uses a couple of context providers to handle state, that might clash with other state management solutions like Redux Toolkit.
• The WFO relies on “default values”, in the pydantic forms code on the backend, for showing forms with pre-populated data. This has the side effect that these fields are not not required any more. When using the forms in a CMS context you often have a “new” and “edit” form. Fetching and injecting the data for the community form is handled by the DynamicForm provider. These strategies should be formalized.
• Multilingual errors implies translated pydantic errors, which implies rather a lot of translate work. The current workaround is to write validators with custom messages for “most” errors that could occur. Partially relying on client side validation hides a lot of the simpler backend validation errors.
• The WFO uses EUI Form fields, the variant in the community rely on Custom React components. It would be nice if this could be pluggable. E.g. For each field you have to implement some logic to tie the Pydantic form logic to a Formfield, covering the label and error messages. The NPM package could be split into a “logic only” + “EUI form field” package to accommodate this. The community will likely provide packages for the other frameworks like Material UI, Tailwind etc.
• The backend and its form fields are still using a lot of “uniforms” in the JSON schemes.
• Some new features are hard to implement without breaking functionality for a lot of users, an example: a conditional field would be nice: so when a users inputs a certain value or checks a checkbox; other form fields should appear. This can be done by adding extra “uniform” properties to the field that triggers it, or the fields that get triggered. The last option will make it possible to have some sort of generic support for conditionals; but it might break all form implementations. The alternative is to let the user add a lot of custom form fields: A custom trigger field and add also custom fields for all FormFields that could be triggered; leading to a lot of duplicate/boilerplate code.

[acidjunk]

Discussed this last wednesday:

• FastAPI introduced "Form" support; this will be investigated
• Rene will investigate multilingual stuff
• After that: a new meeting will be scheduled to see if we can create a roadmap