Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add SQL adapter #779

Draft
wants to merge 16 commits into
base: main
Choose a base branch
from
Draft

add SQL adapter #779

wants to merge 16 commits into from

Conversation

skarakuzu
Copy link
Contributor

preliminary start of sql adapter. to be continued ...

Checklist

  • Add a Changelog entry
  • Add the ticket number which this PR closes to the comment section

@@ -44,6 +44,9 @@ tiled = "tiled.commandline.main:main"

# This is the union of all optional dependencies.
all = [
"adbc_driver_manager",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is used when tiled is installed like pip install "tiled[all]". These three should also be added to the section server, below, so that they are included when tiled is installed like pip install "tiled[server]" # server only.

@danielballan
Copy link
Member

Lifecycle:

  1. Client declares that it wants to create a new tabular dataset, via a request POST /api/v1/metadata/my_table.
  2. In "catalog" SQL database, the server adds a row to the nodes table with any metadata about this table. This is how the new table is connected to any overall dataset, like Bluesky scan and its Scan ID.
  3. Also in the "catalog" SQL database, the server adds a row each to the data_sources table and the assets table. Together, they describe how to locate where the new data will be saved. The Asset part is very locked down. It has room for the URI of the tabular SQL database: postgresql://... and some boilerplate. The DataSource has a freeform area called parameters, which can fit any JSON. We can use this to put in dataset-specific details, like the name of the SQL table (table_name)---derived from the Arrow schema in this case---and a means of selecting the rows of interest for this new dataset (dataset_id).
  4. When data is written or read, a SQLAdapter object is instantiated inside the server. It is passed information extracted from this DataSource and Asset. So, it can know the table_name and the dataset_id.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants