Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-46158: Add schema comparison workflow #277

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions .github/workflows/compare.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
name: Compare Schemas for Changes

on:
push:
branches:
- main
tags:
- "*"
pull_request:

jobs:
compare:
runs-on: ubuntu-latest

steps:
- name: Check out repo
uses: actions/checkout@v4
with:
path: current-ref

- name: Fetch origin main
working-directory: current-ref
run: git fetch origin main

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: "pip"

- name: Install dependencies
working-directory: current-ref
run: |
python -m pip install --upgrade pip uv
uv pip install --system -r requirements.txt

- name: Set relative path to schema directory
run: |
echo "SCHEMA_DIR=python/lsst/sdm_schemas/schemas" >> $GITHUB_ENV

- name: Get list of changed YAML files
working-directory: current-ref
run: |
CHANGED_FILES=$(git diff --name-only origin/main..HEAD -- ${{ env.SCHEMA_DIR }} | sed "s|^${{ env.SCHEMA_DIR }}/||")
echo "Changed YAML files: $CHANGED_FILES"
echo "CHANGED_FILES=$CHANGED_FILES" >> $GITHUB_ENV

- name: Check out main branch
uses: actions/checkout@v4
with:
ref: main
path: main-branch

- name: Run schema comparison with deepdiff
run: |
for file in ${{ env.CHANGED_FILES }}; do
echo "Comparing $file:"
MAIN_FILE=main-branch/${{ env.SCHEMA_DIR }}/$file
CURRENT_FILE=current-ref/${{ env.SCHEMA_DIR }}/$file
felis --log-level ERROR diff -c deepdiff $MAIN_FILE $CURRENT_FILE
done

- name: Run schema comparison with alembic
run: |
for file in ${{ env.CHANGED_FILES }}; do
echo "Comparing $file:"
MAIN_FILE=main-branch/${{ env.SCHEMA_DIR }}/$file
CURRENT_FILE=current-ref/${{ env.SCHEMA_DIR }}/$file
felis --log-level ERROR diff -c alembic $MAIN_FILE $CURRENT_FILE
done
7 changes: 6 additions & 1 deletion python/lsst/sdm_schemas/schemas/dp02_dc2.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: dp02_dc2_catalogs
"@id": "#dp02_dc2_catalogs"
description: "Data Preview 0.2 contains the image and catalog products of the Rubin Science
Pipelines v23 processing of the DESC Data Challenge 2 simulation, which covered 300 square
degrees of the wide-fast-deep LSST survey region over 5 years."
degrees of the wide-fast-deep LSST survey region over 5 years (dummy change)."
tables:
- name: Object
"@id": "#Object"
Expand Down Expand Up @@ -8053,3 +8053,8 @@ tables:
tap:column_index: 999
tap:principal: 1
description: 1 for galaxies, 2 for stars, and 3 for SNe
- name: dummy_column
'@id': '#TruthSummary.dummy_column'
datatype: string
length: 256
description: Dummy column for schema comparison testing
2 changes: 1 addition & 1 deletion requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
lsst-felis @ git+http://github.com/lsst/felis.git@main
lsst-felis @ git+http://github.com/lsst/felis.git@tickets/DM-46130

click
pyyaml
Expand Down
Loading