Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containerize the build process and fill in developer documentation #277

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 86 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,94 @@ Note that the while this repo is named `submission-schema`, the generated artifa

## Developer Documentation

<details>
Use the `make` command to generate project artefacts:
### Updating the submission schema

- `make all`: make everything
- ~~`make deploy`: deploys site~~
Here's how you can update the submission schema to use a new version of `nmdc-schema`:

</details>
1. Update the one occurrence of the `nmdc-schema` version number in `project.Makefile`:

```diff
local/nmdc.yaml:
- wget -O $@ https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml
+ wget -O $@ https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml
```

2. Update all the `nmdc-schema` version numbers in `sheets_and_friends/tsv_in/import_slots_regardless.tsv`

```diff
- schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface
+ schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface
```
> That is one of the _many_ occurrences in that file.
>
> **Note:** The above search-and-replacement (of all occurrences in that file) can be done inside Vim via:
> ```vim
> :%s/nmdc-schema\/v11.0.1/nmdc-schema\/v11.1.0/g
> ```

### Building the submission schema

Here's how you can generate the submission schema release artifacts:

#### Container-based process

##### Prerequisites

- Docker is installed on your computer
- You are in the root directory of the repository

##### Procedure

1. Build the container image you will later use to build the submission schema:
```shell
docker build -t submission-schema-builder -f builder.Dockerfile .
```
2. Use that container image to build the submission schema:
```shell
docker run --rm -it -v ${PWD}:/submission-schema submission-schema-builder
```
3. (Optional) Delete the container image:
```shell
docker image rm submission-schema-builder
```

#### Direct process

##### Prerequisites

- `yq` is installed on your computer, such that the following command shows a version number instead of an error message.
```shell
bash -c 'yq --version'
```
If `yq` is not installed, you can install it by running this, assuming you're using macOS:
```shell
brew install yq
```
- `wget` is installed on your computer, such that the following command shows a version number instead of an error message.
```shell
bash -c 'wget --version'
```
If `wget` is not installed, you can install it by running this, assuming you're using macOS:
```shell
brew install wget
```

##### Procedure

1. Install Python dependencies:
```shell
poetry shell
poetry install
```
2. Generate the release artefacts:
```shell
make all
```
3. Commit the changes, using the new `nmdc-schema` version number as the commit message; like this:
```shell
git add .
git commit -m "11.1.0"
```

## Credits

Expand Down
31 changes: 31 additions & 0 deletions builder.Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Introduction: This Dockerfile can be used to build a container image that can, in turn, be used
# to "build" (hence, its name, "builder") the submission schema release artifacts.

# Use Python 3.9 because that's the Python version listed in `pyproject.toml`.
FROM python:3.9

WORKDIR /submission-schema

# Download and install yq.
# Reference: https://github.com/mikefarah/yq#install
RUN wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq && \
chmod +x /usr/bin/yq

# Install Poetry, a package manager for Python (an alternative to pip).
RUN pip install poetry

# Copy the entire repository into the container image.
#
# Note: Copying _only_ the dependency lists (i.e. `pyproject.toml` and `poetry.lock`) and then
# mounting the entire repository as a volume in the container image at `docker run` time
# results in `make all` failing due to the absence of an `nmdc_submission_schema` package.
# I haven't yet figured out why that happens, but it doesn't happen if I have copied the
# entire repository into the container image here at build time—so, I do the latter.
#
ADD . /submission-schema

# Install the project's Python dependencies.
RUN poetry install --no-interaction

# Run the command that builds the submission schema release artifacts.
CMD ["make", "all"]
Loading