Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(sage-monorepo): create Nx plugin to centralize task definitions #2884

Conversation

tschaffter
Copy link
Member

@tschaffter tschaffter commented Oct 4, 2024

Changelog

  • Create the project sage-monorepo/nx-sage-monorepo
  • Create an Nx plugin that dynamically adds the target build-image to projects that meet the conditions
  • Remove the task build-image from the project.json files
  • Add the configurations local and ci to the task build-image of the OC projects
  • Enable and configure the Nx plugin in nx.json
  • Configure ESLint to lint JSON files with jsonc-eslint-parser (suggested when generating the Nx plugin)
  • Remove Nx patch initially introduced to fix nx format

Notes

  • Centralizing the task definition in a plugin as defined in the first part of the article series will make this task available to all projects, which is not what we want. For example, adding the task build-image to an Angular library does not make sense and even generate error when attempt to run nx run-many --target=build-image --projects=openchallenges-*

References

Preview

Generate the Nx plugin

nx g @nx/plugin:plugin sage-monorepo \
  --directory libs/sage-monorepo/nx-sage-monorepo \
  --projectNameAndRootFormat as-provided \
  --importPath=@sagebionetworks/nx-sage-monorepo \
  --dry-run

How do I know which task is available for a given project?

nx show project myreactapp --web

This will open the page shown below in your browser.

image

Docker images build locally

The same images are built compared to before the creation of this PR. What has changed is that the definition of the task build-image has been centralized in a custom Nx plugin.

$ docker images
REPOSITORY                                                          TAG           IMAGE ID       CREATED          SIZE
ghcr.io/sage-bionetworks/openchallenges-app                         local         67445917dfc3   9 minutes ago    198MB
ghcr.io/sage-bionetworks/openchallenges-app                         sha-5932f25   67445917dfc3   9 minutes ago    198MB
ghcr.io/sage-bionetworks/openchallenges-challenge-service           local         4c2bb38078cc   9 minutes ago    353MB
ghcr.io/sage-bionetworks/openchallenges-challenge-service           sha-5932f25   4c2bb38078cc   9 minutes ago    353MB
ghcr.io/sage-bionetworks/openchallenges-edam-etl                    local         673a4869a08a   9 minutes ago    375MB
ghcr.io/sage-bionetworks/openchallenges-edam-etl                    sha-5932f25   673a4869a08a   9 minutes ago    375MB
ghcr.io/sage-bionetworks/openchallenges-api-gateway                 local         965a2646cb4c   9 minutes ago    320MB
ghcr.io/sage-bionetworks/openchallenges-api-gateway                 sha-5932f25   965a2646cb4c   9 minutes ago    320MB
ghcr.io/sage-bionetworks/openchallenges-elasticsearch               local         1ea8c7949100   10 minutes ago   622MB
ghcr.io/sage-bionetworks/openchallenges-elasticsearch               sha-5932f25   1ea8c7949100   10 minutes ago   622MB
ghcr.io/sage-bionetworks/openchallenges-mariadb                     local         9ffbe47f51b3   10 minutes ago   384MB
ghcr.io/sage-bionetworks/openchallenges-mariadb                     sha-5932f25   9ffbe47f51b3   10 minutes ago   384MB
ghcr.io/sage-bionetworks/openchallenges-config-server               local         d9ce91073110   10 minutes ago   308MB
ghcr.io/sage-bionetworks/openchallenges-config-server               sha-5932f25   d9ce91073110   10 minutes ago   308MB
ghcr.io/sage-bionetworks/openchallenges-organization-service        local         b2529554f4c7   10 minutes ago   351MB
ghcr.io/sage-bionetworks/openchallenges-organization-service        sha-5932f25   b2529554f4c7   10 minutes ago   351MB
ghcr.io/sage-bionetworks/openchallenges-apex                        local         7759418b255b   10 minutes ago   42.7MB
ghcr.io/sage-bionetworks/openchallenges-apex                        sha-5932f25   7759418b255b   10 minutes ago   42.7MB
ghcr.io/sage-bionetworks/openchallenges-image-service               local         881c9f17d693   11 minutes ago   322MB
ghcr.io/sage-bionetworks/openchallenges-image-service               sha-5932f25   881c9f17d693   11 minutes ago   322MB
ghcr.io/sage-bionetworks/openchallenges-service-registry            local         25ecc9660687   11 minutes ago   322MB
ghcr.io/sage-bionetworks/openchallenges-service-registry            sha-5932f25   25ecc9660687   11 minutes ago   322MB
ghcr.io/sage-bionetworks/openchallenges-api-docs                    local         efbfb4b327e2   13 minutes ago   57.1MB
ghcr.io/sage-bionetworks/openchallenges-api-docs                    sha-5932f25   efbfb4b327e2   13 minutes ago   57.1MB
ghcr.io/sage-bionetworks/openchallenges-thumbor                     local         912b23687292   7 days ago       877MB
ghcr.io/sage-bionetworks/openchallenges-thumbor                     sha-5932f25   912b23687292   7 days ago       877MB
paketobuildpacks/run                                                base-cnb      f2e5000af0cb   15 months ago    87.1MB
ghcr.io/sage-bionetworks/openchallenges-zipkin                      local         ad648bbe31df   16 months ago    157MB
ghcr.io/sage-bionetworks/openchallenges-zipkin                      sha-5932f25   ad648bbe31df   16 months ago    157MB
ghcr.io/sage-bionetworks/openchallenges-grafana                     local         c89968b5071d   19 months ago    329MB
ghcr.io/sage-bionetworks/openchallenges-grafana                     sha-5932f25   c89968b5071d   19 months ago    329MB
ghcr.io/sage-bionetworks/openchallenges-prometheus                  local         800ad2ab7df4   20 months ago    232MB
ghcr.io/sage-bionetworks/openchallenges-prometheus                  sha-5932f25   800ad2ab7df4   20 months ago    232MB
ghcr.io/sage-bionetworks/openchallenges-kafka                       local         e1261d222979   22 months ago    794MB
ghcr.io/sage-bionetworks/openchallenges-kafka                       sha-5932f25   e1261d222979   22 months ago    794MB
ghcr.io/sage-bionetworks/openchallenges-mysqld-exporter             local         63eebb258351   2 years ago      17.8MB
ghcr.io/sage-bionetworks/openchallenges-mysqld-exporter             sha-5932f25   63eebb258351   2 years ago      17.8MB
ghcr.io/sage-bionetworks/openchallenges-organization-service-base   local         e80ad3a7fab0   44 years ago     341MB
ghcr.io/sage-bionetworks/openchallenges-config-server-base          local         b20a105562e7   44 years ago     299MB
ghcr.io/sage-bionetworks/openchallenges-challenge-service-base      local         71f9982158b9   44 years ago     343MB
ghcr.io/sage-bionetworks/openchallenges-api-gateway-base            local         6f833d5e8824   44 years ago     310MB
ghcr.io/sage-bionetworks/openchallenges-image-service-base          local         b5a3b40a55b4   44 years ago     313MB
ghcr.io/sage-bionetworks/openchallenges-service-registry-base       local         3a72e10470a3   44 years ago     312MB
paketobuildpacks/builder                                            base          050ed48532b2   44 years ago     1.31GB

@tschaffter
Copy link
Member Author

tschaffter commented Oct 9, 2024

The Nx environment fails to initialize

Right after opening VS Code. This may also be way the CI workflow currently fail.

image

CI error

[Nxls] - 2024-10-09T03:49:41.424Z - LoadPluginError: Could not load plugin @sagebionetworks/nx-sage-monorepo
    at loadNxPluginAsync (/workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/project-graph/plugins/loader.js:208:15)
    at loadNxPlugin (/workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/project-graph/plugins/loader.js:182:9)
    at load (/workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker.js:25:65)
    at consumeMessage (/workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/project-graph/plugins/isolation/messaging.js:38:32)
    at /workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/project-graph/plugins/isolation/plugin-worker.js:21:47
    at Socket.<anonymous> (/workspaces/sage-monorepo/node_modules/.pnpm/[email protected]_patch_hash=vxuy2ap6ceuowh22gcuahf5n2m_@[email protected]_@[email protected]_@sw_usm4h2vcgpymoofs5hqaew7zke/node_modules/nx/src/utils/consume-messages-from-socket.js:13:17)
    at Socket.emit (node:events:519:28)
    at addChunk (node:internal/streams/readable:559:12)
    at readableAddChunkPushByteMode (node:internal/streams/readable:510:3)
    at Readable.push (node:internal/streams/readable:390:5)

Related issues

@tschaffter
Copy link
Member Author

Replaced by #2890

@tschaffter tschaffter closed this Oct 11, 2024
@tschaffter tschaffter deleted the sage-monorepo/centralize-build-image branch October 11, 2024 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant