diff --git a/.gitignore b/.gitignore index 1be293e36f..9acf8eb4ed 100644 --- a/.gitignore +++ b/.gitignore @@ -51,3 +51,5 @@ static/**/.garden-version #debug files debug-info-* + +/yarn.lock \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1423393758..a6569a3974 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,296 +1,12 @@ -# Developing the Garden CLI framework +# Contributing -## Contributing guidelines +> [!NOTE] +> If you love Garden, please ★ star this repository to show your support :green_heart:. Looking for support? Join our [Discord](https://go.garden.io/discord). -We heartily welcome any form of contribution to the project, including issue reports, feature requests, -discussion, pull requests and any type of feedback. We request that all contributors -adhere to the [Contributor Covenant](CODE_OF_CONDUCT.md) and work with us to make the collaboration and -community productive and fun for everyone :) +:tada: Thanks for taking an interest in contributing to Garden and Garden's docs. We appreciate your effort and passion! :clap: -### How to report an issue +This page is a redirect to our main contributing guide. To learn more about how you can contribute, please follow the link below: :point_down: -If you have found a bug, or want to submit a feature request, or suggest any other change, please create -a [new issue](https://github.com/garden-io/garden/issues/new/choose). +[Click here to go to the docs!](./contributing/README.md) :rocket: -If you report a bug, please describe the steps to reproduce it. You can share the complete Garden configuration of your -project (without any code included) with us by running the `garden get debug-info` command. It will produce a single archive -that matches the directory structure of your project, but contains only your Garden configuration files. This can be very helpful for us to -reproduce and fix the issue. - -## Project structure - -The project code is composed of several components, most of which are written in TypeScript. There's also a number of supporting scripts, documents, examples etc. Here is an overview of the top-level project folders: - -| Name | Description | -| ---- | ----------- | -| `bin` | Executable commands, to use for development. _Note that you need to build the project before these work._ | -| `cli` | The Garden CLI package, which composes code from different packages into the final CLI executable. | -| `core` | The bulk of the Garden code and tests live here. | -| `docs` | Markdown documentation, which is used to generate [docs.garden.io](https://docs.garden.io). _Note that the reference docs are auto-generated, and should not be edited by hand!._ | -| `examples` | Various Garden example projects. | -| `images` | Supporting container images, used by e.g. the `kubernetes` provider. | -| `plugins` | Plugins that are bundled with Garden. We are in the process of moving plugins out of `core` and into separate packages here. | -| `scripts` | Various scripts for builds, releases and development. | -| `sdk` | The `@garden-io/sdk` package, used for Garden plugin development. | -| `secrets` | Encrypted files, used for integ/e2e testing. Only accessible to Garden maintainers. | -| `static` | Static files that are bundled with the CLI. | -| `support` | Supporting files for development, builds, releases etc. | - -## Contributor docs - -In addition to the [public-facing documentation](http://docs.garden.io), we've written a few docs for contributors. These are intended to give an overview of some of the more complex parts of Garden. - -For more, check out the [README in the contributor-docs directory](./contributor-docs/README.md). - -## Setting up your development environment - -### Step 1: Install Docker and Kubernetes - -Please refer to our [installation docs](./docs/getting-started/installation.md) for instructions on how to install Docker and Kubernetes for different platforms. - -### Step 2: Clone the repo - -```sh -git clone https://github.com/garden-io/garden.git -``` - -### Step 3: Install dependencies - -#### OSX - -For Mac we have a script that installs all required dependencies. - -If you haven't already, please [install Homebrew](https://docs.brew.sh/Installation). Then run: - -```sh -./scripts/install-osx-dependencies.sh -``` - -#### Windows / Linux - -Other platforms need to roll their own for now (contributions welcome!). Please have a look at the script for OSX to see what's installed. - -If you have [LinuxBrew](https://docs.brew.sh/Homebrew-on-Linux) installed, [./scripts/install-osx-dependencies.sh](./scripts/install-osx-dependencies.sh) should work if you run it, although you will have to ensure that you've added NPM to your PATH via `.bashrc` `.zshrc` or other shell run command script. - -#### asdf - -If you are an [asdf](https://asdf-vm.com/) user, running `./scripts/install-asdf-dependencies.sh` in order to automatically install the correct plugins and versions as defined in `.tool-versions`. - -### Step 4: Bootstrap project - -Install Node modules for the root package, and `core` package: - -```sh -npm install # To install root dependencies -npm run bootstrap # To bootstrap packages -``` - -from the root directory - -## Developing Garden - -### Initial build - -Before running Garden for the first time, you need to do an initial build by running - -```sh -npm run build -``` - -from the root directory. - -### Developing - -To develop the CLI, run the `dev` command in your console: - -```sh -npm run dev -``` - -This will link it to your global `node_modules` folder, and then watch for -changes and auto-rebuild as you code. You can then run the `garden` command as normal. - -Also, you might like to add a couple of shorthands: - -```sh -alias g='garden' -alias k='kubectl' -``` - -### Formatting - -We use [Prettier](https://prettier.io) for automated formatting. We highly recommend installing the appropriate plugin for your editor to automate formatting as you work on your code. You can also run `npm run fix-format` to fix formatting across the codebase. - -### Debugging - -To enable setting a breakpoint in the code, run the CLI with the `bin/garden-debug` binary, which adds the `--inspect` flag. Developers might find it useful to alias this: - -```sh -alias gd='/path/to/garden/bin/garden-debug' -``` - -You can e.g. use the Chrome DevTools to inspect the code at the breakpoint: - -1. Add a `debugger` statement somewhere in the code. -2. Navigate to [chrome://inspect/](chrome://inspect/) in your Chrome browser. -3. Click the **Open dedicated DevTools for Node** link. -4. Run a CLI command that hits the breakpoint, e.g.: - -```sh -/path/to/garden/bin/garden-debug deploy # or gd deploy, if you've set the alias -``` - -You should now be able to inspect the code at run time in the **Console** tab of the DevTools window. - -### Release binaries and Docker containers - -You can build the release binaries using the command - -``` -npm run dist -``` - -You can then find the release binaries and archives under `dist/`. - -We release a number of Docker containers on [Docker Hub](https://hub.docker.com/u/gardendev). - -The Docker containers meant to be used directly by the general public are defined in `support/docker-bake.hcl`. - -When making changes to the `Dockerfile` definitions in `support/` it is helpful to build the containers on your local machine. - -For that, first run `npm run dist`, and then run `docker buildx bake` like so: - -``` -MAJOR_VERSION=0 MINOR_VERSION=13 PATCH_VERSION=0 CODENAME=bonsai \ - docker buildx bake -f support/docker-bake.hcl all -``` - -The environment variables will influence the tags that `buildx bake` will create on your local machine (e.g. stable release tags, prerelease tags, version number, etc.). - -To run the tests on your local machine, first run `npm run dist` (if not already done so), and then run -``` -bash support/docker-bake-test.sh -``` - -### Tests - -Unit tests are run using `mocha` via `npm run test` from the directory of the package you want to test. To run a specific test, you can grep the test description with the `-g` flag.: - -```sh -cd core -npm run test # run all unit tests -npm run test -- -g "taskGraph" # run only tests with descriptions matching "taskGraph" -``` - -#### ARM64 compatibility -On ARM64 platforms (like Mac machines with M1 chips) the `npm run test` command may fail with the following error: -```sh -FATAL ERROR: wasm code commit Allocation failed - process out of memory -``` -In order to fix it, the terminal must be running in the **Rosetta** mode, the detailed instructions can be found in -[this SO answer](https://stackoverflow.com/a/67813764/2753863). - -Integration tests are run with: - -```sh -npm run integ-local -``` - -End-to-end tests are run with: - -```sh -npm run e2e -``` - -You can also run the end-to-end tests for a specific example project using: - -```sh -npm run e2e-project -- --project= -``` - -End-to-end tests are run in CI by using Garden itself to test the project defined in `./core/test/e2e/garden.yml`. Cf. the appropriate job in `circleci/config.yml` for details. - -### Commit messages - -We follow the [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0-beta.4/#specification) and automatically validate the formatting of our commit messages. In particular, the **type** of the commit header must be one of the following: - -* **chore**: Changes to the build process or auxiliary tools and libraries such as documentation - generation. -* **ci**: Changes to the CI configuration. -* **docs**: Documentation only changes. -* **feat**: A new feature. -* **fix**: A bug fix. -* **improvement**: Changes that improve a current implementation without adding a new feature or fixing a bug. -* **perf**: A code change that improves performance. -* **refactor**: A code change that neither fixes a bug nor adds a feature. -* **revert**: A commit that reverts a previous commit. It should begin with `revert: `, followed by the header of the reverted commit. In the body it should say: `This reverts commit .`, where the hash is the SHA of the commit being reverted. -* **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing - semi-colons, etc). -* **test**: Adding missing or correcting existing tests. - -When generating the changelog, we only include the following types: **feat**, **fix**, **refactor**, **improvement**, and **perf**. This means that any changes that the user should be aware of, should have one of these types. - -### Checks - -We have scripts for checking licenses, docs, linting and more. These can all be run with a single command: - -```sh -npm run check-all -``` - -If the checks fail because of bad formatting, run: - -```sh -npm run fix-format -``` - -If the checks fail because of missing docs, run: - -```sh -npm run generate-docs -``` - -### Pre-push hook - -Before pushing, we automatically run the `check-pre-push` script (which runs the scripts in `check-all`, except for -`check-docs`), as well as unit tests. To skip these, run push with the `--no-verify` flag: - -```sh -git push origin --no-verify -``` - -### Environment Variables - -You should set the following environment variables when developing on Garden: - -```sh -GARDEN_DISABLE_ANALYTICS=true -GARDEN_DISABLE_VERSION_CHECK=true -ANALYTICS_DEV=true -``` - -## CI - -We use [Circle CI](https://circleci.com) for integration and end-to-end testing. The configuration is in `.circleci/config.yml`. - -## License/copyright headers - -Every source file must include the contents of `support/license-header.txt` at the top. This is -automatically checked during CI. Since it's defined as an eslint rule, you can run the check with `npm run lint`. - -## Release process - -### Packages - -Our release process generates the following packages: - -* An executable for OSX, Linux, and Windows, generated by [Pkg](https://github.com/vercel/pkg) and hosted on our [Github page](https://github.com/garden-io/garden/releases). -* A [Homebrew](https://brew.sh/) package for OSX users. - -### Process - -Check out our [release process guide](./RELEASE_PROCESS.md) for more details. - -## Changelog - -We keep a changelog under [CHANGELOG.md](./CHANGELOG.md) that gets updated on every release. For pre-releases, we include every pre-release tag in that release cycle in the changelog. So if we're releasing, say, `0.12.6-3`, the changelog will include entries for `0.12.6-0`, `0.12.6-1`, `0.12.6-2`, assuming those tags exist. Once we make a proper release, we remove the pre-release tags so that the changelog only shows changes between `0.12.5` and `0.12.6`. A changelog with the pre-releases is of course always available in our Git history. +We look forward to your valuable contributions! :heart: diff --git a/contributor-docs/README.md b/contributor-docs/README.md deleted file mode 100644 index 129fd60cad..0000000000 --- a/contributor-docs/README.md +++ /dev/null @@ -1,11 +0,0 @@ -## Contributor Docs - -This directory contains docs on Garden's internals. These are intended to be useful for contributors. - -While much of Garden's implementation can be understood by reading individual functions and classes, a big-picture overview is very useful when dealing with certain subsystems. Examples of this include the graph execution and config resolution flows. - -We'll add more guides to this directory as they're written—please don't hesitate to request more docs or ask questions on our [community Discord channel](https://go.garden.io/discord)! - -## Index -* [Graph execution](./graph-execution.md): Explains the steps involved when `GraphSolver` (solver) processes a set of tasks in dependency order. -* [Config resolution](./config-resolution.md): Explains the high-level steps that Garden takes to go from config files on disk to a fully resolved project (with all modules, actions and workflows resolved with no template strings remaining). diff --git a/docs/README.md b/docs/README.md index f093679788..f330ae4873 100644 --- a/docs/README.md +++ b/docs/README.md @@ -193,3 +193,12 @@ * [FAQ](./misc/faq.md) * [Troubleshooting](./misc/troubleshooting.md) * [Telemetry](./misc/telemetry.md) + +## 🌳 Contributing to Garden + +* [Contributor Covenant Code of Conduct](./contributing/CODE_OF_CONDUCT.md) +* [Contributing to the Docs](./contributing/contributing-docs.md) +* [Setting up your developer environment](./contributing/garden-dev-env-setup.md) +* [Developing Garden](./contributing/developing-garden.md) +* [Config resolution](./contributing/config-resolution.md) +* [Graph execution](./contributing/graph-execution.md) diff --git a/CODE_OF_CONDUCT.md b/docs/contributing/CODE_OF_CONDUCT.md similarity index 93% rename from CODE_OF_CONDUCT.md rename to docs/contributing/CODE_OF_CONDUCT.md index 323c86b4c7..1e8cd0a258 100644 --- a/CODE_OF_CONDUCT.md +++ b/docs/contributing/CODE_OF_CONDUCT.md @@ -1,4 +1,7 @@ -# Contributor Covenant Code of Conduct +--- +title: Contributor Covenant Code of Conduct +order: 2 +--- ## Our Pledge @@ -55,7 +58,7 @@ further defined and clarified by project maintainers. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be -reported by contacting the project team at admin@garden.io. All +reported by contacting the project team at . All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. @@ -68,7 +71,6 @@ members of the project's leadership. ## Attribution This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, -available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html +available at [homepage]: https://www.contributor-covenant.org - diff --git a/docs/contributing/README.md b/docs/contributing/README.md new file mode 100644 index 0000000000..424f59be78 --- /dev/null +++ b/docs/contributing/README.md @@ -0,0 +1,135 @@ +--- +title: Contributing to Garden +order: 99 +--- + +{% hint style="info" %} + +If you love Garden, please ★ star this repository to show your support :green_heart:. Looking for support? Join our [Discord](https://go.garden.io/discord). + +{% endhint %} + +## Contributing guidelines + +We heartily welcome any form of contribution to the project, including issue reports, feature requests, +discussion, pull requests and any type of feedback. We request that all contributors +adhere to the [Contributor Covenant](CODE_OF_CONDUCT.md) and work with us to make the collaboration and +community productive and fun for everyone :smile_cat: + +### How to report an issue + +If you have found a bug, or want to submit a feature request, or suggest any other change, please create +a [new issue](https://github.com/garden-io/garden/issues/new/choose). + +If you report a bug, please describe the steps to reproduce it. You can share the complete Garden configuration of your +project (without any code included) with us by running the `garden get debug-info` command. It will produce a single archive +that matches the directory structure of your project, but contains only your Garden configuration files. This can be very helpful for us to +reproduce and fix the issue. + +## Project structure + +The project code is composed of several components, most of which are written in TypeScript. There's also a number of supporting scripts, documents, examples etc. Here is an overview of the top-level project folders: + +| Name | Description | +| ---- | ----------- | +| `bin` | Executable commands, to use for development. _Note that you need to build the project before these work._ | +| `cli` | The Garden CLI package, which composes code from different packages into the final CLI executable. | +| `core` | The bulk of the Garden code and tests live here. | +| `docs` | Markdown documentation, which is used to generate [docs.garden.io](https://docs.garden.io). _Note that the reference docs are auto-generated, and should not be edited by hand!._ | +| `examples` | Various Garden example projects. | +| `images` | Supporting container images, used by e.g. the `kubernetes` provider. | +| `plugins` | Plugins that are bundled with Garden. We are in the process of moving plugins out of `core` and into separate packages here. | +| `scripts` | Various scripts for builds, releases and development. | +| `sdk` | The `@garden-io/sdk` package, used for Garden plugin development. | +| `secrets` | Encrypted files, used for integ/e2e testing. Only accessible to Garden maintainers. | +| `static` | Static files that are bundled with the CLI. | +| `support` | Supporting files for development, builds, releases etc. | + +### Formatting + +We use [Prettier](https://prettier.io) for automated formatting. We highly recommend installing the appropriate plugin for your editor to automate formatting as you work on your code. You can also run `yarn run fix-format` to fix formatting across the codebase. + +### Commit messages + +We follow the [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0-beta.4/#specification) and automatically validate the formatting of our commit messages. In particular, the **type** of the commit header must be one of the following: + +* **chore**: Changes to the build process or auxiliary tools and libraries such as documentation + generation. +* **ci**: Changes to the CI configuration. +* **docs**: Documentation only changes. +* **feat**: A new feature. +* **fix**: A bug fix. +* **improvement**: Changes that improve a current implementation without adding a new feature or fixing a bug. +* **perf**: A code change that improves performance. +* **refactor**: A code change that neither fixes a bug nor adds a feature. +* **revert**: A commit that reverts a previous commit. It should begin with `revert:`, followed by the header of the reverted commit. In the body it should say: `This reverts commit .`, where the hash is the SHA of the commit being reverted. +* **style**: Changes that do not affect the meaning of the code (white-space, formatting, missing + semi-colons, etc). +* **test**: Adding missing or correcting existing tests. + +When generating the changelog, we only include the following types: **feat**, **fix**, **refactor**, **improvement**, and **perf**. This means that any changes that the user should be aware of, should have one of these types. + +### Checks + +We have scripts for checking licenses, docs, linting and more. These can all be run with a single command: + +```sh +yarn run check-all +``` + +If the checks fail because of bad formatting, run: + +```sh +yarn run fix-format +``` + +If the checks fail because of missing docs, run: + +```sh +yarn run generate-docs +``` + +### Pre-push hook + +Before pushing, we automatically run the `check-pre-push` script (which runs the scripts in `check-all`, except for +`check-docs`), as well as unit tests. To skip these, run push with the `--no-verify` flag: + +```sh +git push origin --no-verify +``` + +### Environment Variables + +You should set the following environment variables when developing on Garden: + +```sh +GARDEN_DISABLE_ANALYTICS=true +GARDEN_DISABLE_VERSION_CHECK=true +ANALYTICS_DEV=true +``` + +## CI + +We use [Circle CI](https://circleci.com) for integration and end-to-end testing. The configuration is in `.circleci/config.yml`. + +## License/copyright headers + +Every source file must include the contents of `support/license-header.txt` at the top. This is +automatically checked during CI. You can run the check with `yarn run check-licenses`. + +## Release process + +### Packages + +Our release process generates the following packages: + +* An executable for OSX, Linux, and Windows, generated by [Pkg](https://github.com/vercel/pkg) and hosted on our [Github page](https://github.com/garden-io/garden/releases). +* A [Homebrew](https://brew.sh/) package for OSX users. + +### Process + +Check out our [release process guide](../../RELEASE_PROCESS.md) for more details. + +## Changelog + +We keep a changelog under [CHANGELOG.md](../../CHANGELOG.md) that gets updated on every release. For pre-releases, we include every pre-release tag in that release cycle in the changelog. So if we're releasing, say, `0.12.6-3`, the changelog will include entries for `0.12.6-0`, `0.12.6-1`, `0.12.6-2`, assuming those tags exist. Once we make a proper release, we remove the pre-release tags so that the changelog only shows changes between `0.12.5` and `0.12.6`. A changelog with the pre-releases is of course always available in our Git history. diff --git a/contributor-docs/config-resolution.md b/docs/contributing/config-resolution.md similarity index 98% rename from contributor-docs/config-resolution.md rename to docs/contributing/config-resolution.md index 2e7968c736..b423db2ff5 100644 --- a/contributor-docs/config-resolution.md +++ b/docs/contributing/config-resolution.md @@ -1,8 +1,12 @@ -# Config resolution +--- +title: Config resolution +order: 6 +--- This doc explains the high-level steps that Garden takes to go from config files on disk to a fully resolved project (with all modules, actions and workflows resolved with no template strings remaining). This includes: + * The steps involved in resolving Garden template strings into concrete values. For example: * E.g. `${environment.name}.mydomain.com` -> `dev.mydomain.com` * or `${actions.build.api.outputs.deployment-image-id}` -> `some-registry.io/my-org/api:v-abf3f8dca`. @@ -18,6 +22,7 @@ Then we'll move on to describing the high-level resolution flow and how it provi ## How template strings are resolved - The parser and the `resolveTemplateStrings` function Let's say we're resolving template strings in this action config: + ```yaml kind: Deploy type: container @@ -27,7 +32,9 @@ spec: $merge: ${var.commonEnvVars} # evaluates to: { SOME_VAR: A, OTHER_VAR: B } ENV_NAME: ${environment.name} # evaluates to: dev ``` + The end result should be: + ```yaml kind: Deploy type: container @@ -38,25 +45,30 @@ spec: OTHER_VAR: B ENV_NAME: dev ``` + The following needs to happen here: + * `${environment.name}` and `${var.commonEnvVars}` need to be resolved (into `dev` and `{ SOME_VAR: A, OTHER_VAR: B }`). * The `{ SOME_VAR: A, OTHER_VAR: B }` map needs to be merged into the `env` map. Both of these happen inside the `resolveTemplateStrings` function. We highly recommend reading the source code for `resolveTemplateStrings` along with this doc, since the full details of the implementation reside there. The `resolveTemplateStrings` function takes the following params: + * `value` is a JSON-like object. For example, a project/action/module config. * `context` provides the template keys & values that the parser uses for lookups—in the example below, this would contain the `environment` and `var` keys (among other things). * `opts` contains a few optional fields, most importantly `allowPartial`, which we'll cover in the section on partial resolution below. In a nutshell, we do the following in `resolveTemplateStrings`: -* We recursively walk through `value`, looking for string values and calling the parser on them (the parser is also provided with the `context` and ` opts` params, among other things). + +* We recursively walk through `value`, looking for string values and calling the parser on them (the parser is also provided with the `context` and `opts` params, among other things). * These values are replaced with the resolved values from the parser. * Structural operators (e.g. `$merge`, `$if`, `$concat` and `$forEach`) are also applied. > Note: To avoid coupling this guide too tightly to the particular implementation (this guide was written in August 2023), we'll leave it to the reader to familiarize themselves with the details by reading the source of `resolveTemplateStrings` and other helpers/data structures around it. The _parser_ is an auto-generated recursive-descent parser, and implements all the syntactic elements available in Garden's template expressions. This includes: + * Boolean expressions * Calling template helper functions (like `camelCase`, `join` or `isEmpty`) * Template context lookups (like `environment.name` or `var.some_key`) @@ -106,8 +118,9 @@ While not all commands will resolve the config graph (which is done via `Garden# > Note: We're using the `Class#instanceMethod` and `Class.classMethod` notation here. At a high level, these are the steps that Garden takes to fully resolve a project (including modules, config templates, templated modules and actions). A close reading of `Garden#getConfigGraph` and the helpers it calls tells you most of what you need to know. + * First, a `Garden` instance is created for the project. - + This involves finding & resolving the project config, and resolving the environment config that the command is being run in. + * This involves finding & resolving the project config, and resolving the environment config that the command is being run in. * The project config has the most limited template context available, since it's resolved first. The environment config context has access to a bit more context, and so forth as we proceed through this flow. * See the `resolveGardenParams` helper. * Next, `Garden#getConfigGraph` calls `scanAndAddConfigs` (`getConfigGraph` is usually called in the `action` method of the command in question). @@ -151,6 +164,7 @@ The `ModuleResolver#resolveModuleConfig` method is where the module config is in After modules have been resolved, they are converted into actions by the `convertModules` function (called inside `Garden#getConfigGraph`). A module usually results in several actions: + * A Build for the build step (if any). * If the module uses the `copyfrom` field, a dummy Build will be included to ensure that the copying takes place when the module's constituent actions are run later. * A Deploy for each service. @@ -165,7 +179,7 @@ To get a better understanding of how module conversion works, we recommend readi ### Phase 1 of action resolution: Preprocessing during config graph construction -Here, we do partial resolution on certain fields on the action configs, fully resolving built-in action config fields (like `include` and ` dependencies`)—that is, framework-level fields that are not plugin-specific. +Here, we do partial resolution on certain fields on the action configs, fully resolving built-in action config fields (like `include` and `dependencies`)—that is, framework-level fields that are not plugin-specific. Some of these built-in fields need to be resolved so the framework can e.g. calculate the action version (`include`/`exclude`) and figure out the dependency structure between actions (`dependencies`) so that the `ConfigGraph` can be constructed. These calculations need to be finished before the solver is called to fully resolve and execute the actions in phase 2 (which is what enables us to resolve references to e.g. action outputs). @@ -180,4 +194,3 @@ For more details on how the solver resolves and executes actions, see the [graph The static and runtime outputs of dependency actions are populated into the `ActionSpecContext` used to fully resolve action specs and variables (see also: `ActionReferencesContext` and the `actions` and `runtime` keys under `ActionSpecContext`). For a closer look at how actions are fully resolved, check out the `process` method of `ResolveActionTask`. - diff --git a/docs/contributing/contributing-docs.md b/docs/contributing/contributing-docs.md new file mode 100644 index 0000000000..b9061f3c74 --- /dev/null +++ b/docs/contributing/contributing-docs.md @@ -0,0 +1,119 @@ +--- +title: Contributing to the Docs +order: 3 +--- + +## How our docs are structured + +Our docs are centralized inside the `docs/` folder of our GitHub repository for `garden`. These files are synced to GitBook, our documentation publishing tool, which composes them into what users see when they navigate to . + +We have an additional class of documentation, which lives as `README.md` files inside each of our examples inside `examples/`. These files are *not* synced to GitBook and are intended to be discovered by our more technical users. + +Our `../../gitbook.yaml` file determines link redirects and the basic structure of our documentation “tree”. You can find more documentation on this file [at GitBook's website](https://docs.gitbook.com/product-tour/git-sync/content-configuration#.gitbook.yaml-1). + +{% hint style="warning" %} +*The `README.md` file at the top of the `docs/` folder and the `reference/*` files are all auto-generated and should never be touched directly. Refer to [Make your changes and open a pull request (PR)](#make-your-changes-and-open-a-pull-request-pr) for details. + +{% endhint %} + +## Making your first contribution + +There's a few things you'll need to make your first contribution to the docs: + +1. A local copy of the `garden` Git repository downloaded to your machine. You can find instructions in GitHub's official documentation for [cloning a Git repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository). +2. Garden's developer tools installed. Our [documentation explains how to install Garden's developer tools](./garden-dev-env-setup.md#step-3-install-dependencies) +3. Some basic knowledge of Git. If one or more users are contributing to the docs at the same time you are, it is likely you will need to resolve merge conflicts on the CLI or in your visual Git tool. GitHub has [documentation on resolving merge conflicts using the CLI](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/resolving-a-merge-conflict-using-the-command-line) or you can use a simple, visual Git tool like [Fork](https://fork.dev/) available for macOS and Windows. + a. In case of emergencies, refer to [Oh Shit, Git!?!](https://ohshitgit.com/). + b. For new learners, the [computer game, Oh my Git!](https://ohmygit.org/) can teach you Git. +4. Some form of Markdown linter. A linter enforces standards and consistency in your Markdown writing. The industry standard is [`markdownlint`](https://github.com/DavidAnson/markdownlint). A [markdownlint extension is available for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint). + +### Make your changes and open a pull request (PR) + +Create a branch to hold your work, change any files you wish to change, save them, then commit them to your branch. Try and keep your branches focused around a specific theme. Your commits will need to be prepended with a *keyword* that follows the Conventional Commits specification, e.g. `docs: added docs contributor page`. See our documentation for [a list of Conventional Commit keywords](./README.md#commit-messages). + +{% hint style="warning" %} +*Before pushing your changes*, run `yarn; yarn build; yarn generate-docs` to refresh the Table of Contents contained at `../docs/README.md` with any additions or changes you've made. + +If you've moved or renamed files, refer to [Moving or renaming files](#moving-or-renaming-files). +{% endhint %} + +Now push your changes to our remote Git repository hosted on GitHub, then [open a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) and request a review from [Eysi](https://github.com/eysi09) and/or [worldofgeese](https://github.com/worldofgeese). View our [contributing guidelines](./README.md#contributing-guidelines) for more. + +## Moving or renaming files + +When moving or renaming files, special care must be taken to ensure any existing internal and external links continue to work. + +First, use `git mv` when moving files. `git mv` is a convenience function that ensures Git can track the renaming as a file rename rather than a file deletion and a new file creation. + +Second, update any internal and external links to point to your moved or renamed file, then update `../gitbook.yaml` to redirect visitors to your new location. To illustrate this, here's a visual example of moving our [Quickstart guide](../getting-started/quickstart.md) to a new Getting Started section. + +![Quickstart link changed to point to new relative URL](https://user-images.githubusercontent.com/59834693/269278499-9e6f9724-fef4-45bf-9f41-fbb462c3b5bf.png) + +We also need to change any existing redirects inside `../gitbook.yaml` to point to our new location: + +![Redirect links pointing to new locations](https://github.com/garden-io/garden/assets/59834693/90c22104-4b5e-4b79-9854-4ad299c1bafb) + +The second line, `quick-start: getting-started/quickstart.md` means any users visiting will automatically redirect to . + +For every file you've renamed or moved, make sure to add a new redirect in `../gitbook.yaml` pointing from its old location to its new location unless you're *sure* no one externally has linked to it. + +You'll also need to find and replace all instances of links within the `garden` repository to your file. An editor like Visual Studio Code will have a [find and replace feature](https://code.visualstudio.com/docs/editor/codebasics#_find-and-replace). + +{% hint style="warning" %} +*Do not change absolute URLs within the same pull request*. Absolute URLs are links beginning with `https:`. + +Instead, change all the relative URLs inside `docs/`, [submit your PR](./README.md#contributing-guidelines), then make a new PR to update these absolute URLs. This is to avoid broken links. +{% endhint %} + +Finally, run `yarn; yarn build; yarn generate-docs` to regenerate the Table of Contents. + +### Absolute URLs vs relative URLs and when to use them + +Use absolute URLs when you need to point to any docs from outside the `docs/` folder. This is often the case when you're working in TypeScript (`.ts`) files. Here's an example from our `../../core/src/graph/actions.ts` file that includes a URL intended for users of `garden`: + +```typescript + if (config.include && config.include.length !== 0) { + throw new ConfigurationError({ + message: deline`Action ${config.kind}.${config.name} (defined at ${configPath}) + tries to include files but excludes all files via "**/*". + Read about including and excluding files and directories here: + https://docs.garden.io/using-garden/configuration-overview#including-excluding-files-and-directories`, + }) + } +``` + +This URL would need to be changed or a redirect link set if the `configuration-overview` file were to move. + +### Creating a new section + +To create a new section, create a folder and add a `README.md` file with the contents `title` and `order` e.g. + +```markdown +--- +order: 2 +title: Tutorials +--- +``` + +The order should correspond to its position from top to bottom in the sidebar on and the title should be the title of the section as it appears in the sidebar. We like to use a floral-themed emoji to demarcate new sections :rose:. + +## Syntax + +Our docs are written in the CommonMark specification of Markdown with additional “blocks” provided by GitBook, our documentation publisher. + +We use the [Hint](https://docs.gitbook.com/content-creation/blocks/hint) and [Tabs](https://docs.gitbook.com/content-creation/blocks/tabs) blocks from GitBook. + +You'll notice YAML “front-matter” at the top of each docs page: this tells GitBook how to present our page visually and order it in any given section. + +The YAML front-matter looks like this: + +```yaml +--- +title: Contributing to the Docs +order: 3 +--- +``` + +Markdown normally expects the first heading of any page to be a top-level, usually the title of the page, e.g. `# Contributing to the Docs`. However, when we specify the title of the page in the front-matter, start the page without the top-level and begin at the second-level heading, `##`, as GitBook will automatically pull in the title for you. + +GitBook supports a [maximum of three levels](https://docs.gitbook.com/content-creation/blocks/heading) of headings. diff --git a/docs/contributing/developing-garden.md b/docs/contributing/developing-garden.md new file mode 100644 index 0000000000..d8245dbbfe --- /dev/null +++ b/docs/contributing/developing-garden.md @@ -0,0 +1,99 @@ +--- +title: Developing Garden +order: 5 +--- + +Once you've [set up your developer environment](./garden-dev-env-setup.md), you're ready to hack on Garden! + +## Debugging + +To enable setting a breakpoint in the code, run the CLI with the `bin/garden-debug` binary, which adds the `--inspect` flag. Developers might find it useful to alias this: + +```sh +alias gd='/path/to/garden/bin/garden-debug' +``` + +You can e.g. use the Chrome DevTools to inspect the code at the breakpoint: + +1. Add a `debugger` statement somewhere in the code. +2. Navigate to [chrome://inspect/](chrome://inspect/) in your Chrome browser. +3. Click the **Open dedicated DevTools for Node** link. +4. Run a CLI command that hits the breakpoint, e.g.: + +```shell +/path/to/garden/bin/garden-debug deploy # or gd deploy, if you've set the alias +``` + +You should now be able to inspect the code at run time in the **Console** tab of the DevTools window. + +## Release binaries and Docker containers + +You can build the release binaries using the command + +```shell +yarn dist +``` + +You can then find the release binaries and archives under `dist/`. + +We release a number of Docker containers on [Docker Hub](https://hub.docker.com/u/gardendev). + +The Docker containers meant to be used directly by the general public are defined in `support/docker-bake.hcl`. + +When making changes to the `Dockerfile` definitions in `support/` it is helpful to build the containers on your local machine. + +For that, first run `yarn dist`, and then run `docker buildx bake` like so: + +```shell +MAJOR_VERSION=0 MINOR_VERSION=13 PATCH_VERSION=0 CODENAME=bonsai \ + docker buildx bake -f support/docker-bake.hcl all +``` + +The environment variables will influence the tags that `buildx bake` will create on your local machine (e.g. stable release tags, prerelease tags, version number, etc.). + +To run the tests on your local machine, first run `yarn dist` (if not already done so), and then run + +```shell +bash support/docker-bake-test.sh +``` + +## Tests + +Unit tests are run using `mocha` via `yarn test` from the directory of the package you want to test. To run a specific test, you can grep the test description with the `-g` flag.: + +```sh +cd core +yarn test # run all unit tests +yarn test -- -g "taskGraph" # run only tests with descriptions matching "taskGraph" +``` + +### ARM64 compatibility + +On ARM64 platforms (like Mac machines with M1 chips) the `yarn test` command may fail with the following error: + +```sh +FATAL ERROR: wasm code commit Allocation failed - process out of memory +``` + +In order to fix it, the terminal must be running in the **Rosetta** mode, the detailed instructions can be found in +[this SO answer](https://stackoverflow.com/a/67813764/2753863). + +Integration tests are run with: + +```sh +yarn integ-local +``` + +End-to-end tests are run with: + +```sh +yarn run e2e +``` + +You can also run the end-to-end tests for a specific example project using: + +```sh +yarn run e2e-project -- --project= +``` + +End-to-end tests are run in CI by using Garden itself to test the project defined in `./core/test/e2e/garden.yml`. Cf. the appropriate job in `circleci/config.yml` for details. diff --git a/docs/contributing/garden-dev-env-setup.md b/docs/contributing/garden-dev-env-setup.md new file mode 100644 index 0000000000..4afbd05575 --- /dev/null +++ b/docs/contributing/garden-dev-env-setup.md @@ -0,0 +1,84 @@ +--- +title: Setting up your developer environment +order: 4 +--- + +## Step 1: Install Docker and Kubernetes + +Please refer to our [installation docs](../getting-started/installation.md) for instructions on how to install Docker and Kubernetes for different platforms. + +## Step 2: Clone the repo + +```sh +git clone https://github.com/garden-io/garden.git +``` + +## Step 3: Install dependencies + +### macOS + +For Mac we have a script that installs all required dependencies. + +If you haven't already, please [install Homebrew](https://docs.brew.sh/Installation). Then run: + +```sh +./scripts/install-osx-dependencies.sh +``` + +### Windows / Linux + +Other platforms need to roll their own for now (contributions welcome!). Please have a look at the script for OSX to see what's installed. + +If you have [LinuxBrew](https://docs.brew.sh/Homebrew-on-Linux) installed, [install-osx-dependencies.sh](../../scripts/install-osx-dependencies.sh) should work if you run it, although you will have to ensure that you've added NPM to your PATH via `.bashrc` `.zshrc` or other shell run command script. + +### asdf + +If you are an [asdf](https://asdf-vm.com/) user, running [install-asdf-dependencies.sh](../../scripts/install-asdf-dependencies.sh) in order to automatically install the correct plugins and versions as defined in `.tool-versions`. + +## Step 4: Bootstrap project + +Install Node modules for the root package, and `core` package: + +```sh +yarn install # To install root dependencies +yarn run bootstrap # To bootstrap packages +``` + +from the root directory + +You may need to install the Node modules in the core package manually due to [lerna/lerna#1457](https://github.com/lerna/lerna/issues/1457). + +```sh +cd core +yarn +``` + +## Developing Garden + +### Initial build + +Before running Garden for the first time, you need to do an initial build by running + +```sh +yarn build +``` + +from the root directory. + +### Developing + +To develop the CLI, run the `dev` command in your console: + +```sh +yarn dev +``` + +This will link it to your global `node_modules` folder, and then watch for +changes and auto-rebuild as you code. You can then run the `garden` command as normal. + +Also, you might like to add a couple of shorthands: + +```sh +alias g='garden' +alias k='kubectl' +``` diff --git a/contributor-docs/graph-execution.md b/docs/contributing/graph-execution.md similarity index 85% rename from contributor-docs/graph-execution.md rename to docs/contributing/graph-execution.md index 9a233a8976..ca9d74b64f 100644 --- a/contributor-docs/graph-execution.md +++ b/docs/contributing/graph-execution.md @@ -1,20 +1,25 @@ -# Graph execution +--- +title: Graph execution +order: 7 +--- This doc explains the steps involved when `GraphSolver` (solver) processes a set of tasks in dependency order. The solver and the task classes (e.g. `ResolveActionTask` and `BuildTask`) are tightly integrated, so this doc will cover them as a single whole. > Note: This doc is intended to be read side-to-side with the source code. We'll try to stay high-level here to avoid coupling too tightly to the implementation (which may change over time). The goal is to give a high-level overview, not to cover the finest details. Internally, the solver and the task classes are used for: + * Resolving provider configs. * Resolving action configs. * Executing actions. Each of the above use-cases benefits from the capabilities of the solver: + * Dynamic caching & dependency processing: The solver uses the `getStatus` method of the tasks it's processing to determine whether they (or their dependencies) need to be processed at all. - * Note: `ResolveProviderTask` and `ResolveActionTask` currently always return a null status (so they don't make use of this capability of the solver). - * Importantly, `BuildTask`, `DeployTask`, `TestTask` and `RunTask` all implement a `getStatus` method that's designed to include a `ready` state in the status when the underlying action doesn't need to be run (because an up-to-date build/deploy/test/run has already taken place). + * Note: `ResolveProviderTask` and `ResolveActionTask` currently always return a null status (so they don't make use of this capability of the solver). + * Importantly, `BuildTask`, `DeployTask`, `TestTask` and `RunTask` all implement a `getStatus` method that's designed to include a `ready` state in the status when the underlying action doesn't need to be run (because an up-to-date build/deploy/test/run has already taken place). * Collecting & providing dependency results to dependants. - * Having access to the results of processing a task's dependencies is essential for e.g. resolving actions that reference static or runtime outputs from another action. + * Having access to the results of processing a task's dependencies is essential for e.g. resolving actions that reference static or runtime outputs from another action. * Concurrency control, standardized error handling and logging. While this system may look somewhat thorny from the outside, it's actually relatively simple and intuitive once the moving parts are understood. @@ -22,6 +27,7 @@ While this system may look somewhat thorny from the outside, it's actually relat ## High-level flow At its most basic level, the solver flow is as follows: + * `Garden#processTasks` is called with a list of tasks, which in turn calls `GraphSolver.solve` * The solver wraps each of these tasks in a node. * Nodes for any tasks that already have an up-to-date result available (i.e. whose `getStatus` method returns a ready status) are completed without needing to be processed. @@ -37,10 +43,12 @@ We'll start from the bottom up, beginning with the task classes (which are the m ## The lifecycle of a task - config resolution, status checking and execution The task classes are wrappers around operations that can have dependencies. These include: + * Action execution task classes for each of the four action kinds: `BuildTask`, `DeployTask`, `TestTask` and `RunTask`. * `ResolveActionTask` and `ResolveProviderTask` - for resolving action and provider configs, respectively. The role of task classes is to: + * Optionally define a `getStatus` method that the solver can call to determine whether the task needs to be executed. * Provide the solver with a `process` method to call (to run the task) if the `getStatus` method returns `null` or a non-ready status. * Tell the solver which dependencies the task has - those that are needed to check the status, and those that are needed to actually process the task (see the next section for more details on task dependencies) . @@ -66,6 +74,7 @@ The primary example of this is for the action execution task classes, which all Similarly, _process dependencies_ are the pre-requisites for the task to be run to completion. We highly recommend reading the code for the `resolveProcessDependencies` methods of `ResolveActionTask` and `BaseActionTask` to start with. The specifics vary in how the `resolveProcessDependencies` method is implemented on the various task classes, but usually this involves iterating over the dependency references of the underlying action (or provider, for `ResolveProviderTask`) and returning different types of task depending on the the dependency spec: + ```typescript export interface ActionDependencyAttributes { explicit: boolean // Set to true if action config explicitly states the dependency @@ -73,6 +82,7 @@ export interface ActionDependencyAttributes { needsExecutedOutputs: boolean // Set to true if action cannot be resolved without the dependency executed } ``` + The idea here is to use the cheapest / least processed type of task that's required to satisfy the dependency. This helps avoid unnecessary work, which is important for keeping things nice and fast. From least to most processed, this means: _No dependency < Resolved action/provider < Executed action/provider_. @@ -94,29 +104,31 @@ The details of how this is hashed out are liable to change over time, but this i Now that we've covered the task classes, their status checking and dependency calculation logic at some length, we're well prepared to understand how the solver itself works. Internally, the solver uses three node classes (each of which wraps a task instance): + * `RequestTaskNode` - * Initially, one of these is created for each task requested in the call to `GraphSolver.solve`. - * These are essentially the root nodes of the graph to be executed, and represent the result that is to be returned for the requested task in question. - * They don't have an `execute` method themselves, and depend on either a process or status node (depending on whether the request is for a `statusOnly` result — such requests come e.g. from commands that just need to check that a Deploy is running). + * Initially, one of these is created for each task requested in the call to `GraphSolver.solve`. + * These are essentially the root nodes of the graph to be executed, and represent the result that is to be returned for the requested task in question. + * They don't have an `execute` method themselves, and depend on either a process or status node (depending on whether the request is for a `statusOnly` result — such requests come e.g. from commands that just need to check that a Deploy is running). * `StatusTaskNode` - * When executed, calls the `getStatus` method of its underlying task. - * Its dependencies are the status dependencies of the underlying task. + * When executed, calls the `getStatus` method of its underlying task. + * Its dependencies are the status dependencies of the underlying task. * `ProcessTaskNode` - * When executed, calls the `process` method of its underlying task. - * Its dependencies are the process dependencies of the underlying task. + * When executed, calls the `process` method of its underlying task. + * Its dependencies are the process dependencies of the underlying task. Additionally, the solver maintains Maps of pending and in-progress nodes. The flow is as follows: + * `Garden#processTasks` is called with a list of tasks, which in turn calls `GraphSolver.solve`. * The solver wraps each of these tasks in a `RequestTaskNode` and starts the loop (see the `loop` and `evaluateRequests` methods on the solver). * In each iteration of the loop: - * The solver looks at each incomplete request, checks if any of them were completed in the last iteration and completes them. - * A status or process node may be added to the pending graph. + * The solver looks at each incomplete request, checks if any of them were completed in the last iteration and completes them. + * A status or process node may be added to the pending graph. * Note: Please refer to the source code for the specifics — any detailed description here is likely to drift from the implementation over time. and processes them in dependency order, making sure to first process any required dependencies. * The solver ensures that any remaining dependencies of `this.pendingNodes` are included. -* A dependency graph is generated from the remaining (i.e. not complete) pending nodes, and queried for the leaf nodes. This step also detects circular dependencies. +* A dependency graph is generated from the remaining (i.e. not complete) pending nodes, and queried for the leaf nodes. This step also detects circular dependencies. * Up to a certain concurrency limit, the leaf nodes of the dependency graph are processed (and added to the in-progress Map). * When a node finishes processing, it's marked ad complete, and the result of processing is saved on the node (this will be the return value of the `getStatus` or `process` method of the underlying task, for status and process nodes respectively). * Eventually, all the requested tasks will be completed, and the results will be returned. diff --git a/docs/misc/README.md b/docs/misc/README.md index e7aebc491b..1c153a44f8 100644 --- a/docs/misc/README.md +++ b/docs/misc/README.md @@ -1,6 +1,6 @@ --- title: Misc -order: 99 +order: 98 --- # Overview diff --git a/package.json b/package.json index 84dda986d1..f21953802a 100644 --- a/package.json +++ b/package.json @@ -67,11 +67,11 @@ "add-licenses": "copyright-header --license-file support/license-header.txt --syntax support/license-syntax.yaml --add-path core/src:core/test --output-dir .", "bootstrap": "npm install", "build": "npm run clean && npm run bootstrap && ./scripts/run-script.ts build", - "check-all": "npm run check-package-lock && npm run check-licenses && npm run lint && npm run check-docs", + "check-all": "npm run check-package-lock && npm run lint && npm run check-docs", "check-docs": "./scripts/check-docs.sh", "check-package-licenses": "./scripts/check-package-licenses.ts", "check-package-lock": "git diff --quiet HEAD -- package-lock.json || (echo 'package-lock.json is dirty!' && exit 1)", - "check-pre-push": "npm run check-package-lock && npm run check-licenses && npm run lint", + "check-pre-push": "npm run check-package-lock && npm run lint", "clean": "./scripts/run-script.ts clean && find . -name \".garden\" -type d -prune -exec rm -rf '{}' '+'", "clean-node-modules": "find . -name \"node_modules\" -type d -prune -exec rm -rf '{}' '+'", "dev": "./scripts/run-script.ts dev --parallel",