Development Mode

[hiremaga]

The Idea

Taking inspiration from GOOGLE_DEVMODE, let's discuss a standard way for a platform like pack to signal to buildpacks that they are being used in development, rather than to create a production image.

• To open the door to CNB buildpacks being used in the inner-loop, what's the smallest change we might make?
• Are there considerations, risks & constraints we must take into account early, even for a narrowly scoped RFC?
• Are you aware of any other work/experimentation being done in this area?

Context

There have already been discussions about using buildpacks for development & testing. Heroku's Testpack API extends buildpacks to run automated tests. Similarly, Google extends its buildpacks for use in a developer's inner-loop via Skaffold. Tilt, another inner-loop orchestrator like Skaffold, includes an extension for pack. All this suggests a healthy appetite to use buildpacks during development in addition to using them to build images for production.

Today, running buildpacks after every change in development is slow. Skaffold uses a special flag GOOGLE_DEVMODE to signal to Google's buildpacks that the application is being built is intended for development. Introducing a CNB standard alternative to this flag, e.g. BP_MODE=development or BP_DEVELOPMENT=true, would open the door for other families of buildpacks, like Paketo, to introduce similar capablities.

---

A few people I've had discussions with to learn about related prior work include: @jabrown85, @ekcasey, @natalieparellano, @briandealwis & @paulcwarren.

[dfreilich]

This sounds really incredible. I'd love to see us flesh out the path forward for this!

[briandealwis]

Full credit for the Skaffold GOOGLE_DEVMODE solution goes to @dgageot. It's very elegant:

1. skaffold dev configures the build-time environment variable GOOGLE_DEVMODE=1
2. GCP Buildpacks see this environment variable and change their behaviour to
   1. install the compiler toolchains into a launch layer
   2. configures a file-watcher (currently watchexec) for a fixed set of extensions, that rebuilds and then restarts the app
   3. describes a set of Skaffold sync-rules using the buildpacks build metadata (globs mapping file-types to destination, usually of the form **/*.go → /workspace)
3. Skaffold retrieves the built image's config and sets up sync rules from those found in the build metadata
4. On a file change that matches a sync rule, Skaffold transmits the changed files to the remote image which triggering an in-container rebuild. For files that don't match a sync rule, Skaffold rebuilds and redeploys the image.

Key to this experience is that Skaffold handles the deployment and is able to sync files to that environment.

Three aspects worth calling out:

1. The sync+rebuild happens within the resulting run image rather than the builder image: it's very close to what will be deployed in production. There's less of a chance that the application will accidentally make use of some package that was installed in the builder image.
2. The buildpacks can evolve their sync support independently of Skaffold: if we enable other sync rules for Python, Skaffold will pick up the changes immediately. Similarly we can introduce new file-watch mechanisms, such as to supporting more idiomatic tools like nodemon or ts-node-dev, without any changes required in Skaffold.
3. Skaffold's image caching, that seeks to avoid image rebuilds when there are no source changes, incorporates the build-mode so as to prevent a skaffold dev-created image from escaping. So if the user subsequently performs a skaffold build, Skaffold would rebuild the image and not use the skaffold dev-created image.

Limitations in our implementation is that the set of supported filetypes and file-watch mechanisms are hard-coded into the buildpacks. I've been thinking that we could allow users to describe a set of filetypes and a watch mechanism in the project.toml.

[jabrown85]

skaffold dev requires a kubernetes cluster right?

[briandealwis]

Skaffold is Kubernetes focussed currently, though we have some experimental work to support deployments to other environments.

[jabrown85]

I would love to see a version of the inner loop that does not create an image, but rather build and then launch within a single container. When measuring inner loop time around a local docker experience, a very large chunk of time was spent exporting the image to the daemon.

We experimented with a process that mounted the code into the builder image and then kicked off /cnb/lifecycle/detector and /cnb/lifecycle/builder and then /cnb/lifecycle/launcher. It also used watchexec to watch the mounted directory for changes and then re-kick the cnb binaries. It was fast, but needed more work as we were violating specs by not cleaning up layers and buildpacks would sometimes fail due to some of these violations.

The reason I like this idea is because it doesn't require kubernetes for local development and I think it could mean easier integration with VS Code's in-container workflows (local and remote), CodeSpaces, etc. It also means that if you wanted to use kubernetes pods as the platform for development, your nodes are ready faster due to only requiring the builder image + the source code.

[Chuxel]

This thread is older, but wanted to say that, from the GitHub and VS Code perspective, I would love work with folks to see if we couldn't help improve things for these kinds of scenarios. One of the questions we've had whether the Buildpack community would be open to the idea of enabling dev container image generation from Buildpacks. What seems valuable would be to enable someone to people can develop inside of a container based environment - this avoid having to do any image generation during the developer "inner loop", simplify debugging, etc. Heck, even reusing these kinds of images during CI. Production symmetry with even something as simple as where something is installed can be a huge win.

So far, we've explored the idea of a "dev container mode" (even if informally defined) where a Buildpack similar to Paketo's cpython could vary its behaviors - or be a requirement for another Buildpack. e.g. You'd want pylint in development, but not in production. Similar problems exist between SDKs and Runtimes, etc. Beyond the layer types, we could provide a way for requires.metadata to pass along options as well.

However, in investigating this in detail, we hit several snags that currently require some sort of post-processor to resolve near as we can tell.

1. It is pretty common to use something like "docker exec" to interact with a container during development rather than exclisvely relying on the entrypoint. Currently there are a number of things the /cnb/lifecycle/launcher adds to the operating environment. Notably even things like setting a static env var (FOO=true) go through this process. The only way we can currently see to enable exec to get these environment vairables is to add its execution into any interactive or login shell via /etc/bash.bashrc and /etc/profile.d as a part of post-processing - or do a secondary Dockerfile pass and convert them to ENV statements. Not sure if there's a better way to address this with the spec today.

2. In this scenario, we find it useful to exclude the source code from the final "launch" image. That source code can be in a volume or other mount point that is dynamically added to the image - which also keeps the image size down. The trick is that the Buildpack would still need access to the source code for the "detect" phase at a minimum - just not copied into the final launch image. Right now we appear to need to explictly remove all the code via a one-off Buildpack that does this as the last step the order for the builder. Love to figure out a way to address this more formally since its easy to forget to add it.

3. Many quality of life updates require native packages or updating content outside of the layers folder. A simple example here is installing zsh and Oh My Zsh. You can add all of these into the Stack images, but ideally you'd be able to inject these in the same way you can something else. At the moment, some sort of post-processing seems like it will be required, though RFC: Support Dockerfiles rfcs#173 could solve that. This would also help a ton with installing utilities like "htop" devs would normally want that wouldn't be in a typical production image.

4. We also hit a challenge with processe that should only fire in the entrypoint. Given we're having to use the launcher in other places to resolve the "docker exec" challenge, processes that should start up when the container starts require another workarounds. Right now, it seems like a post processing step to inject a entrypoint wrapper that then does "exec" on /cnb/lifecycle/launcher after it completes seems like the best answer at the moment. Not sure if RFC: Support Dockerfiles rfcs#173 would solve this one.

5. We also found that certain Buildpacks would need be able to affect container settings. For example, debugging Go in a container requires the ptrace capability to be enabled. We should be able to use image labels to accomplish this, but I thought I'd call it out in the event this has been discussed.

I'd be up for making some proposals / collaborating if it would be of interest to the community. We're looking at opening up a number of things we haven't been able to so far already so it's top of mind.

[jabrown85]

@Chuxel Super happy to discuss this with you as well! We at Salesforce explored a cnb-develop flow a bit but never finished it up.

My first question back to you is around what "dev container mode" means to you. 😄

Your example given was producing a developer image and then running on that CNB produced image as the tight inner loop. Personally, we were exploring a path where we put the user's code onto the builder image. Acting more like a platform rather than consuming CNB created images.

{
  "name": "cnb-debug",
  "dockerFile": "Dockerfile",
  "forwardPorts": [8080,9222],
  "workspaceMount": "type=bind,src=${localWorkspaceFolder},dst=/workspace",
  "workspaceFolder": "/workspace/"
}

FROM heroku/buildpacks:20

Then we execute the phases of lifecycle manually (detector, builder, but never exporter for instance) like pack would inside the container. One major advantage, in my experience, was not having to deal with CNB built images that were intended for running a single app. Running on fat builder images as opposed to tiny run images are also inherently much more useful for debugging and general development. Not having some deps on the run image (gcc) also makes some re-running buildpacks for the inner loop impossible.

Our goal was local docker-based development though, not Skaffold/Tilt type of work. Though I think we could ship the user's code into a builder image for Skaffold/Title as well perhaps in the same way.

I'll throw some of my thoughts out here to continue the discussion from your points.

1. We also came to the figured out that we had call into /cnb/lifecycle/launcher all the time. This works ok. Things like exec.d make it complicated to remove launcher.
2. We also ran into issues similar to this. I eventually landed on mounting in the source code in read only volume not in /workspace when running the dev container and essentially replacing /workspace when executing a rebuild. It is gross, but needed. Some buildpacks remove all the code and leave a single binary in /workspace.
3. As I mentioned above, we actually ran the builder images and not the run images so the fatter build image had most everything we wanted. I do think the RFC: Support Dockerfiles rfcs#173 would help here.
4. You may also inject a custom develop-buildpack first that contributes an exec.d that does the things you want in an entrypoint when building this dev image.
5. I don't have any thoughts on this right now.

[Chuxel]

Yeah, makes sense - thanks for the great insights - super interesting to see you hit the same challenges! RE: the build image, that was the other thing we were thinking about - updating "pack" with some command to enable this kind of scenario. Developers then can do the inner loop in a generated build image rather than a "launch" image. If a "dev mode" would do this, that would be perfect.

Essentially to your point, what we'd have to do otherwise is make the Buildpack aware that it was in "dev mode" and then include these deps in the launch image... which works but without Docker support, native dependencies were a problem like you said.

The main question we'd had is whether doing something like exporting the build image would be considered going "outside the spec". The platform steps would not be, but we started staying within the walls of the spec to see where we could get. (Certainly, it something we could contribute back to something like pack if it would be interesting.)

1. We also came to the figured out that we had call into /cnb/lifecycle/launcher all the time. This works ok. Things like exec.d make it complicated to remove launcher.

Yeah, we got it working, but it made me wonder whether there should be a better way. Like you said, the conversion was going to be quite complicated. We added this to the top of /etc/bash.bashrc, /etc/profile, and /etc/zshenv in post-processing:

if [ ! -z "${CNB_APP_DIR}" ]; then mapfile -d $'\0' _cmd_line < /proc/$$/cmdline; exec /cnb/lifecycle/launcher "${_cmd_line[@]//\"/\\\"}"; fi

...which works since CNB_APP_DIR gets filtered by the launcher.

2. We also ran into issues similar to this. I eventually landed on mounting in the source code in read only volume not in /workspace when running the dev container and essentially replacing /workspace when executing a rebuild. It is gross, but needed. Some buildpacks remove all the code and leave a single binary in /workspace.

Yeah, agreed. I guess if we made a custom command we could take care of cleaning out the workspace folder automatically.

3. As I mentioned above, we actually ran the builder images and not the run images so the fatter build image had most everything we wanted. I do think the RFC: Support Dockerfiles rfcs#173 would help here.

Did you still run into problems with all content needing to be in the layers folder (at least per the spec)? e.g. We can't update ~/.bashrc. Or did you make those modifications in the Buildpacks since they ended up getting persisted in the resulting build image? We ended up post-processing to make these types of changes in our PoC.

4. You may also inject a custom develop-buildpack first that contributes an exec.d that does the things you want in an entrypoint when building this dev image.

Agreed, but I'd assume exec.d would end up firing multiple times because of what we have to do for issue 1, correct? We can inject an env var to check for it, but that was the problem we hit that caused us to consider post-processing. The entrypoint scripts end up needing to be aware that they'll happen more than once.

5. I don't have any thoughts on this right now.

Yeah, this one is interesting and we want to be orchestrator agnostic. Really, devcontainer.json is a tool/service a metadata format that we're opening up. The goal there really was to have a place to store these kinds of tweaks and tool specific content in a structured, reusable form. With that in mind, its entirely reasonable to persist those same values as label metadata on an image (or in other formats). So long as a given buildpack is aware of it, it could add the labels - but if there was something else being considered, that would be of interest to know for sure.

[jabrown85]

Did you still run into problems with all content needing to be in the layers folder (at least per the spec)? e.g. We can't update ~/.bashrc. Or did you make those modifications in the Buildpacks since they ended up getting persisted in the resulting build image? We ended up post-processing to make these types of changes in our PoC.

There were some problems with all the stuff being in /layers. Specifically around tooling like VS Code extensions expecting to find all the node_modules in the working directory /workspace but they are really in /layers/node/node_modules. I think we ended up symlinking the node_modules into /workspace from the /layers. It was not great and that is not something I'd want to have to do.

Agreed, but I'd assume exec.d would end up firing multiple times because of what we have to do for issue 1, correct? We can inject an env var to check for it, but that was the problem we hit that caused us to consider post-processing. The entrypoint scripts end up needing to be aware that they'll happen more than once.

Yeah - this was also true for actual buildpack invocations as well. You run into similar problems where a buildpack might have left something around in /layers/bp/whatever that does not get put into the final app image or cache image and a re-execution of build means it takes an unexpected path thinking it already did work that it really should be re-doing. A postbuild process is likely needed to ensure compliance for the next execution.

[Chuxel]

Thanks for the tips! We'll do some exploring along these lines!

[dmikusa]

FYI, this is what Paketo currently has on the roadmap in regards to using buildpacks in development. The work hasn't been implemented as of yet, but the details of the plan are in the doc.

https://github.com/paketo-buildpacks/rfcs/blob/main/text/0032-reloadable-process-types.md