-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add compile command #315
add compile command #315
Conversation
6873040
to
951b809
Compare
d64807e
to
72f475b
Compare
An (imo non-blocking) issue with this PR: It vendors "gopkg.in/yaml.v2", v2 which no longer supported. I made a PR here to do the bump: cloudfoundry/bosh-cli#639 |
I feel bad that the release tarball contains the Go package; from a deployment perspective, the number duplicate bytes across all the releases that contain the Go package is huge. I wish we had some way to annotate packages with a field like |
8fcc80d
to
6874cad
Compare
We actually modified compilation to exclude packages not required by a job earlier this year. Unfortunately it broke everything. The problem is that bosh uses a package fingerprint to know if the package has been modified and is different from another copy of that package. The fingerprint is based on the source files for the package (their bits, as well as their paths and permissions). This means if you change the version of the golang package you built a package with, the package fingerprint doesn't change, and bosh thinks it's the same, even though the compiled bits are built with a totally different version of golang. So to fix it, we'd need to change the structure of release metadata to include the fingerprint data for packages a package depends on, without actually including those packages. It would likely be a largish breaking change, so we just backed away from it. |
it would be a good idea to use this in our bosh director pipeline |
I tried running it and got this:
Keep in mind, this isn't the exact desired use case since I didn't have a stemcell with this agent handy. But I imported the latest jammy stemcell into docker, then ran the container locally and mounted in the bosh-agent I had built. The error seems to imply it's looking for |
ab92806
to
ea05500
Compare
@jpalermo I added the a MkdirAll to the command. |
I've been vendoring the HEAD of "github.com/cloudfoundry/bosh-cli/v7/release/manifest" from a SHA so I get the yaml.v3 changes (cloudfoundry/bosh-cli#639). Any chance there will be a tagged v7 revision of the BOSH CLI soon? |
cc6f41f
to
8f72f51
Compare
I force-pushed a manual rebase, go mod tidy, and go mod download. |
This has been released in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to compile the bosh-director just fine with it:
2024/01/06 19:33:07 Compiling with stemcell ubuntu-jammy/1.329
Reading BOSH Release Manifest from tarball ./bosh-280.0.12.tgz
Release bosh/280.0.12 has 17 packages
Extracting packages
Copying package nginx to /var/vcap/data/blobs/nginx.tgz
Copying package director to /var/vcap/data/blobs/director.tgz
Copying package postgres-13 to /var/vcap/data/blobs/postgres-13.tgz
Copying package azure-storage-cli to /var/vcap/data/blobs/azure-storage-cli.tgz
Copying package davcli to /var/vcap/data/blobs/davcli.tgz
Copying package bosh-gcscli to /var/vcap/data/blobs/bosh-gcscli.tgz
Copying package postgres-15 to /var/vcap/data/blobs/postgres-15.tgz
Copying package director-ruby-3.1 to /var/vcap/data/blobs/director-ruby-3.1.tgz
Copying package health_monitor to /var/vcap/data/blobs/health_monitor.tgz
Copying package verify_multidigest to /var/vcap/data/blobs/verify_multidigest.tgz
Copying package director-ruby-3.2 to /var/vcap/data/blobs/director-ruby-3.2.tgz
Copying package postgres-10 to /var/vcap/data/blobs/postgres-10.tgz
Copying package libpq to /var/vcap/data/blobs/libpq.tgz
Copying package mysql to /var/vcap/data/blobs/mysql.tgz
Copying package powerdns to /var/vcap/data/blobs/powerdns.tgz
Copying package nats to /var/vcap/data/blobs/nats.tgz
Copying package s3cli to /var/vcap/data/blobs/s3cli.tgz
Starting packages compilation
Compiling package azure-storage-cli/7be841fab23c7dac21de007f1ef7956dda59333be3b6a7ae51884afe0dfdd49b
Finished compiling release azure-storage-cli/7be841fab23c7dac21de007f1ef7956dda59333be3b6a7ae51884afe0dfdd49b BlobstoreID=78028c3f-440b-438c-78cb-d7b0dd373f01
Compiling package bosh-gcscli/7812b42fb1b70d42d8c552a677e3752bfbc6f63c8885c5c8de040c74b69e5237
Finished compiling release bosh-gcscli/7812b42fb1b70d42d8c552a677e3752bfbc6f63c8885c5c8de040c74b69e5237 BlobstoreID=eb131610-6a63-4d59-42db-47bdc54e0578
Compiling package davcli/ba8e4c230136743bc3c6a0425026a855c02480dc8c9e04196921d909edcc8fff
Finished compiling release davcli/ba8e4c230136743bc3c6a0425026a855c02480dc8c9e04196921d909edcc8fff BlobstoreID=684dcdb2-b310-4760-57b9-18240d52c827
Compiling package libpq/b309a72768019e24e2c592f3f25ded2679e98cbb90f774c3a4d6b7745760079f
Finished compiling release libpq/b309a72768019e24e2c592f3f25ded2679e98cbb90f774c3a4d6b7745760079f BlobstoreID=3c6107e2-b2d1-4ae0-4114-ca1d0a2b1c41
Compiling package mysql/5bf574c1f12cd6b5c07bf0c266b4166584bf0503634f1281bf58637f46f2bd5f
Finished compiling release mysql/5bf574c1f12cd6b5c07bf0c266b4166584bf0503634f1281bf58637f46f2bd5f BlobstoreID=6746f492-b8cc-4868-5591-269f6d636339
Compiling package director-ruby-3.2/84ee2f9d0485530a75822fa03e7fd0c73544aa4c2f6fe24aaebebe1757195efe
Finished compiling release director-ruby-3.2/84ee2f9d0485530a75822fa03e7fd0c73544aa4c2f6fe24aaebebe1757195efe BlobstoreID=0de1b578-ace6-4906-52e2-9fa7c8103d6c
Compiling package director/1e8ac7cdd35cd2277706fb607b6b373729b9614cc949e20be10f7e22a151b190
Finished compiling release director/1e8ac7cdd35cd2277706fb607b6b373729b9614cc949e20be10f7e22a151b190 BlobstoreID=55b62c93-0571-4e2e-5b5a-f436a98c8c1e
Compiling package director-ruby-3.1/82651486268c5a872706d5e1a2458b76af7e69fe36dbfed64903c64a3af39eae
Finished compiling release director-ruby-3.1/82651486268c5a872706d5e1a2458b76af7e69fe36dbfed64903c64a3af39eae BlobstoreID=371f3897-2c2f-4251-610a-506f6e80b89e
Compiling package health_monitor/4d2d309eb06ddaf2993cc4d2253460122375786dab011591047ccf6efa932737
Finished compiling release health_monitor/4d2d309eb06ddaf2993cc4d2253460122375786dab011591047ccf6efa932737 BlobstoreID=010e49b6-262e-4a95-548f-ea2f83619fd9
Compiling package nats/a9355d553cab9913f195eaf00cbbd01ec3c2b8b692b851ce34e0b95dad7ca0b0
Finished compiling release nats/a9355d553cab9913f195eaf00cbbd01ec3c2b8b692b851ce34e0b95dad7ca0b0 BlobstoreID=4562253f-191b-4199-6250-794ad21cb917
Compiling package nginx/82a22b536cf378d354f9325dadcbcb2fa70b1ce9e37eb65a8a7a97cd35e8fc45
Finished compiling release nginx/82a22b536cf378d354f9325dadcbcb2fa70b1ce9e37eb65a8a7a97cd35e8fc45 BlobstoreID=e6f41a33-cf50-4a38-68d0-5d7091befc22
Compiling package postgres-10/e3f2ed31116e1a0c929ae6fcdde983a9d6c000c25cafde8a784fd126e06400f9
Finished compiling release postgres-10/e3f2ed31116e1a0c929ae6fcdde983a9d6c000c25cafde8a784fd126e06400f9 BlobstoreID=23d100d8-886d-4561-757a-32adc6f4d76d
Compiling package postgres-13/a3141b9f3664abe145c6fb452a54b3bbc4b772933083c2c1ef725c0a7c71824f
Finished compiling release postgres-13/a3141b9f3664abe145c6fb452a54b3bbc4b772933083c2c1ef725c0a7c71824f BlobstoreID=8da9d28a-345c-4fe4-75e8-f80d26869ce2
Compiling package postgres-15/1059ac62d543dc19011001f80f8c0bb99cc3a9ea4f8c14736e480701051ce9f0
Finished compiling release postgres-15/1059ac62d543dc19011001f80f8c0bb99cc3a9ea4f8c14736e480701051ce9f0 BlobstoreID=170a3822-e7e9-45ff-77dd-8a4b4fae9707
Compiling package powerdns/95d1ce42da1bbaa19021bcaa295ed4f0f05e82c44b40638fcc934d91d212d2af
Finished compiling release powerdns/95d1ce42da1bbaa19021bcaa295ed4f0f05e82c44b40638fcc934d91d212d2af BlobstoreID=7750d851-c5f2-4c5d-66bc-262504508fb2
Compiling package s3cli/66e805d974a48631f5c963f009b9a14d451ed5bdf88900f45fa86c9a7342a2ff
Finished compiling release s3cli/66e805d974a48631f5c963f009b9a14d451ed5bdf88900f45fa86c9a7342a2ff BlobstoreID=e9077821-2c1e-4df0-4105-4e33d6ca4bf6
Compiling package verify_multidigest/5a45c1115b9a7478682ba63804050a723d9eb6e28d91432ad36eba30cc084c59
Finished compiling release verify_multidigest/5a45c1115b9a7478682ba63804050a723d9eb6e28d91432ad36eba30cc084c59 BlobstoreID=bed88047-72f7-4007-68a8-fcd030de2bab
Finished packages compilation after 47m23.551391571s
Archiving compiled BOSH Release bosh/280.0.12 with stemcell ubuntu-jammy-1.329
Finished archiving compiled tarball /tmp/build/e55deab7/artifacts/bosh-280.0.12-ubuntu-jammy-1.329.tgz
- add create-tarball CLI - this is intended to run in a container image of a linux stemcell - fix path in output tarball - remove path prefix characters - ensure same file mode
squashed dev commits - unit test happy path for releasetarball.Compile - refactor: use bosh-cli BOSH Release manifest structures - add test coverage for already compiled and zero package releases - fix readStemcellSlug on GOOS=windows - rename file to compile becuase it more closely represents the command - rename command to "compile" from "compile-release-tarballs{,s}" - docs: add compilation help text
I rewrote the tests to match existing test conventions. This commit also adds test coverage for compilation and release parsing failure cases. fix-vendor
refactor: move stemcell metadata loading to a separate package refactor: improve error message in stemcellmetadata.SlugParts
I accidentally duplicated a test case.
I don't know if it is okay to use 0x766 for the directory permissions.
8f72f51
to
c5c1e32
Compare
I force pushed a rebase on main fixing the conflicts in go.mod, go.sum, and vendor/modules.txt. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really nice work, only found an unused file.
Thanks! @crhntr |
Problem
Compiling BOSH release packages in CI requires either
This blocks integrating build observers and limits where we can build compiled tarballs.
We wrote a script that can build release tarballs with compiled packages by reverse engineering the commands issued to the agent from the BOSH Director. Unfortunately, this script still requires the warden stemcell container be run with
privileged: true
because the bosh-agent does a few things during startup that are not required for package compilation.Both using an internal RPC interface and requiring privileged containers make using the existing BOSH stemcell images for release compilation in CI harder and more brittle than than it needs to be.
Solution
Add a sub-command to the bosh-agent that sidesteps the privileged parts of initialization and that can compile releases tarballs and that can be called directly.
Example using the
docker
CLIA Concourse task wrapping the following example should be trivial.
Given some warden stemcell
jammy-stemcell-warden:1.260
asTAG
,And you have donwloaded a BOSH Release tarball at
releases-input/log-cache-release-3.0.7.tgz
When you run
Then You will get
releases-output/log-cache-release-3.0.7-ubuntu-jammy-1.260.tgz
Note the docker command does not require a "--privileged" flag.
If you want to try using this functionality locally, I added a script to replace the bosh-agent in the image and load the appropriate volumes. You can find it in a script in the repo at
bin/docker-run-bosh-agent
.Limitations/Warnings
The following are not a problem when running the command on a warden linux stemcell in CI.
/var/vcap/bosh/warden-cpi-agent-env.json
is ignored and configuration of the "blobstore" comes from the initialization functiongithub.com/cloudfoundry/bosh-agent/releasetarball.NewCompiler
. This may be surprising if someone uses the compile command on non-warden stemcells with different agent configurations.app.DefaultBaseDirectory
as the base directory for the directories provider and may mutate files there. I am not sure if this will cause problems for a running BOSH agent.bosh-agent compile
will fail windows with a helpful message. The code I wrote likely supports windows paths (I believe I used "filepath" and "path" appropriately); however, the part of the code that loads the stemcell name/version uses linux specific paths. See the new package "github.com/cloudfoundry/bosh-agent/stemcellmetadata" for details.