Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update go install for minimal images to use the existing go version #1450

Merged
merged 16 commits into from
Jul 3, 2024
Merged

Update go install for minimal images to use the existing go version #1450

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
1513776
update builder-base to pull upstream tar, unpack, and link matching e…
rcrozean Jul 1, 2024
447cf3d
update golang minimal image to pull upstream tar, unpack, and link ma…
rcrozean Jul 1, 2024
c69402b
add single arch check and remove unneeded comments
rcrozean Jul 2, 2024
51e6d9d
add checksum validation on golang download
rcrozean Jul 2, 2024
e205f59
use correct aarch64 arch for eksgo check
rcrozean Jul 2, 2024
f354d11
keep sha check but do not exit on failure for time being
rcrozean Jul 2, 2024
77e16fe
add go checksums
rcrozean Jul 2, 2024
2dbf2c1
add update checksums to update golang as well
rcrozean Jul 2, 2024
4abe0a8
update scripts to verify golang sha
rcrozean Jul 2, 2024
909f77e
comment out golang from for time being
rcrozean Jul 2, 2024
094bac3
update dockerfile to include golang checksums
rcrozean Jul 2, 2024
7df6ce0
correct path for golang checksums
rcrozean Jul 2, 2024
e77f860
remove unused file
rcrozean Jul 2, 2024
b594895
fix path to golang checksums
rcrozean Jul 2, 2024
3b4b269
update checksum golang path
rcrozean Jul 3, 2024
e264c81
call download for each arch
rcrozean Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions builder-base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,8 +359,8 @@ ENV PATH="/go/bin/:$PATH"
COPY --link --from=upx /upx /
COPY ./scripts/install_base_yum_packages.sh ./scripts/remove_yum_packages.sh ./scripts/common_vars.sh \
./scripts/install_golang.sh /
COPY $GOLANG_RPM_SOURCE_DIR/x86_64/golang*1.21*.rpm /tmp/x86_64/
COPY $GOLANG_RPM_SOURCE_DIR/aarch64/golang*1.21*.rpm /tmp/aarch64/
COPY $GOLANG_RPM_SOURCE_DIR/linux/arm64/go1.21*.tar.gz /tmp/linux/arm64/
COPY $GOLANG_RPM_SOURCE_DIR/linux/amd64/go1.21*.tar.gz /tmp/linux/amd64/
RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
/install_base_yum_packages.sh && \
/install_golang.sh $GOLANG_VERSION_121 && \
Expand All @@ -377,8 +377,8 @@ ENV PATH="/go/bin/:$PATH"
COPY --link --from=upx /upx /
COPY ./scripts/install_base_yum_packages.sh ./scripts/remove_yum_packages.sh ./scripts/common_vars.sh \
./scripts/install_golang.sh /
COPY $GOLANG_RPM_SOURCE_DIR/x86_64/golang*1.22*.rpm /tmp/x86_64/
COPY $GOLANG_RPM_SOURCE_DIR/aarch64/golang*1.22*.rpm /tmp/aarch64/
COPY $GOLANG_RPM_SOURCE_DIR/linux/arm64/go1.22*.tar.gz /tmp/linux/arm64/
COPY $GOLANG_RPM_SOURCE_DIR/linux/amd64/go1.22*.tar.gz /tmp/linux/amd64/
RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
/install_base_yum_packages.sh && \
/install_golang.sh $GOLANG_VERSION_122 && \
Expand Down
1 change: 1 addition & 0 deletions builder-base/checksums/go-go1.21-amd64-checksum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
54a87a9325155b98c85bc04dc50298ddd682489eb47f486f2e6cb0707554abf0 /home/prow/go/src/github.com/aws/eks-distro-build-tooling/builder-base/tmp/golang-downloads/linux/amd64/go1.21.11.linux-amd64.tar.gz
1 change: 1 addition & 0 deletions builder-base/checksums/go-go1.21-arm64-checksum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
715d9a7ff72e4e0e3378c48318c52c6e4dd32a47c4136f3c08846f89b2ee2241 /home/prow/go/src/github.com/aws/eks-distro-build-tooling/builder-base/tmp/golang-downloads/linux/arm64/go1.21.11.linux-arm64.tar.gz
1 change: 1 addition & 0 deletions builder-base/checksums/go-go1.22-amd64-checksum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ba79d4526102575196273416239cca418a651e049c2b099f3159db85e7bade7d /home/prow/go/src/github.com/aws/eks-distro-build-tooling/builder-base/tmp/golang-downloads/linux/amd64/go1.22.4.linux-amd64.tar.gz
1 change: 1 addition & 0 deletions builder-base/checksums/go-go1.22-arm64-checksum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
a8e177c354d2e4a1b61020aca3562e27ea3e8f8247eca3170e3fa1e0c2f9e771 /home/prow/go/src/github.com/aws/eks-distro-build-tooling/builder-base/tmp/golang-downloads/linux/arm64/go1.22.4.linux-arm64.tar.gz
104 changes: 66 additions & 38 deletions builder-base/scripts/download_golang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,44 +27,72 @@ RELEASE_NUMBER="$(echo $VERSION | cut -d'-' -f 2)"

source $SCRIPT_ROOT/common_vars.sh

function build::go::download(){
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
local outputDir=${2}
local arch=${3}

for artifact in golang golang-bin; do
local filename="$outputDir/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm"
if [ ! -f $filename ]; then
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm -o $filename --create-dirs
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm.sha256 -o $filename.sha256

if [[ $(sha256sum ${filename} | cut -d' ' -f1) != $(cut -d' ' -f1 "${filename}.sha256") ]] ; then
echo "Checksum doesn't match!"
exit 1
fi
fi
done

for artifact in golang-docs golang-misc golang-tests golang-src; do
local filename="$outputDir/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm"
if [ ! -f $filename ]; then
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/noarch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm -o $filename --create-dirs
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/noarch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm.sha256 -o $filename.sha256

if [[ $(sha256sum ${filename} | cut -d' ' -f1) != $(cut -d' ' -f1 "${filename}.sha256") ]] ; then
echo "Checksum doesn't match!"
exit 1
fi
fi
done
function build::eksgo::download() {
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
local outputDir=${2}
local arch=${3}

for artifact in golang golang-bin; do
local filename="$outputDir/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm"
if [ ! -f $filename ]; then
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm -o $filename --create-dirs
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.$arch.rpm.sha256 -o $filename.sha256

if [[ $(sha256sum ${filename} | cut -d' ' -f1) != $(cut -d' ' -f1 "${filename}.sha256") ]]; then
echo "Checksum doesn't match!"
exit 1
fi
fi
done

for artifact in golang-docs golang-misc golang-tests golang-src; do
local filename="$outputDir/$arch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm"
if [ ! -f $filename ]; then
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/noarch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm -o $filename --create-dirs
curl -sSLf --retry 5 https://distro.eks.amazonaws.com/golang-go$version/releases/$RELEASE_NUMBER/$arch/RPMS/noarch/$artifact-$version-$RELEASE_NUMBER.amzn2.eks.noarch.rpm.sha256 -o $filename.sha256

if [[ $(sha256sum ${filename} | cut -d' ' -f1) != $(cut -d' ' -f1 "${filename}.sha256") ]]; then
echo "Checksum doesn't match!"
exit 1
fi
fi
done
}

function build::go::download {
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
local outputDir=${2}
local archs=${3}

for arch in ${archs/,/ }; do
local filename="$outputDir/${arch}/go$version.${arch/\//-}.tar.gz"
if [ ! -f $filename ]; then
curl -sSLf --retry 5 "https://go.dev/dl/go$version.${arch/\//-}.tar.gz" -o $filename --create-dirs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would probably be best to check the checksums here. if you want to follow the pattern we use elsewhere, we could check those in or download them from go.dev, like we did the rpm shas above. again but not strictly required but would be at least slightly better

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the additions for checking the sha and here is the local output.

+ curl -sSLf --retry 5 https://go.dev/dl/go1.22.4.linux-amd64.tar.gz -o /tmp/test/linux/amd64/go1.22.4.linux-amd64.tar.gz --create-dirs
++ curl -sSLf --retry 5 'https://go.dev/dl/?mode=json'
++ jq -r --arg tar go1.22.4.linux-amd64.tar.gz '.[].files[] | if .filename == $tar then .sha256 else "" end'
++ xargs
+ sha256sum=ba79d4526102575196273416239cca418a651e049c2b099f3159db85e7bade7d
++ sha256sum /tmp/test/linux/amd64/go1.22.4.linux-amd64.tar.gz
++ cut '-d ' -f1
+ [[ ba79d4526102575196273416239cca418a651e049c2b099f3159db85e7bade7d != \b\a\7\9\d\4\5\2\6\1\0\2\5\7\5\1\9\6\2\7\3\4\1\6\2\3\9\c\c\a\4\1\8\a\6\5\1\e\0\4\9\c\2\b\0\9\9\f\3\1\5\9\d\b\8\5\e\7\b\a\d\e\7\d ]]

sha256sum=$(curl -sSLf --retry 5 "https://go.dev/dl/?mode=json" | jq -r --arg tar "go$version.${arch/\//-}.tar.gz" '.[].files[] | if .filename == $tar then .sha256 else "" end' | xargs)

#TODO: Add better way for checking checksums for older version.
go_major_version=$(if [[ $(echo "$version" | awk -F'.' '{print NF}') -ge 3 ]]; then echo ${version%.*}; else echo ${version%-*}; fi)
sha256sum -c $SCRIPT_ROOT/../checksums/go-go$go_major_version-${arch##*/}-checksum
fi
done
}

if [[ $ARCHITECTURE =~ "linux/amd64" ]] ; then
build::go::download "${VERSION}" "$OUTPUT_DIR" "x86_64"
fi
function download_golang {
if [[ ${VERSION:2:2} -ge "21" ]]; then
build::go::download "${VERSION}" "$OUTPUT_DIR" "$ARCHITECTURE"
else
if [[ $ARCHITECTURE =~ amd64 ]]; then
build::eksgo::download "${VERSION}" "$OUTPUT_DIR" "x86_64"
fi

if [[ $ARCHITECTURE =~ arm64 ]]; then
build::eksgo::download "${VERSION}" "$OUTPUT_DIR" "aarch64"
fi
fi
}

if [[ $ARCHITECTURE =~ "linux/arm64" ]] ; then
build::go::download "${VERSION}" "$OUTPUT_DIR" "aarch64"
fi
[ ${SKIP_INSTALL:-false} != false ] || download_golang
121 changes: 75 additions & 46 deletions builder-base/scripts/install_golang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ SCRIPT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"

VERSION="$1"

GOLANG_MAJOR_VERSION=$(if [[ $(echo "$VERSION"|awk -F'.' '{print NF}') -ge 3 ]]; then echo ${VERSION%.*}; else echo ${VERSION%-*}; fi)
GOLANG_MAJOR_VERSION=$(if [[ $(echo "$VERSION" | awk -F'.' '{print NF}') -ge 3 ]]; then echo ${VERSION%.*}; else echo ${VERSION%-*}; fi)

NEWROOT=/golang-${GOLANG_MAJOR_VERSION}

Expand All @@ -30,68 +30,97 @@ source $SCRIPT_ROOT/common_vars.sh
mkdir -p /go/src /go/bin /go/pkg

function build::go::symlink() {
local -r version=$1

# Removing the last number as we only care about the major version of golang
local -r majorversion=$(if [[ $(echo "$version"|awk -F'.' '{print NF}') -ge 3 ]]; then echo ${version%.*}; else echo ${version%-*}; fi)
mkdir -p ${GOPATH}/go${majorversion}/bin
for binary in go gofmt; do
ln -s /root/sdk/go${version}/bin/${binary} ${GOPATH}/go${majorversion}/bin/${binary}
done
if [ "$version" != "$majorversion" ]; then
ln -s ${GOPATH}/bin/go${version} ${GOPATH}/bin/go${majorversion}
fi
local -r version=$1

# Removing the last number as we only care about the major version of golang
local -r majorversion=$(if [[ $(echo "$version" | awk -F'.' '{print NF}') -ge 3 ]]; then echo ${version%.*}; else echo ${version%-*}; fi)
mkdir -p ${GOPATH}/go${majorversion}/bin

for binary in go gofmt; do
ln -s /root/sdk/go${version}/bin/${binary} ${GOPATH}/go${majorversion}/bin/${binary}
done

if [ "$version" != "$majorversion" ]; then
ln -s ${GOPATH}/bin/go${version} ${GOPATH}/bin/go${majorversion}
fi
}

function build::go::install(){
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
build::go::extract $version
build::go::symlink $version
function build::go::install {
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
build::go::extract $version
build::go::symlink $version
}

function build::go::extract() {
local version=$1
function build::go::extract {
local version=$1

if [ $TARGETARCH == 'amd64' ]; then
local arch='x86_64'
else
local arch='aarch64'
fi
mkdir -p /tmp/go-extracted
cd /tmp/go-extracted
tar -xzf /tmp/linux/$TARGETARCH/go$version.linux-$TARGETARCH.tar.gz

mkdir -p /tmp/go-extracted
for rpm in /tmp/$arch/golang-*.noarch.rpm /tmp/$arch/golang-*.$arch.rpm ; do $(cd /tmp/go-extracted && rpm2cpio $rpm | cpio -idm && rm -f $rpm); done
local -r golang_version=$(/tmp/go-extracted/go/bin/go version | grep -o "go[0-9].* " | xargs)

local -r golang_version=$(/tmp/go-extracted/usr/lib/golang/bin/go version | grep -o "go[0-9].* " | xargs)
mkdir -p /root/sdk/$golang_version
mv /tmp/go-extracted/go/* /root/sdk/$golang_version

mkdir -p /root/sdk/$golang_version
mv /tmp/go-extracted/usr/lib/golang/* /root/sdk/$golang_version
version=$(echo "$golang_version" | grep -o "[0-9].*")
ln -s /root/sdk/go${version}/bin/go ${GOPATH}/bin/$golang_version

for license_dir in "/usr/share/licenses/golang" "/usr/share/doc/golang-$VERSION"; do
if [ -d /tmp/go-extracted/$license_dir ]; then
mv /tmp/go-extracted/$license_dir/* /root/sdk/$golang_version
fi
done
rm -rf /tmp/go-extracted /tmp/linux/$TARGETARCH/go*.tar.gz
}

version=$(echo "$golang_version" | grep -o "[0-9].*")
ln -s /root/sdk/go${version}/bin/go ${GOPATH}/bin/$golang_version
function build::eksgo::install() {
# Set up specific go version by using go get, additional versions apart from default can be installed by calling
# the function again with the specific parameter.
local version=${1%-*}
build::eksgo::extract $version
build::go::symlink $version
}

# newer versions of golang stopped shipping the compiled .a lib files
# removing from old versions since we do not need these during our builds
find /root/sdk/go${version}/pkg -type f -name "*.a" -delete
function build::eksgo::extract() {
local version=$1

rm -rf /tmp/go-extracted /tmp/golang-*.rpm
}
if [ $TARGETARCH == 'amd64' ]; then
local arch='x86_64'
else
local arch='aarch64'
fi

mkdir -p /tmp/go-extracted
for rpm in /tmp/$arch/golang-*.noarch.rpm /tmp/$arch/golang-*.$arch.rpm; do $(cd /tmp/go-extracted && rpm2cpio $rpm | cpio -idm && rm -f $rpm); done

local -r golang_version=$(/tmp/go-extracted/usr/lib/golang/bin/go version | grep -o "go[0-9].* " | xargs)

mkdir -p /root/sdk/$golang_version
mv /tmp/go-extracted/usr/lib/golang/* /root/sdk/$golang_version

for license_dir in "/usr/share/licenses/golang" "/usr/share/doc/golang-$VERSION"; do
if [ -d /tmp/go-extracted/$license_dir ]; then
mv /tmp/go-extracted/$license_dir/* /root/sdk/$golang_version
fi
done

version=$(echo "$golang_version" | grep -o "[0-9].*")
ln -s /root/sdk/go${version}/bin/go ${GOPATH}/bin/$golang_version

# newer versions of golang stopped shipping the compiled .a lib files
# removing from old versions since we do not need these during our builds
find /root/sdk/go${version}/pkg -type f -name "*.a" -delete

rm -rf /tmp/go-extracted /tmp/golang-*.rpm
}

build::go::install "${VERSION}"
if [[ ${VERSION:2:2} -ge "21" ]]; then
build::go::install "${VERSION}"
else
build::eksgo::install "${VERSION}"
fi

# symlink default golang install to newroot bin
for binary in go gofmt; do
ln -s ${GOPATH}/go${GOLANG_MAJOR_VERSION}/bin/${binary} ${USR_BIN}/${binary}
ln -s ${GOPATH}/go${GOLANG_MAJOR_VERSION}/bin/${binary} ${USR_BIN}/${binary}
done

mkdir -p ${NEWROOT}/root
Expand Down
Loading
Loading