Reproducible Builds

[keraliss]

What mobile OS are you using?

Android

Please provide details on the device you experienced the bug on

Hi, i'm Keraliss from walletscrutiny, i tried to build your app from the dockerfile provided in the repository, but it resulted in this error -

BUILD FAILED in 26m 49s
Running Gradle task 'assembleRelease'...                         1609.5s
Gradle task assembleRelease failed with exit code 1
+ '[' '!' -f /tmp/test_com.foundationdevices.envoy/build/app/outputs/flutter-apk/app-release.apk ']'
+ echo 'Error: APK file not found at /tmp/test_com.foundationdevices.envoy/build/app/outputs/flutter-apk/app-release.apk'
Error: APK file not found at /tmp/test_com.foundationdevices.envoy/build/app/outputs/flutter-apk/app-release.apk

Description

As the build from provided dockerfile failed, we created our own dockerfile based on the one provided. This is the dockerfile

# Use Ubuntu 22.04 as the base image
FROM ubuntu:22.04

# Set environment variables
ENV TZ=America/New_York
ENV ANDROID_SDK_ROOT=/root/Android/sdk
ENV PATH="$PATH:/root/Android/sdk/platform-tools:/root/Android/sdk/ndk/25.2.9519653/toolchains/llvm/prebuilt/linux-x86_64/bin:/root/flutter/bin:/root/.cargo/bin"
ENV DEBIAN_FRONTEND=noninteractive

# Install necessary packages
RUN apt-get update && apt-get upgrade -y && \
    apt-get install -y --no-install-recommends \
    postgresql curl build-essential libssl-dev pkg-config libpq-dev git unzip xz-utils zip \
    libglu1-mesa openjdk-8-jdk openjdk-17-jdk wget python2 autoconf clang cmake ninja-build \
    libgtk-3-0 libgtk-3-dev v4l2loopback-dkms v4l2loopback-utils libzbar-dev libzbar0 \
    libzbargtk-dev libjsoncpp-dev libsecret-1-dev libsecret-1-0 ffmpeg xvfb xdotool x11-utils \
    libstdc++-12-dev llvm-14 libsdl2-dev libclang1-14 libtool sudo libusb-1.0-0-dev \
    python3-virtualenv xorg xdg-user-dirs xterm tesseract-ocr ca-certificates openssl && \
    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Update SSL certificates
RUN update-ca-certificates

# Install Android SDK Command-line Tools
RUN update-java-alternatives --set /usr/lib/jvm/java-1.8.0-openjdk-amd64 && \
    mkdir -p /root/Android/sdk /root/.android && \
    touch /root/.android/repositories.cfg && \
    wget -O sdk-tools.zip https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip && \
    unzip sdk-tools.zip -d /root/Android/sdk && rm sdk-tools.zip

# Install required Android components
RUN yes | /root/Android/sdk/tools/bin/sdkmanager --licenses && \
    /root/Android/sdk/tools/bin/sdkmanager "build-tools;30.0.3" "platforms;android-30" \
    "platform-tools" "cmdline-tools;latest" "ndk;25.2.9519653"

# Install Flutter SDK
RUN git clone https://github.com/flutter/flutter.git /root/flutter && \
    flutter channel stable && \
    flutter upgrade && \
    flutter config --enable-linux-desktop && \
    flutter precache && \
    flutter doctor

# Configure Cargo to use git-fetch-with-cli and add crates.io mirror
RUN mkdir -p /root/.cargo && \
    echo '[net]' > /root/.cargo/config.toml && \
    echo 'git-fetch-with-cli = true' >> /root/.cargo/config.toml && \
    echo '[source.crates-io]' >> /root/.cargo/config.toml && \
    echo 'replace-with = "tuna"' >> /root/.cargo/config.toml && \
    echo '[source.tuna]' >> /root/.cargo/config.toml && \
    echo 'registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git"' >> /root/.cargo/config.toml

# Install Rust
RUN curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain 1.69.0 -y && \
    rustup target add aarch64-linux-android

# Clone the repository and set working directory
RUN git clone https://github.com/Foundation-Devices/envoy.git /root/repo
WORKDIR /root/repo

# Copy the build script
COPY build_ffi_android.sh /root/repo/
RUN chmod +x /root/repo/build_ffi_android.sh

# Create a new script to run the build and keep the container running
RUN echo '#!/bin/bash' > /root/build_and_keep_running.sh && \
    echo 'cd /root/repo' >> /root/build_and_keep_running.sh && \
    echo 'pkill -f cargo' >> /root/build_and_keep_running.sh && \
    echo 'rm -f /root/.cargo/.package-cache' >> /root/build_and_keep_running.sh && \
    echo './build_ffi_android.sh || {' >> /root/build_and_keep_running.sh && \
    echo '    echo "Build failed, attempting to clean and rebuild..."' >> /root/build_and_keep_running.sh && \
    echo '    cargo clean' >> /root/build_and_keep_running.sh && \
    echo '    cargo build --target=aarch64-linux-android' >> /root/build_and_keep_running.sh && \
    echo '    cargo build --target=aarch64-linux-android --release' >> /root/build_and_keep_running.sh && \
    echo '    flutter pub get' >> /root/build_and_keep_running.sh && \
    echo '    flutter build apk --release' >> /root/build_and_keep_running.sh && \
    echo '}' >> /root/build_and_keep_running.sh && \
    echo 'echo "Build process completed. Checking for APK..."' >> /root/build_and_keep_running.sh && \
    echo 'find /root/repo -name "*.apk"' >> /root/build_and_keep_running.sh && \
    echo 'echo "Versions of tools:"' >> /root/build_and_keep_running.sh && \
    echo 'cargo --version' >> /root/build_and_keep_running.sh && \
    echo 'rustc --version' >> /root/build_and_keep_running.sh && \
    echo 'flutter --version' >> /root/build_and_keep_running.sh && \
    echo 'tail -f /dev/null' >> /root/build_and_keep_running.sh && \
    chmod +x /root/build_and_keep_running.sh

# Set the new script as the CMD
CMD ["/root/build_and_keep_running.sh"]

And this is the com.foundationdevices.envoy.build_ffi_android.sh file

#!/bin/bash
set -e

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

# Build Rust libraries
log_message "Starting Rust build for aarch64-linux-android"
cargo +1.69.0 build --target=aarch64-linux-android
cargo +1.69.0 build --target=aarch64-linux-android --release

# Get Flutter dependencies
log_message "Getting Flutter dependencies"
flutter pub get

# Build the APK
log_message "Building APK"
flutter build apk --release

# Check if APK was built successfully
APK_PATH="/root/repo/build/app/outputs/flutter-apk/app-release.apk"
if [ -f "$APK_PATH" ]; then
    log_message "Build completed successfully. APK is at $APK_PATH"
else
    log_message "ERROR: APK not found at $APK_PATH"
    exit 1
fi

# Print versions for debugging
log_message "Tool versions:"
cargo --version
rustc --version
flutter --version

Please provide the steps that you've used to reproduce the issue

Docker Build and Run Commands:

docker build -t envoy_wallet -f envoy.dockerfile .

the build was successful

Successfully built 67097ce5b7b9
Successfully tagged envoy_wallet:latest

Then we tried to run the container

docker run -d --name envoy_build_container_new envoy_wallet

And got this

Error building OpenSSL:
   Command: cd "/root/repo/target/aarch64-linux-android/release/build/openssl-sys-9f47e4e5d09d2dda/out/openssl-build/build/src" && MAKEFLAGS="-j --jobserver-fds=6,7 --jobserver-auth=6,7" "make" "build_libs"
   Exit status: exit status: 2


   , /root/.cargo/registry/src/mirrors.tuna.tsinghua.edu.cn-df7c3c540f42cdbd/openssl-src-300.2.1+3.2.0/src/lib.rs:611:9
   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
   Resolving dependencies... (23.0s)
   Note: intl is pinned to version 0.19.0 by flutter_localizations from the flutter SDK.
   See https://dart.dev/go/sdk-version-pinning for details.


   Because envoy depends on flutter_localizations from sdk which depends on intl 0.19.0, intl 0.19.0 is required.
   So, because envoy depends on intl ^0.18.0, version solving failed.
   Resolving dependencies... (4.8s)
   Note: intl is pinned to version 0.19.0 by flutter_localizations from the flutter SDK.
   See https://dart.dev/go/sdk-version-pinning for details.


   Because envoy depends on flutter_localizations from sdk which depends on intl 0.19.0, intl 0.19.0 is required.
   So, because envoy depends on intl ^0.18.0, version solving failed.
   Build process completed. Checking for APK...
   Versions of tools:
   cargo 1.69.0 (6e9a83356 2023-04-12)
   rustc 1.69.0 (84c898d65 2023-04-16)
   Flutter 3.24.0 • channel stable • https://github.com/flutter/flutter.git
   Framework • revision 80c2e84975 (8 days ago) • 2024-07-30 23:06:49 +0700
   Engine • revision b8800d88be
   Tools • Dart 3.5.0 • DevTools 2.37.2

If applicable, add screenshots or screen recordings to help explain the issue.

No response

If applicable, add logs to help us diagnose the issue.

No response

Anything else?

Issues Encountered:

1. OpenSSL Build Failure:
   The build process for OpenSSL failed with the following error:

   Error building OpenSSL:
   Command: cd "/root/repo/target/aarch64-linux-android/release/build/openssl-sys-9f47e4e5d09d2dda/out/openssl-build/build/src" && MAKEFLAGS="-j --jobserver-fds=6,7 --jobserver-auth=6,7" "make" "build_libs"
   Exit status: exit status: 2
   

   This indicates potential issues with missing tools or incorrect configuration for building OpenSSL.

2. Flutter Dependency Conflict:
   There is a version conflict with the intl package. The project specifies intl ^0.18.0, but the Flutter SDK requires intl 0.19.0 due to dependencies like flutter_localizations. This conflict prevents the successful build of the APK. The specific error encountered was:

   Error resolving dependencies:
   Because envoy depends on flutter_localizations from sdk which depends on intl 0.19.0, intl 0.19.0 is required.
   So, because envoy depends on intl ^0.18.0, version solving failed.
   

   Attempts to resolve this by updating the intl version in the pubspec.yaml file led to further issues:

   Error: Version solving failed because every version of webfeed from git depends on intl ^0.18.0 and envoy depends on intl ^0.19.0.
   

Can anyone suggest on how to fix this?

Impact of the issue

None

If you found any similar or related issues, provide links to them here.

n/a

[InvertedX]

Hi @keraliss,
It looks like the Flutter version you're using to build is 3.24.0. Currently, Envoy doesn't support this version, which is causing a conflict with the intl package.

Could you please use version 3.19 and try again? You can refer to our Docker file to find the correct package version required for the build: DockerFile.

[keraliss]

Sure, thanks for the response, will try with 3.19 and revert back!

[keraliss]

hey, update for envoy, i tried build from the docker file and run the container. The build was successful, it runs as well, but was unable to generate an apk despite my efforts. would highly appreciate if you can let me know how you guys generate the apks, so i can replicate that!

Thanks in advance!

[icota]

Hi @keraliss

We run just docker-build-android and the APK and AAB files end up in the release folder. I tried building with your colleague in #1206 previously but got no feedback. Is it okay to close that issue now?

[keraliss]

heyy, quick update, we were able to build the apk, and compared it with the official app we got from play store. we found a significant diff,

Files fromBuild/assets/dexopt/baseline.profm and fromOfficial/assets/dexopt/baseline.profm differ
Files fromBuild/assets/flutter_assets/AssetManifest.bin and fromOfficial/assets/flutter_assets/AssetManifest.bin differ
Files fromBuild/assets/flutter_assets/assets/components/icons/ramp.svg and fromOfficial/assets/flutter_assets/assets/components/icons/ramp.svg differ
Files fromBuild/assets/flutter_assets/assets/components/icons/ramp_without_name.svg and fromOfficial/assets/flutter_assets/assets/components/icons/ramp_without_name.svg differ
Files fromBuild/assets/flutter_assets/fonts/MaterialIcons-Regular.otf and fromOfficial/assets/flutter_assets/fonts/MaterialIcons-Regular.otf differ
Only in fromBuild/assets/flutter_assets: integration_test
Files fromBuild/assets/flutter_assets/NOTICES.Z and fromOfficial/assets/flutter_assets/NOTICES.Z differ
Files fromBuild/assets/flutter_assets/shaders/ink_sparkle.frag and fromOfficial/assets/flutter_assets/shaders/ink_sparkle.frag differ
Files fromBuild/assets/lua/meta/art/00_musicbrainz.lua and fromOfficial/assets/lua/meta/art/00_musicbrainz.lua differ
Files fromBuild/assets/lua/meta/art/01_googleimage.lua and fromOfficial/assets/lua/meta/art/01_googleimage.lua differ
Files fromBuild/assets/lua/meta/art/02_frenchtv.lua and fromOfficial/assets/lua/meta/art/02_frenchtv.lua differ
Files fromBuild/assets/lua/meta/art/03_lastfm.lua and fromOfficial/assets/lua/meta/art/03_lastfm.lua differ
Files fromBuild/assets/lua/meta/reader/filename.lua and fromOfficial/assets/lua/meta/reader/filename.lua differ
Files fromBuild/assets/lua/modules/common.lua and fromOfficial/assets/lua/modules/common.lua differ
Files fromBuild/assets/lua/modules/sandbox.lua and fromOfficial/assets/lua/modules/sandbox.lua differ
Files fromBuild/assets/lua/modules/simplexml.lua and fromOfficial/assets/lua/modules/simplexml.lua differ
Files fromBuild/assets/lua/playlist/anevia_streams.lua and fromOfficial/assets/lua/playlist/anevia_streams.lua differ
Files fromBuild/assets/lua/playlist/appletrailers.lua and fromOfficial/assets/lua/playlist/appletrailers.lua differ
Files fromBuild/assets/lua/playlist/bbc_co_uk.lua and fromOfficial/assets/lua/playlist/bbc_co_uk.lua differ
Files fromBuild/assets/lua/playlist/break.lua and fromOfficial/assets/lua/playlist/break.lua differ
Files fromBuild/assets/lua/playlist/cue.lua and fromOfficial/assets/lua/playlist/cue.lua differ
Files fromBuild/assets/lua/playlist/extreme.lua and fromOfficial/assets/lua/playlist/extreme.lua differ
Files fromBuild/assets/lua/playlist/france2.lua and fromOfficial/assets/lua/playlist/france2.lua differ
Files fromBuild/assets/lua/playlist/jamendo.lua and fromOfficial/assets/lua/playlist/jamendo.lua differ
Files fromBuild/assets/lua/playlist/katsomo.lua and fromOfficial/assets/lua/playlist/katsomo.lua differ
Files fromBuild/assets/lua/playlist/lelombrik.lua and fromOfficial/assets/lua/playlist/lelombrik.lua differ
Files fromBuild/assets/lua/playlist/liveleak.lua and fromOfficial/assets/lua/playlist/liveleak.lua differ
Files fromBuild/assets/lua/playlist/metacafe.lua and fromOfficial/assets/lua/playlist/metacafe.lua differ
Files fromBuild/assets/lua/playlist/mpora.lua and fromOfficial/assets/lua/playlist/mpora.lua differ
Files fromBuild/assets/lua/playlist/newgrounds.lua and fromOfficial/assets/lua/playlist/newgrounds.lua differ
Files fromBuild/assets/lua/playlist/pinkbike.lua and fromOfficial/assets/lua/playlist/pinkbike.lua differ
Files fromBuild/assets/lua/playlist/rockbox_fm_presets.lua and fromOfficial/assets/lua/playlist/rockbox_fm_presets.lua differ
Files fromBuild/assets/lua/playlist/soundcloud.lua and fromOfficial/assets/lua/playlist/soundcloud.lua differ
Files fromBuild/assets/lua/playlist/twitch.lua and fromOfficial/assets/lua/playlist/twitch.lua differ
Files fromBuild/assets/lua/playlist/vocaroo.lua and fromOfficial/assets/lua/playlist/vocaroo.lua differ
Files fromBuild/assets/lua/playlist/youtube.lua and fromOfficial/assets/lua/playlist/youtube.lua differ
Files fromBuild/assets/lua/playlist/zapiks.lua and fromOfficial/assets/lua/playlist/zapiks.lua differ
Only in fromBuild/: billing.properties
Only in fromOfficial/: build-data.properties
Files fromBuild/classes.dex and fromOfficial/classes.dex differ
Files fromBuild/DebugProbesKt.bin and fromOfficial/DebugProbesKt.bin differ
Only in fromBuild/: firebase-encoders-json.properties
Only in fromBuild/: firebase-encoders.properties
Only in fromBuild/: firebase-encoders-proto.properties
Only in fromBuild/: lib
Files fromBuild/META-INF/androidx.activity_activity.version and fromOfficial/META-INF/androidx.activity_activity.version differ
Files fromBuild/META-INF/androidx.annotation_annotation-experimental.version and fromOfficial/META-INF/androidx.annotation_annotation-experimental.version differ
Files fromBuild/META-INF/androidx.appcompat_appcompat-resources.version and fromOfficial/META-INF/androidx.appcompat_appcompat-resources.version differ
Files fromBuild/META-INF/androidx.appcompat_appcompat.version and fromOfficial/META-INF/androidx.appcompat_appcompat.version differ
Files fromBuild/META-INF/androidx.core_core-ktx.version and fromOfficial/META-INF/androidx.core_core-ktx.version differ
Files fromBuild/META-INF/androidx.core_core.version and fromOfficial/META-INF/androidx.core_core.version differ
Files fromBuild/META-INF/androidx.emoji2_emoji2.version and fromOfficial/META-INF/androidx.emoji2_emoji2.version differ
Files fromBuild/META-INF/androidx.emoji2_emoji2-views-helper.version and fromOfficial/META-INF/androidx.emoji2_emoji2-views-helper.version differ
Files fromBuild/META-INF/androidx.fragment_fragment.version and fromOfficial/META-INF/androidx.fragment_fragment.version differ
Only in fromBuild/META-INF: androidx.lifecycle_lifecycle-livedata-core-ktx.version
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-livedata-core.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-livedata-core.version differ
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-livedata.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-livedata.version differ
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-process.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-process.version differ
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-runtime.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-runtime.version differ
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-viewmodel-savedstate.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-viewmodel-savedstate.version differ
Files fromBuild/META-INF/androidx.lifecycle_lifecycle-viewmodel.version and fromOfficial/META-INF/androidx.lifecycle_lifecycle-viewmodel.version differ
Files fromBuild/META-INF/androidx.media_media.version and fromOfficial/META-INF/androidx.media_media.version differ
Files fromBuild/META-INF/androidx.profileinstaller_profileinstaller.version and fromOfficial/META-INF/androidx.profileinstaller_profileinstaller.version differ
Files fromBuild/META-INF/androidx.security_security-crypto.version and fromOfficial/META-INF/androidx.security_security-crypto.version differ
Files fromBuild/META-INF/androidx.tracing_tracing.version and fromOfficial/META-INF/androidx.tracing_tracing.version differ
Only in fromBuild/META-INF: androidx.window.extensions.core_core.version
Files fromBuild/META-INF/androidx.window_window-java.version and fromOfficial/META-INF/androidx.window_window-java.version differ
Files fromBuild/META-INF/androidx.window_window.version and fromOfficial/META-INF/androidx.window_window.version differ
Files fromBuild/META-INF/com/android/build/gradle/app-metadata.properties and fromOfficial/META-INF/com/android/build/gradle/app-metadata.properties differ
Files fromBuild/META-INF/kotlinx_coroutines_android.version and fromOfficial/META-INF/kotlinx_coroutines_android.version differ
Files fromBuild/META-INF/kotlinx_coroutines_core.version and fromOfficial/META-INF/kotlinx_coroutines_core.version differ
Only in fromBuild/META-INF/services: a4.f
Only in fromBuild/META-INF/services: B4.B
Only in fromOfficial/META-INF/services: c7.f
Only in fromOfficial/META-INF/services: d8.d0
Only in fromBuild/META-INF/services: G4.t
Only in fromOfficial/META-INF/services: kotlinx.coroutines.internal.r
Only in fromBuild/META-INF/services: u3.a
Only in fromOfficial/META-INF/services: w5.a
Only in fromBuild/META-INF: version-control-info.textproto
Only in fromBuild/: play-services-basement.properties
Only in fromBuild/: play-services-base.properties
Only in fromBuild/: play-services-location.properties
Only in fromBuild/: play-services-places-placereport.properties
Only in fromBuild/: play-services-tasks.properties
Only in fromOfficial/res: anim
Only in fromOfficial/res: animator
Only in fromOfficial/res: anim-v21
Only in fromOfficial/res: color-v26
Only in fromOfficial/res: drawable
Only in fromOfficial/res: drawable-anydpi-v23
Only in fromOfficial/res: drawable-hdpi-v4
Only in fromOfficial/res: drawable-v21
Only in fromOfficial/res: drawable-v23
Only in fromOfficial/res: drawable-watch-v20
Only in fromBuild/res: E5.ogg
Only in fromOfficial/res: interpolator
Only in fromOfficial/res: layout
Only in fromOfficial/res: layout-v21
Only in fromOfficial/res: layout-v26
Only in fromOfficial/res: layout-watch-v20
Only in fromOfficial/res: mipmap-anydpi-v26
Only in fromOfficial/res: mipmap-hdpi-v4
Only in fromOfficial/res: mipmap-mdpi-v4
Only in fromOfficial/res: mipmap-xhdpi-v4
Only in fromOfficial/res: mipmap-xxhdpi-v4
Only in fromOfficial/res: mipmap-xxxhdpi-v4
Only in fromOfficial/res: raw
Only in fromOfficial/res: xml
Files fromBuild/resources.arsc and fromOfficial/resources.arsc differ
Only in fromOfficial/: stamp-cert-sha256
Only in fromBuild/: transport-api.properties
Only in fromBuild/: transport-backend-cct.properties
Only in fromBuild/: transport-runtime.properties

There were around 1000+ more lines of diff containing png, webp and xml changes. Would be great if you can look into it!

[icota]

we were able to build the apk

Did you build it by running just docker-build-android? On your website you mention a test script but it 404s.

Please tell us how you built and we'll resolve this in no time

[xrviv]

Hello @icota 😊,

Danny here from walletscrutiny as well. I was revisiting this issue and noticed something.

1. The APKs distributed from Google Play are AABs. If we used my apkextractor_sync.sh script, to extract the 4 split apks based on my phone's arch, it will output 4 split apks:

   • base.apk
   • split_config.arm64_v8a.apk
   • split_config.en.apk
   • split_config.xxhdpi.apk

2. The apk built from the previous methodology above involves building a singular apk (Is it the universal APK? I am not so sure, but it is likely - hence an explanation for the many diffs.)

3. I was able to generate the split apks from an AAB build, resulting in far less diffs, with some parts I cannot explain yet.

4. You mentioned simply running just docker-build-android but on my end, this failed within the container:

just docker-build-android
docker build -t envoy .
sh: 1: docker: not found
error: Recipe `docker-build` failed on line 9 with exit code 127

What did work for me was to execute:

root@318043c4056e:/app# ./build_ffi_android.sh
root@318043c4056e:/app# flutter build appbundle

Here are the steps I undertook to build the app:

# 0.2

# Clone the repository with submodules and checkout latest tag
git clone --recurse-submodules [email protected]:Foundation-Devices/envoy.git
cd envoy
git fetch --tags
git checkout <latest-tag>

# Build the Docker image
docker build -t envoy .

# Run the Docker container and attach
docker run -it --rm -v $(pwd):/app envoy /bin/bash

# Inside Docker: Navigate to project directory and install dependencies
cd /app
curl -fsSL https://just.systems/install.sh | bash -s -- --to /usr/local/bin
yes | sdkmanager --licenses
sdkmanager "platform-tools" "build-tools;30.0.3" "platforms;android-30"
curl -L -o /root/bundletool.jar https://github.com/google/bundletool/releases/download/1.14.1/bundletool-all-1.14.1.jar
alias bundletool="java -jar /root/bundletool.jar"

# Generate keystore and configure signing
keytool -genkeypair -v -keystore /root/android/key.jks -keyalg RSA -keysize 2048 -validity 10000 -alias dummykey -storepass password123 -keypass password123

# Modify build.gradle to build split apks
nano android/app/build.gradle

> defaultConfig {
>      applicationId "com.foundationdevices.envoy"
>      minSdkVersion 29
>      targetSdkVersion 34
>      versionCode generateVersionCode(flutterVersionName, flutterVersionCode.toInteger())
>      versionName flutterVersionName
>
>      ndk {
>            abiFilters 'arm64-v8a', 'x86_64'
>        }
>    }
>
>    signingConfigs {
>        release {
>            storeFile file("/root/android/key.jks")
>            storePassword = "password123"
>            keyAlias = "dummykey"
>            keyPassword = "password123"
>        }
>    }
>
>    packagingOptions {
>        pickFirst 'lib/**/libc++_shared.so'
>    }
>
>    buildTypes {
>        release {
>            if (!project.hasProperty("nosign")) {
>                signingConfig signingConfigs.release
>            }
>
>            minifyEnabled true
>            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
>        }
>    }
>
>    bundle {
>        language {
>            enableSplit = true
>        }
>        density {
>            enableSplit = true
>        }
>    }
>
>    namespace 'com.foundationdevices.envoy'

# substituted ./build_ffi_android.sh for just docker-build-android
./build_ffi_android.sh
flutter build appbundle

# Copy the device-spec.json from the host to the container
docker cp /path/to/device-spec.json <container_id>:/app/device-spec.json

# Generate device-specifc split APKs
bundletool build-apks --device-spec=/app/device-spec.json --output=envoy.apks --bundle=build/app/outputs/bundle/release/app-release.aab --ks=/root/android/key.jks --ks-pass=pass:password123 --ks-key-alias=dummykey --key-pass=pass:password123

# Extract split APKs
unzip envoy.apks -d split_apks

# On the host copy the AAB from the container to the host
docker cp 318043c4056e:/app/split_apks/splits /home/danny/work/builds/com.foundationdevices.envoy/2/compare/

# Exit the Docker container
exit

From there we will run a diff -r on the matching sets of split apks. (ie. arm64_v8a vs arm64_v8a)

arm64_v8a diffs

$ diff -r ~/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/ arm64_v8a/
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/AndroidManifest.xml and arm64_v8a/AndroidManifest.xml differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/lib/arm64-v8a/libapp.so and arm64_v8a/lib/arm64-v8a/libapp.so differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/lib/arm64-v8a/libbackup_ffi.so and arm64_v8a/lib/arm64-v8a/libbackup_ffi.so differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/lib/arm64-v8a/libhttp_ffi.so and arm64_v8a/lib/arm64-v8a/libhttp_ffi.so differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/lib/arm64-v8a/librive_text.so and arm64_v8a/lib/arm64-v8a/librive_text.so differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/lib/arm64-v8a/libtor.so and arm64_v8a/lib/arm64-v8a/libtor.so differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/META-INF/BNDLTOOL.RSA and arm64_v8a/META-INF/BNDLTOOL.RSA differ
diff -r /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/META-INF/BNDLTOOL.SF arm64_v8a/META-INF/BNDLTOOL.SF
3c3
< SHA1-Digest-Manifest: JKS3fuknS2RnUSQ9K3NRGtEnBv4=
---
> SHA1-Digest-Manifest: nT+qdPolSHvokiJOw7UypL1m0us=
7c7
< SHA1-Digest: T0gR69CJilFHmEyoUQgESqENYMo=
---
> SHA1-Digest: pc09KlCrZ8vfAz2GcDzmF6ozNjQ=
10c10
< SHA1-Digest: x9T+aOuTvJqtRVrSJ4NENHjUsy0=
---
> SHA1-Digest: N4yZzaiDLnXF21YQFg3cRCfPxgc=
13c13
< SHA1-Digest: khFZGFem60K0/RE5r6kaP9OSfxs=
---
> SHA1-Digest: gpKUsQEdvZN6BNRVe3ik5EjzIRk=
22c22
< SHA1-Digest: BZkkBFryu1/a9Xw5f2QXQvljH2Y=
---
> SHA1-Digest: 6tUI29gNXC6MDVd9X8O5RwctSIU=
25c25
< SHA1-Digest: brTzjvz3VjoaDUaiJEwNWLgyBgo=
---
> SHA1-Digest: GPhVBsTjGsLCa97+pR+1bK8Q1iw=
28c28
< SHA1-Digest: jIi/qZrQizask42rHiEoRdG0s9w=
---
> SHA1-Digest: +cEorGEH2WtgEP5+7NKaXpFhVmc=
41,43d40
< 
< Name: stamp-cert-sha256
< SHA1-Digest: Bs40BvDZ1xrbgQrPTD2x5w/NfHQ=
diff -r /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/META-INF/MANIFEST.MF arm64_v8a/META-INF/MANIFEST.MF
4c4
< SHA1-Digest: uC6lNLnlLIgNRbOus9lYl0quwpg=
---
> SHA1-Digest: tMPqbGeIAfcsUCKprxnFXIvzUEs=
7c7
< SHA1-Digest: alMNHcnfyb0weHzQaisvOYUVcqs=
---
> SHA1-Digest: kmpJzlPxQNevZw3DDuWIeaqz25k=
10c10
< SHA1-Digest: Uvax2MM37dKOn34Fo2ELTsdNjzs=
---
> SHA1-Digest: 8ZrSYNIsWfGgouMuCNplMJ6Pgzc=
19c19
< SHA1-Digest: zNT/doBxq0bofJ8CwzNK6qLGke8=
---
> SHA1-Digest: JIYqGHUoJMnLI6lAgLrHg4xY0Io=
22c22
< SHA1-Digest: k9R4Yq+Iz9sRfKUiRpZ+uWU+lp8=
---
> SHA1-Digest: eXW6xW7ZYA6spSF77PU/B8AMvQg=
25c25
< SHA1-Digest: S4IRRzIjBdP4rMkv8mBrQAgjb5E=
---
> SHA1-Digest: bNLLLX7r9OB+uPX9YzsIX8U4gg8=
38,40d37
< 
< Name: stamp-cert-sha256
< SHA1-Digest: 5vTkm69vqAbusD/fho7oWwc7j+I=
Only in /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/arm64_v8a/: stamp-cert-sha256

Summarized, the diffs for arm64_v8a are:

• AndroidManifest.xml
• lib/arm64-v8a/libapp.so
• lib/arm64-v8a/libbackup_ffi.so
• lib/arm64-v8a/libhttp_ffi.so
• lib/arm64-v8a/librive_text.so
• lib/arm64-v8a/libtor.so
• META-INF/BNDLTOOL.RSA
• META-INF/BNDLTOOL.SF
• META-INF/MANIFEST.MF
• stamp-cert-sha256

With the AndroidManifest.xml, the META-INF, and stamp-cert-sha256 as signing-related.

en diffs:

Summary:

AndroidManifest.xml
META-INF/BNDLTOOL.RSA
META-INF/BNDLTOOL.SF
META-INF/MANIFEST.MF
resources.arsc
stamp-cert-sha256

xxhdpi diffs:

Summary:

AndroidManifest.xml
META-INF/BNDLTOOL.RSA
META-INF/BNDLTOOL.SF
META-INF/MANIFEST.MF
resources.arsc
stamp-cert-sha256

master diffs:

$ diff -r ~/work/builds/com.foundationdevices.envoy/1/compare/official/master/ master/
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/master/AndroidManifest.xml and master/AndroidManifest.xml differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/master/res/xml/splits0.xml and master/res/xml/splits0.xml differ
Binary files /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/master/resources.arsc and master/resources.arsc differ
Only in /home/danny/work/builds/com.foundationdevices.envoy/1/compare/official/master/: stamp-cert-sha256

I do believe we're getting closer, now that we've eliminated many hundreds or thousands of diffs.

The build log:
com.foundationdevices.envoy_v1.8.4-build2-1104-1647.log

Running the diffs are in another terminal session, which I was not able to record.

[icota]

You mentioned simply running just docker-build-android but on my end, this failed within the container:

sh: 1: docker: not found

There is no container because it seems you don't have a docker install. Can you install docker and try again please? I would prefer if we could get the same artifacts before we diff.

A docker build will make sure we are using the same versions of Flutter, NDK, Java, etc

[xrviv]

Alright, will do again tomorrow - different timezones. Thank you sir.

Addendum:

• The failure was already inside a docker container.
• Strangely, on our debian build server which we ssh into, we have this:

danny@lw10:~/work/builds/com.foundationdevices.envoy/2/compare/built$ docker --version
Docker version 20.10.24+dfsg1, build 297e128

[xrviv]

So some updates:

• Container com.foundationdevices.envoy_v1.8.4.log

• Host com.foundationdevices.envoy_v1.8.4-host.log

diff master

$ diff -r built/master official/master
Binary files built/master/AndroidManifest.xml and official/master/AndroidManifest.xml differ
Binary files built/master/res/xml/splits0.xml and official/master/res/xml/splits0.xml differ
Binary files built/master/resources.arsc and official/master/resources.arsc differ
Only in official/master: stamp-cert-sha256

diff en

$ diff -r built/en official/en
Binary files built/en/AndroidManifest.xml and official/en/AndroidManifest.xml differ
Binary files built/en/META-INF/BNDLTOOL.RSA and official/en/META-INF/BNDLTOOL.RSA differ
diff -r built/en/META-INF/BNDLTOOL.SF official/en/META-INF/BNDLTOOL.SF
3c3
< SHA1-Digest-Manifest: aYrWoCU4vlJlOkudHM842ZjCqhk=
---
> SHA1-Digest-Manifest: XqGs4D09MgmtE/K2rhvDiIa1two=
7c7
< SHA1-Digest: 3qKCuuO51xzHmPJdNXhSK9JAWEE=
---
> SHA1-Digest: 2wKfJdO1fOBCmcBYYiTvOlncGz0=
10c10,13
< SHA1-Digest: ydDoOmsuLWKGdzcEFoasKRXMqgk=
---
> SHA1-Digest: VqGMhqD475opELgpHnvQAChO0Mg=
> 
> Name: stamp-cert-sha256
> SHA1-Digest: Bs40BvDZ1xrbgQrPTD2x5w/NfHQ=
diff -r built/en/META-INF/MANIFEST.MF official/en/META-INF/MANIFEST.MF
4c4
< SHA1-Digest: EDH8VjVKx8N2623goME4CdenLgw=
---
> SHA1-Digest: 8bO6ocd+h4F5ROGM+thmLxvZ+P8=
7c7,10
< SHA1-Digest: o28foFvAAGgTLC13w8kgQpa4ziE=
---
> SHA1-Digest: 8GsP8Cipotu+tQv6NY8Mo5v3KsU=
> 
> Name: stamp-cert-sha256
> SHA1-Digest: 5vTkm69vqAbusD/fho7oWwc7j+I=
Binary files built/en/resources.arsc and official/en/resources.arsc differ
Only in official/en: stamp-cert-sha256

diff arm64_v8a

$ diff -r built/arm64_v8a official/arm64_v8a
Binary files built/arm64_v8a/AndroidManifest.xml and official/arm64_v8a/AndroidManifest.xml differ
Binary files built/arm64_v8a/lib/arm64-v8a/libapp.so and official/arm64_v8a/lib/arm64-v8a/libapp.so differ
Binary files built/arm64_v8a/lib/arm64-v8a/libbackup_ffi.so and official/arm64_v8a/lib/arm64-v8a/libbackup_ffi.so differ
Binary files built/arm64_v8a/lib/arm64-v8a/libhttp_ffi.so and official/arm64_v8a/lib/arm64-v8a/libhttp_ffi.so differ
Binary files built/arm64_v8a/lib/arm64-v8a/librive_text.so and official/arm64_v8a/lib/arm64-v8a/librive_text.so differ
Binary files built/arm64_v8a/lib/arm64-v8a/libtor.so and official/arm64_v8a/lib/arm64-v8a/libtor.so differ
Binary files built/arm64_v8a/META-INF/BNDLTOOL.RSA and official/arm64_v8a/META-INF/BNDLTOOL.RSA differ
diff -r built/arm64_v8a/META-INF/BNDLTOOL.SF official/arm64_v8a/META-INF/BNDLTOOL.SF
3c3
< SHA1-Digest-Manifest: NHbPU4AA5Qfy8jtIzTHQiaNrsS8=
---
> SHA1-Digest-Manifest: JKS3fuknS2RnUSQ9K3NRGtEnBv4=
7c7
< SHA1-Digest: pc09KlCrZ8vfAz2GcDzmF6ozNjQ=
---
> SHA1-Digest: T0gR69CJilFHmEyoUQgESqENYMo=
10c10
< SHA1-Digest: N4yZzaiDLnXF21YQFg3cRCfPxgc=
---
> SHA1-Digest: x9T+aOuTvJqtRVrSJ4NENHjUsy0=
13c13
< SHA1-Digest: gpKUsQEdvZN6BNRVe3ik5EjzIRk=
---
> SHA1-Digest: khFZGFem60K0/RE5r6kaP9OSfxs=
22c22
< SHA1-Digest: 6tUI29gNXC6MDVd9X8O5RwctSIU=
---
> SHA1-Digest: BZkkBFryu1/a9Xw5f2QXQvljH2Y=
25c25
< SHA1-Digest: u8WRHTC1SCBG/8L2zzJ0FVvl6Js=
---
> SHA1-Digest: brTzjvz3VjoaDUaiJEwNWLgyBgo=
28c28
< SHA1-Digest: +cEorGEH2WtgEP5+7NKaXpFhVmc=
---
> SHA1-Digest: jIi/qZrQizask42rHiEoRdG0s9w=
40a41,43
> 
> Name: stamp-cert-sha256
> SHA1-Digest: Bs40BvDZ1xrbgQrPTD2x5w/NfHQ=
diff -r built/arm64_v8a/META-INF/MANIFEST.MF official/arm64_v8a/META-INF/MANIFEST.MF
4c4
< SHA1-Digest: tMPqbGeIAfcsUCKprxnFXIvzUEs=
---
> SHA1-Digest: uC6lNLnlLIgNRbOus9lYl0quwpg=
7c7
< SHA1-Digest: kmpJzlPxQNevZw3DDuWIeaqz25k=
---
> SHA1-Digest: alMNHcnfyb0weHzQaisvOYUVcqs=
10c10
< SHA1-Digest: 8ZrSYNIsWfGgouMuCNplMJ6Pgzc=
---
> SHA1-Digest: Uvax2MM37dKOn34Fo2ELTsdNjzs=
19c19
< SHA1-Digest: JIYqGHUoJMnLI6lAgLrHg4xY0Io=
---
> SHA1-Digest: zNT/doBxq0bofJ8CwzNK6qLGke8=
22c22
< SHA1-Digest: bu858n8D0K6CkiS8gDhh0y5wltQ=
---
> SHA1-Digest: k9R4Yq+Iz9sRfKUiRpZ+uWU+lp8=
25c25
< SHA1-Digest: bNLLLX7r9OB+uPX9YzsIX8U4gg8=
---
> SHA1-Digest: S4IRRzIjBdP4rMkv8mBrQAgjb5E=
37a38,40
> 
> Name: stamp-cert-sha256
> SHA1-Digest: 5vTkm69vqAbusD/fho7oWwc7j+I=
Only in official/arm64_v8a: stamp-cert-sha256

diff xxhdpi

$ diff -r built/xxhdpi official/xxhdpi
Binary files built/xxhdpi/AndroidManifest.xml and official/xxhdpi/AndroidManifest.xml differ
Binary files built/xxhdpi/META-INF/BNDLTOOL.RSA and official/xxhdpi/META-INF/BNDLTOOL.RSA differ
diff -r built/xxhdpi/META-INF/BNDLTOOL.SF official/xxhdpi/META-INF/BNDLTOOL.SF
3c3
< SHA1-Digest-Manifest: O8u4rkM+eHvkr5tCzvMJJ5IdQAk=
---
> SHA1-Digest-Manifest: LDKuMvAB1sMHSmvinUtbo8IaYh0=
7c7
< SHA1-Digest: x1UD6hcelQYbPP9tDaqzg8Pb//A=
---
> SHA1-Digest: +Z/sfO4Go2f08e14LuK8gUCEW1s=
139c139,142
< SHA1-Digest: r4vksVVpE3dIjBOvwQ5cB9fSzwQ=
---
> SHA1-Digest: iLvQXZsWq83EEPhPDzeQHTZ+YXQ=
> 
> Name: stamp-cert-sha256
> SHA1-Digest: Bs40BvDZ1xrbgQrPTD2x5w/NfHQ=
diff -r built/xxhdpi/META-INF/MANIFEST.MF official/xxhdpi/META-INF/MANIFEST.MF
4c4
< SHA1-Digest: XFI5Jjv+wRzIMBRkZp8fmfPaAHY=
---
> SHA1-Digest: ICEwR4fXNnxO8kBikVXmqS2BjVo=
136c136,139
< SHA1-Digest: Aatp8IOwmSg6rjaoRS9ehlZOT4g=
---
> SHA1-Digest: K27bOq1RumOmp8pBrrhmkZjAPEA=
> 
> Name: stamp-cert-sha256
> SHA1-Digest: 5vTkm69vqAbusD/fho7oWwc7j+I=
Binary files built/xxhdpi/resources.arsc and official/xxhdpi/resources.arsc differ
Only in official/xxhdpi: stamp-cert-sha256

I will give it another try tomorrow, thanks again.
The build steps I undertook are in the log, if you'd like to give some pointers, that would be welcome.

Good day! :)

[icota]

Thank you @xrviv!

We are now looking specifically into the binary diffs that you found. They are to do with our Rust libraries. We might be missing certain flags that are necessary for binary equivalence.

Will keep you posted here and thanks again.

[xrviv]

You're most welcome. And thank you for the prompt responses. I did another round for this app, this time through:
You'd be pleased to find that the results are easier to read, compared to the previous build. Have a nice weekend!

Build through 'just docker-build-android' for version 1.8.4

https://asciinema.org/a/691653

========================================
**Official APKs Hashes**
base.apk - 7d883cf4ac02b16238e2075a726db8dcefd7250f8461d06565e2daf9b520e2be
split_config.arm64_v8a.apk - c7283366b3c0857aae728dfa38d5776985362c000864d11ad95e5ea446019bae
split_config.en.apk - 2e8d46dc9d6df35d69b1f796e1ff44dd5eef92aafcb94c88159a7dd96c6527bd
split_config.xxhdpi.apk - a3830481af62bf78d71a3e628f837fc76959d38028e985945bea956f8e80a931
========================================
Built APKs:
========================================
**Built APKs Hashes**
base-arm64_v8a.apk - 3e8f3eeb18c4ef6ca109e353db416e7a7839265549881793d1d3ebcc1ca4dfbc
base-en.apk - b60c2f36cff6cd5d2d52f16c6342145b276957e9a422728bba99e2cf4abc1a0c
base-master.apk - 54602149925cb94beccc4bb1e35960e3fbfc6025c848991d3415255ac6ca6856
base-xxhdpi.apk - 71181091b49726b1f445ba842ab57057ebdb947e5f92020eafec9e046893dd5e
========================================

We then process the diffs

Differences found between /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/arm64_v8a

Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a/AndroidManifest.xml and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/arm64_v8a/AndroidManifest.xml differ
Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a/lib/arm64-v8a/libapp.so and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/arm64_v8a/lib/arm64-v8a/libapp.so differ
Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a/lib/arm64-v8a/libtor.so and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/arm64_v8a/lib/arm64-v8a/libtor.so differ
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a: META-INF
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/arm64_v8a: stamp-cert-sha256

Differences found between /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/en and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/en

Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/en/AndroidManifest.xml and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/en/AndroidManifest.xml differ
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/en: META-INF
Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/en/resources.arsc and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/en/resources.arsc differ
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/en: stamp-cert-sha256

Differences found between /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/xxhdpi and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/xxhdpi

Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/xxhdpi/AndroidManifest.xml and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/xxhdpi/AndroidManifest.xml differ
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/xxhdpi: META-INF
Binary files /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/xxhdpi/resources.arsc and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/xxhdpi/resources.arsc differ
Only in /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/xxhdpi: stamp-cert-sha256

Differences found between /tmp/test_com.foundationdevices.envoy_1.8.4/fromPlay-unzipped/base and /tmp/test_com.foundationdevices.envoy_1.8.4/fromBuild-unzipped/base were too large. We uploaded it in a file.

diff_base.txt