feat: update content_type_matching logic for binary payloads #429

YOU54F · 2024-05-28T22:40:07Z

fixes #171

Content Type - Detection Mechanisms for Binary Content

pact_matching now performs the following for matching Binary content-types

Determines expected Content-Type header requested by user in test
Read content buffer with infer library, and guess Content-Type based on magic bytes
If unsuccessful
1. Read content buffer with tree_magic_mini library, and guess Content-Type based on shared-mime-info DB
  1. MagicDB is not shipped with pact_matching, due to GPL restrictions, users can add manually
    1. Linux Alpine - apk add shared-mime-info
    2. MacOS brew install shared-mime-info
      1. arm64 MacOS requires tree_magic_mini fork
    3. Linux - Debianapt-get install -y shared-mime-info
If either result returns text/plain, then manually read bytes using detect_content_type_from_bytes function in pact_models
If all of 2, 3, or 4 fails, then throw error, otherwise return Ok

certainly better ways to implement this, but wanted to make sure I had a full round trip in all the consuming languages, and could remove the content-type hacks there.

The plus side is we have consistent behaviour in all of our test examples, on all supported platforms and architectures (win/lin/mac arm64 and amd64) without the use of tree_magic and the existence of shared-mime-info (which is great for container users, and great for avoiding GPL woes)

Rust libraries used:

https://github.com/mbrubeck/tree_magic
- fork https://github.com/you54f/tree_magic (for m1 support), PR raised
https://github.com/bojand/infer
Explore (not used but worth a look at) - https://github.com/ebassi/xdg-mime-rs

Questions

Do we need tree_magic_mini, I don't seem to hit that check in the wild
We probably want some vendored format examples that are harder to detect
Provide user full opt-out of content-type matching, in case of things going weird?

Notes to merger

will need updating in pact_mock_server (use of pact_matching) which is then used in this repo, so I believe we will need to build/release pact_matching to crates.io, update pact_mock_server, release, and then back to pact_ref to do the works there

Tested in

pact rust
pact rust compat suite
pact-js-core
- Fixes/content type detection plus aarch64 musl so YOU54F/pact-js-core#42
- Feat/alpine so YOU54F/pact-js-core#43
pact-js
- Feat/alpine content types YOU54F/pact-js#1
pact-php
- fixes: content_type_detection_plus_aarch64_musl_so YOU54F/pact-php#1
pact-net
- feat: binary matching YOU54F/pact-net#1
pact-ruby-ffi
- Fixes/content type detection plus aarch64 musl so YOU54F/pact-ruby-ffi#3

rholshausen · 2024-05-28T23:46:39Z

Looks like infer will handle the most common types, but it has all the checks build in in Rust code, not config, so it is not extensible and will not be able to detect new types unless the crate is updated. So having tree_magic as a fallback is probably a good idea.

compatibility-suite/pact-compatibility-suite/features/V3/matching_rules.feature

- use updated tree_magic with merged fix for apple silicon

rust/pact_matching/src/json.rs

… with application/json body

…ted, before we match

YOU54F · 2024-05-29T20:04:53Z

👋🏾 Tagged a few of you for review!

Always room for improvement on what is proposed but this gives us

consistent cross platform and cross arch content-type detection with the new crate for the existing raised issues
falls back to the existing behaviour we have today, with additional improvements to also add support for arm64 macos in the tree_magic_mini crate for shared-mime-info db lookups
improved debug logging, so we are aware of which content type lookups are being performed.

The compatibility suite has been updated to test cross platform (windows x86_64 + macos x86_64/arm64 ) and skipped tests for windows have been been re-instated 👍🏾

Happy to make any amendments, as I know this might not be the best solution, especially with respect to my coding style, or the overall (existing) approach to matching in this function, but please be reminded this is a massive improvement over our existing confusing behaviour, and we can always progress forwards.

YOU54F · 2024-05-29T20:07:08Z

rust/pact_consumer/Cargo.toml

@@ -30,7 +30,7 @@ itertools = "0.12.1"
 lazy_static = "1.4.0"
 maplit = "1.0.2"
 pact_matching = { version = "~1.2.2", path = "../pact_matching", default-features = false }
-pact_mock_server = { version = "~1.2.6", default-features = false }
+pact_mock_server = { version = "=1.2.8", default-features = false, git = "https://github.com/you54f/pact-core-mock-server.git", branch = "main" }


please update to released pact_mock_server, when pact_matching is released, and pact_mock_server updated to reference released crate

YOU54F · 2024-05-29T20:07:17Z

rust/pact_ffi/Cargo.toml

@@ -31,7 +31,8 @@ maplit = "1.0.2"
 multipart = { version = "0.18.0", default-features = false, features = ["client", "mock"] }
 onig = { version = "6.4.0", default-features = false }
 pact_matching = { version = "~1.2.2", path = "../pact_matching" }
-pact_mock_server = { version = "~1.2.6" }
+pact_mock_server = { version = "=1.2.8", git = "https://github.com/you54f/pact-core-mock-server.git", branch = "main" }


please update to released pact_mock_server, when pact_matching is released, and pact_mock_server updated to reference released crate

YOU54F · 2024-05-29T20:10:09Z

rust/pact_ffi/Cargo.toml

@@ -31,7 +31,8 @@ maplit = "1.0.2"
 multipart = { version = "0.18.0", default-features = false, features = ["client", "mock"] }
 onig = { version = "6.4.0", default-features = false }
 pact_matching = { version = "~1.2.2", path = "../pact_matching" }
-pact_mock_server = { version = "~1.2.6" }
+pact_mock_server = { version = "=1.2.8", git = "https://github.com/you54f/pact-core-mock-server.git", branch = "main" }
+# pact_mock_server = { version = "=1.2.8", path = "../../../pact-core-mock-server/pact_mock_server" }


path useful for dev guidelines if users are working across both repos locally.

. ├── pact-core-mock-server └── pact-reference

rholshausen · 2024-05-30T02:06:57Z

Just a note, if you want to override a dependency, the easiest way is to use the [patch.crates-io] section in the workspace Cargo.toml

tienvx

LGTM.

I wonder: since this PR is merged, when can we revert custom changes related to the pact_mock_server?

tienvx · 2024-05-30T01:57:36Z

rust/pact_matching/README.md

+      1. MagicDB is not shipped with pact_matching, due to GPL restrictions, users can add manually
+         1. Linux Alpine - `apk add shared-mime-info`
+         2. MacOS `brew install shared-mime-info`
+            1. `arm64` MacOS requires `tree_magic_mini` [fork](https://github.com/you54f/tree_magic) 


I believe a new version has been released v3.1.5, and this fork is no longer needed

it has been yes! i need to update that comment to reflect, we are consuming 3.1.5 in here now

rholshausen · 2024-05-30T04:01:14Z

Two tests are failing on Windows: https://github.com/pact-foundation/pact-reference/actions/runs/9295821959/job/25583357274

rholshausen · 2024-05-30T05:36:17Z

Two tests are failing on Windows: https://github.com/pact-foundation/pact-reference/actions/runs/9295821959/job/25583357274

All good. Mock server is using the old pact matching. Updated it and ran the tests on Windows, all pass.

see pact-foundation/pact-reference#429 for more detail

feat: update content_type_matching logic for binary payloads

968be07

YOU54F mentioned this pull request May 28, 2024

Feat multipart file upload support pact-foundation/pact-net#447

Open

1 task

rholshausen reviewed May 28, 2024

View reviewed changes

compatibility-suite/pact-compatibility-suite/features/V3/matching_rules.feature Outdated Show resolved Hide resolved

YOU54F marked this pull request as ready for review May 28, 2024 23:59

YOU54F marked this pull request as draft May 29, 2024 00:19

YOU54F added 2 commits May 29, 2024 03:31

chore: refactor to preserve existing error messages

b0fd7e2

- use updated tree_magic with merged fix for apple silicon

test: compat suite x-plat

758b81c

YOU54F commented May 29, 2024

View reviewed changes

rust/pact_matching/src/json.rs Outdated Show resolved Hide resolved

YOU54F and others added 4 commits May 29, 2024 04:03

chore: inferred_content_type default is empty string if no match

6094477

chore: only check with tree_magic if inferred_content_type is empty

57b16b8

chore: preserve original behaviour for text/plain content-type header…

72c6e57

… with application/json body

chore: refactor matching logic and add debug logs if text/plain detec…

e0ffa61

…ted, before we match

YOU54F marked this pull request as ready for review May 29, 2024 19:53

YOU54F requested review from adamrodger, mefellows, tienvx and JP-Ellis and removed request for mefellows May 29, 2024 19:53

YOU54F commented May 29, 2024

View reviewed changes

rholshausen merged commit dece679 into pact-foundation:master May 30, 2024
32 checks passed

tienvx reviewed May 30, 2024

View reviewed changes

YOU54F mentioned this pull request Jun 12, 2024

Binary Content Matching is odd - cross platform/arch pact-foundation/pact-js-core#447

Closed

YOU54F mentioned this pull request Jun 17, 2024

feat: alpine support pact-foundation/pact-js-core#507

Merged

YOU54F added a commit to pact-foundation/pact-js that referenced this pull request Jun 20, 2024

chore(test): remove content-type workarounds post fix in pact-ref

9721d29

see pact-foundation/pact-reference#429 for more detail

YOU54F deleted the feat/content_type_matching branch June 26, 2024 01:12

YOU54F mentioned this pull request Sep 6, 2024

feat: add pactffi_log_to_file for consumer/verifier pact-foundation/pact-js-core#428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: update content_type_matching logic for binary payloads #429

feat: update content_type_matching logic for binary payloads #429

YOU54F commented May 28, 2024

rholshausen commented May 28, 2024

YOU54F commented May 29, 2024

YOU54F May 29, 2024

YOU54F May 29, 2024

YOU54F May 29, 2024

rholshausen commented May 30, 2024

tienvx left a comment

tienvx May 30, 2024

YOU54F May 30, 2024

rholshausen commented May 30, 2024

rholshausen commented May 30, 2024

feat: update content_type_matching logic for binary payloads #429

feat: update content_type_matching logic for binary payloads #429

Conversation

YOU54F commented May 28, 2024

Content Type - Detection Mechanisms for Binary Content

rholshausen commented May 28, 2024

YOU54F commented May 29, 2024

YOU54F May 29, 2024

Choose a reason for hiding this comment

YOU54F May 29, 2024

Choose a reason for hiding this comment

YOU54F May 29, 2024

Choose a reason for hiding this comment

rholshausen commented May 30, 2024

tienvx left a comment

Choose a reason for hiding this comment

tienvx May 30, 2024

Choose a reason for hiding this comment

YOU54F May 30, 2024

Choose a reason for hiding this comment

rholshausen commented May 30, 2024

rholshausen commented May 30, 2024