feat: download pkg

[t0rr3sp3dr0]

This patch implements support for downloading PKGs from the Mac App Store.

[t0rr3sp3dr0]

@blacktop, I implemented this in the simplest way possible to get some feedback from you and hear your thoughts on some stuff.

1. I'm not sure if we want this as a separate command ipsw download pkg or if we want to have this functionality in ipsw download ipa.

2. I think that because we are impersonating the Apple Configurator and not the Mac App Store, we don't get the MAS Receipt in the response. I can also be because, unlike the App Store, we are sending the requests to /WebObjects/MZFinance.woa/wa/volumeStoreDownloadProduct instead of /WebObjects/MZBuy.woa/wa/buyProduct. One final theory is that we don't send kbSync as a parameter on the requests and this is required to get the receipt.

3. Decrypting PKGs from ipsw requires Cgo, to perform some Objective-C calls. Another options would be to call an external binary to do that for you. For now, made ipsw write the .dpInfo and .hwInfo files write next to the .pkg so you can use this tool I built to decrypt the package: https://github.com/t0rr3sp3dr0/PKGDecrypt. All the alternatives require you to do this on a Mac, until someone reverses FairPlay.

4. For my particular use case, I want to run this inside a CI pipeline and would like to add some extra flags so I can download the packages using the salableAdamId instead of the bundleId, be able to specify an alternative appExtVrsId, get the raw response for the download request instead of downloading the files, and some way of passing and storing cookies without Keychain. An alternative would be to modularize the code and make it importable, so I can do whatever I want by having ipsw as a dependency in my project. I can always fork the project if you think this is too specific to be merged here.

Let me know what you think about these points.

[blacktop]

I'll try and run this soon. I'm still traveling etc.

1. I guess pkg is macOS only 'ipa' but pkg is too generic so I'd say either ipsw download mas or we have ipsw download store and then maybe ipa and pkg sub-commands

[blacktop]

2. Have you experimented with those options to get the receipt?

[blacktop]

3. I have a lot of lil cod's that use cgo so we can definitely support that. I always feel very gross calling other binaries and avoid it if I can

[blacktop]

4. Cobra supports 'hidden' flags and I've used them before for undocumented test things etc in the past

[t0rr3sp3dr0]

2. I tried sending download requests as the App Store, the Apple Configurator, and the Books Store using this tool I built: https://github.com/t0rr3sp3dr0/MASDownloadMetadata. The only responses that come with the receipt are the ones made on behalf of the Mac App Store.

• App Store

• Apple Configurator

• Books Store

[t0rr3sp3dr0]

2. I experimented changing the User-Agent to MacAppStore/3.0 (Macintosh; OS X 10.14.6; 18G103) AppleWebKit/4607.3.9 CommerceKit/708.5 but all authentication attempts failed with 403. I believe there is a different authentication flow specific to the MAS and more time would be needed to reverse it.

HTTP/1.1 403 Apple WebObjects 5.8.59-b05
Server: Apple
Date: Tue, 21 May 2024 02:28:21 GMT
Content-Length: 0
Connection: keep-alive
x-responding-instance: MZFinance:541067:::
apple-timing-app: 3 ms
cache-control: no-cache, no-store
g: t
edge-control: no-store
x-apple-lokamai-no-cache: true
Set-Cookie: ns-mzf-inst=105-88-443-54-221-9531---st44; version=1; Max-Age=1800; path=/WebObjects/; domain=.apple.com; httponly
Strict-Transport-Security: max-age=31536000; includeSubdomains
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

I think it is okay to compromise on the App Receipt for this first version. The impact of not having the receipt is that paid apps that look for it will not work.

[t0rr3sp3dr0]

2. I forgot to mention that when I sent requests as the Apple Configurator and the Books Store, kbsync was sent as part of buyParameters in the request. So that alone is not what makes the App Receipt come in the response, but it still can be part of the requirements. If that's true, we may be out of luck implementing this purely in Go, even if we figure out the Mac App Store authentication.

[t0rr3sp3dr0]

2. I did some analysis on the Mac App Store of OS X Mavericks and it looks like we need to add the X-Apple-ActionSignature header to be able to authenticate as the App Store. This basically signs the request body and any change to its raw content, even just adding or removing a whitespace, invalidates the signature.

curl \
    -X 'POST' \
    -i 'https://buy.itunes.apple.com/WebObjects/MZFinance.woa/wa/authenticate' \
    -H 'X-Apple-ActionSignature: ApihkrdgFfk3AWg/9leQfJ/10TgL5bAXRnTQOf4Ua/Z8AAAB0AMAAAABAAABAMeUUZwgWi3XDfewdH6biENNaTuQg9Hi2wr0svhmIhbwCTK0ILz9aADHKY7UH0Pvi6YqB/vfLc3gk/9C4VYQXe5g0KczEZcvuQUt+jKc4JWw27n2rctwaFFeb77FTbQagYjP6+L/NKkq0eQGoWDZJ+HLjhdUcmDvTzbzCK8toZtaOB/nllKdTboyIcYP2vX0tWBd8Q/omlPLdyaOiass3fInO7NFnOAQMfJVKvlAlipk7iQD9Zln5AYP9kfHTD+ih7FSXEBOkpq8P7Eq8WPuDRGrNjebuhq0K1b3AvhqkMX69IciySxcZsMDySTDI7qfxqFxaFoFH1f9mp/1jwbo3pUAAAARRtt+jeHNsLhsz/6hhs5ANE8AAACfAYRqkNSp8/hlGy2/BaoN5Ds0hy6mAAAAhgMEKO5SiJtb/y//nFhEOAYOpWZQXG3fhaiD+tJNTEDCzPtpRtYN8T1pvkEPF1PQk6cFooPccXNLteoLuyFqw9Qsba36Qc1LoYKwfrwRNcBEKp1RggQiBLCl53bJldaJmT1d9ijDvWjhx5lCTnTUGy0ohpms35/U1Fl0sJq92SdEYrLOalqgAAAAAAAAAAAAAAAAAAAA' \
    -H 'User-Agent: MacAppStore/1.3 (Macintosh; OS X 10.9.5) AppleWebKit/537.78.2' \
    -d '<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>appleId</key>
	<string>***</string>
	<key>attempt</key>
	<string>2</string>
	<key>guid</key>
	<string>***</string>
	<key>password</key>
	<string>***</string>
	<key>rmp</key>
	<string>0</string>
	<key>why</key>
	<string>signIn</string>
</dict>
</plist>
'

Right before making the authentication request, the MAS performed two other requests that look interesting.

1. GET https://s.mzstatic.com/sap/setupCert.plist

• Response

<plist>
<dict>
<key>sign-sap-setup-cert</key>
<data>AQIAAAQWMIIEEjCCAvqgAwIBAgIBHDANBgkqhkiG9w0BAQUFADBiMQswCQYDVQQGEwJVUzETMBEGA1UEChMKQXBwbGUgSW5jLjEmMCQGA1UECxMdQXBwbGUgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkxFjAUBgNVBAMTDUFwcGxlIFJvb3QgQ0EwHhcNMTEwMTI2MTkwMTM0WhcNMTkwMTI2MTkwMTM0WjCBhTELMAkGA1UEBhMCVVMxEzARBgNVBAoMCkFwcGxlIEluYy4xJjAkBgNVBAsMHUFwcGxlIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MTkwNwYDVQQDDDBBcHBsZSBTeXN0ZW0gSW50ZWdyYXRpb24gQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDa4A+Yl8tYKYYqC7ieGVoxwy0OaixSAe4dA/uCQWnNUCY2ercMbw45A7jUGFajCLI8w/s2QeTXyGdgMgtOMn2H9/3NU7AauvwfbMlFB62COPOofMROwrFW2T6ybW0EQRrBmkfArBV8LXiRqweiZbF6g92YS3dA2O5Q68drWAgGl1dVfSf4Cua1Feenk/nxgOZCeT8W0zKdEXZBKQoxCe8PW/jzp6n3Ug27+C10rKZJHx/OewWnhT2+z6KnqiOFZv7FFhJ+W+Ixd5ECCd9+fuSK4OxBrBcsBOC8eaSJeEQGiztLoLyE4rCCtTK+BBwD7YI+dTcUz3WfgjFtzwkUhtEnAgMBAAGjga4wgaswDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFPAwc2Py7x2szOYJMsH6eXqxaVBoMB8GA1UdIwQYMBaAFCvQaUeUdgn+9GuNLkCm90dNfwheMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly93d3cuYXBwbGUuY29tL2FwcGxlY2Evcm9vdC5jcmwwEAYKKoZIhvdjZAYCBAQCBQAwDQYJKoZIhvcNAQEFBQADggEBAD17j60fDCKKm0ujz/grsB9o4Qz3nCSDFgMt07Ko0EPorzyXJsit1SzETFVTAUnQ4rT75tty0Zi7/JvITrePzGWGf0S52icqTt/L39N930Fx+LPAHaIKM7nsK8Vzcvvhyl2OLzT0a8RPD8iKrA/7byVut66Ox+QCuCBOXVZMSZexJHR+yZOTNIyZ0afAHNPUwq5p65+fV+Jox8rVxSKCZEFY/njRysH5NmprRPezhnJ6ZEAXMZ28rHXw+jNR5b0Balg/8ACumVwKwsnpXhyHAuygCFVBKpuMZIWOUAPN4BGvznIZ61Lzr5Ktky6Undav/8Am8d6UkhzZvD02zFX6ONsAAAUxMIIFLTCCBBWgAwIBAgIISnQ/0ZM7Y5AwDQYJKoZIhvcNAQEFBQAwgYUxCzAJBgNVBAYTAlVTMRMwEQYDVQQKDApBcHBsZSBJbmMuMSYwJAYDVQQLDB1BcHBsZSBDZXJ0aWZpY2F0aW9uIEF1dGhvcml0eTE5MDcGA1UEAwwwQXBwbGUgU3lzdGVtIEludGVncmF0aW9uIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTExMDQxNDIyMTE1NVoXDTE0MDQxMzIyMTE1NVowaTEdMBsGA1UEAwwURFJNIFRlY2hub2xvZ2llcyBBMDExJjAkBgNVBAsMHUFwcGxlIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MRMwEQYDVQQKDApBcHBsZSBJbmMuMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL4tkXdyBqnMthEsRK5gt/jeP83WbyB6RoltOli/TPr+DPipyRVPqSn391cfFiPVAEcZ/d8OI8hdTc5gdxTasGug6RLkC0prjq/Tf67ZQxG1X082Vr2T1ZWZa6iYid5v3UmtM+fKN6XCb+3Gj0opLqwXxTrpbfogirVOar2cIlJOdxyQj+ZCZla6NhvjBeCDMUCOMkV8zU7qtLIwIL0tEGX1wgiCUdYOljNPU1j1SMHeWruhqn47Y3inQRMTBGyb/kX9p1LEqrBbEoY0S/TU1pV+BaZS1HzkoXi9YJyL0/9K8j4zLNf7oMcGFj7T+c6QZxYSBTUBrj3qUMQFUk125zUCAwEAAaOCAbowggG2MB0GA1UdDgQWBBQMbHKfljPJRvMMXWJDZxYlhrNcVDAMBgNVHRMBAf8EAjAAMB8GA1UdIwQYMBaAFPAwc2Py7x2szOYJMsH6eXqxaVBoMIIBDgYDVR0gBIIBBTCCAQEwgf4GCSqGSIb3Y2QFATCB8DAoBggrBgEFBQcCARYcaHR0cDovL3d3dy5hcHBsZS5jb20vYXBwbGVjYTCBwwYIKwYBBQUHAgIwgbYMgbNSZWxpYW5jZSBvbiB0aGlzIGNlcnRpZmljYXRlIGJ5IGFueSBwYXJ0eSBhc3N1bWVzIGFjY2VwdGFuY2Ugb2YgdGhlIHRoZW4gYXBwbGljYWJsZSBzdGFuZGFyZCB0ZXJtcyBhbmQgY29uZGl0aW9ucyBvZiB1c2UsIGNlcnRpZmljYXRlIHBvbGljeSBhbmQgY2VydGlmaWNhdGlvbiBwcmFjdGljZSBzdGF0ZW1lbnRzLjAvBgNVHR8EKDAmMCSgIqAghh5odHRwOi8vY3JsLmFwcGxlLmNvbS9hc2ljYS5jcmwwDgYDVR0PAQH/BAQDAgWgMBMGCiqGSIb3Y2QGDAIBAf8EAgUAMA0GCSqGSIb3DQEBBQUAA4IBAQCzR9MjeoFy1yK32EyHlabYoUy1XY+OfcDN/qZkDJj3XJVKKrYVjfGtsyCQdMFaRhi9ItDUfk+GNjRC8rp6F3vaWqghiSyxQ9YlPKq+jVQGRBQbgIcSqLmso3b21KrJU3mdGcjgCwt7AIV8/sZAKuSgExUOb+RVF0clhTiWRdolLcdsYTM6Bq8LkZ8Ts/7CnwN0VC6Txp9gR+vuK1YPpVni9mxqGsiA1C2U42QZAYAytXYSHC7G25kCf4coEXOZRSCaDwWs9Aft0NMeLzLstJyKKRdGI1P9ynH6FM+BNhpWdL8ej9BpdNHYsVQTb8KrHSyxiHp11wYrOFZlGB58wcWr</data>
</dict>
</plist>

2. POST https://play.itunes.apple.com/WebObjects/MZPlay.woa/wa/signSapSetup

• Request

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>sign-sap-setup-buffer</key>
	<data>
	AQhKdD/RkztjkJ7mPuEMP55hQlU0+5QpqEJPEkZPVoUyCMrZqn+iLDK5D7H3pDBDG2iP
	TkD+/lh4vG2Uwx1TWuhi8UvMD87W3dfzNo4TuMNPKdX1smzdyMtwz/zhT27m8A3HyDp5
	uXIF7Jfl2MPlyBMrEjjKUD3jzd7gdZ8c+qOAZbI+o784fJreifNij+XSIPNojlEQPgkN
	DxHViOLIK6N/2BJ64MHgIcV/8thju3ij4ua/kO5Frq77P3SHBFrdpwOa+NdebpuI8Ux5
	Wv+4SKgqH4DwMnyyLzKnf/npocTzNoF7vTqq1W8oxTI/q+4daqgaItC/6DebxjV+EzCj
	JOYgZrURzLez4buylvhXdJiWwnsXl3S4k26XAAAAMBqOmFtYyU89G1viAUMEDnPB0LSS
	n33UQOFgslX17sPkI8m6Cn7S5cOZlwQZK1bsGKbBwbVVI1cKgH6AzyKmtmYEpYsT
	</data>
</dict>
</plist>

• Response

<plist>
<dict>
<key>sign-sap-setup-buffer</key>
<data>AsC1MFA+NMqN9r93PrIBSbAAAATgC31iU1uEq08p3PuP2cISZm+qQc5nYPz+bA962LF3RuxyvgsLMc0mAGBFGkomhpFjgU4hlCZfun4n/Z/TcGk+liahJPjCn850xP51jLrFBjJIQW9oTWFYqwpVmZWaUnsbtya9mPMM/Pqz/0YyR+xMiqvvN8CVvkQ7uaJjuGI1Hlped5Z/QOHM10fhd0CeR7XPEdLHKHEz7jkw+aAb2nUSLCyDdpuqoi8wFx/7cLFlR6NMS5nz6mNCMCsfryUw4biIuAPhwPq5HamMSG/v6vTjal7rgEhBSBll2HnzusziXElifEvTEy/CJvMjh6aHtXoxoTSRBdtlTs2feX0OJnmwqf938nR9haG5LVQKdrvsq4Ky94Wj0yCkUe0Q3VFZjOYKREr9YD2KdEpD2/s6Nmdim0TYqrdf+aOBnEgXitNYsQsTVPKurrbm+OCUkCadNekY0EVhc1wOlRfI6cDhk2O2/3J8eo1c7EFMr8Yc10Sox82gvSLLzgS0Kf1juImXM+OoQWuokS+jhu8EH9w/KDBHC5kKJusQJ6E5odJPGG1QebPKCYyV/osHqKWKeqthbfLBbQWWbzqbcpt5i03UW7pHhE0AMp2x7m3/OLqru28lpQ9I1o0/ZD7Cipgtsg/YN2tZ1paO9rrHUVS1A4mmrj47TjPyJL3+qDQ3yL/PzTZqI90HvznQKxnby6gO8vMERNjmDssIPYi4/rk8hJWxyXw8qe6AvfRhXievIZWtYajhuH1oN/HSt1o71LT1sqg48AEl5f1xJuaUWzb3EVlb/5BVMX4dYmufOXfZqScjckIkXStx5XgNHbibcCRMnDqgYDWT+ydSf0qC7FAVSbqvZSvVir6mdRn2HZ33Rc9GjyBaKUEjnnB2dF4tjfRDi/NxenCtqopJ95cBozcODFT0/0ae6oDKZHghacA+51j4zbZ5ptiIWjb3dvuMBUpz0o9AVufbZkhGsyzCeUu56vebMbwKxrhMAYvfWq1d4XRps8VhWfwsS6DKhJO53lgoDIVvVbzmoxS2OGVCMwyN6XBBqP13bvtsYaHytICkqsQpKJNbVishfW2OTytDDv9pDlrlkz/3KivVQffcjYiUsgcT1YmFiFqTMqDFljetMFUK9T+pQfldk7noLwEzFFjuwbDskL5637C+GzaLQRhdkotsDrlTa6v9GouvMz3+/GXj0K4N2LHWTZHxDdsnAVvcJBl4zPC6WJvhePfwCmy+Lvdt6MJRnR9LLihsbHeu934vnJZ0jdKxrY8ZAFuewL1CKzN54hxyIfFtPQiPn8bhPz9PVPWitSr7hl68999p2TWFtPO42esOR4LbxBIseLY/eFMjmvpzEGfjAZegsyh+yTzrux7nYmdKRpDuSVDiBHnBCjn071VVbVt8zFR75gaBY+i01yvnpxClpBVTiBE6fpLPxi2xZ+z7m4b3py21hXhWs5ZrVlg1DcNN9fdLSehY4yWSrhxe1M0/+ZGFjbovttW+9BgC+WykpLUkSzHN3qJnHTpoFyp8Xq9AxE8ZmKngh3uEtTRVSNYkx+6A93MHu+y5vFgRyS7xAdy8FXiuizHengSvo3FTIUQbDA43jjZQvX5c1+v4dV7var/zqjrKsQLWGHQ5/DA3jMe4e5apJ4wG5yvK4VLeYH423vyVNZaUF8ahTMQ+AYRqkNSp8/hlGy2/BaoN5Ds0hy6mAAAAhgMEKO5SiJtb/y//nFhEOAYOpWZQXG3fhaiD+tJNTEDCzPtpRtYN8T1pvkEPF1PQk6cFooPccXNLteoLuyFqw9Qsba36Qc1LoYKwfrwRNcBEKp1RggQiBLCl53bJldaJmT1d9ijDvWjhx5lCTnTUGy0ohpms35/U1Fl0sJq92SdEYrLOalqg</data>
</dict>
</plist>

I'm not entirely sure yet, but it looks like FairPlay may be not involved and we might be able to implement the signing algorithm.

[t0rr3sp3dr0]

2. I talked too soon... This looks very promising, but unfortunately FairPlay is involved: https://github.com/adm1n007/pingguo_jihuo/blob/6bd9a88036aa7f64f71c93852cc872b7215e8cbd/ituneslib/login.go#L70

   Given a Mac is already needed to decrypt packages and you are willing to use CGO, we can definitely implement this. But I was hoping to make this work on Linux, at least the metadata retrieval part.

   I think we can have a subset of the functionality available on any OS by impersonating the Apple Configurator, and impersonate the App Store when running on macOS to allow package decryption, retrieval of the MAS Receipt, and download of older versions of apps.

[blacktop]

2. I talked too soon... This looks very promising, but unfortunately FairPlay is involved: https://github.com/adm1n007/pingguo_jihuo/blob/6bd9a88036aa7f64f71c93852cc872b7215e8cbd/ituneslib/login.go#L70

   Given a Mac is already needed to decrypt packages and you are willing to use CGO, we can definitely implement this. But I was hoping to make this work on Linux, at least the metadata retrieval part.

   I think we can have a subset of the functionality available on any OS by impersonating the Apple Configurator, and impersonate the App Store when running on macOS to allow package decryption, retrieval of the MAS Receipt, and download of older versions of apps.

Not sure if this is too heavy handed (and I've not researched FairPlay) but I do use the unicorn emulator in another cmd. If we are only calling a few functions in FP maybe we could emulate it?

[t0rr3sp3dr0]

I thought a lot about emulation this past week, but I didn't know about Unicorn. It's a really amazing tool. But to use it, we first need to reverse some client interfacing with the CoreFP library so we can limit the scope of the emulation.

What I have implemented in Objective-C is mostly calls to private frameworks methods that interface with the commerce process via XPC. That process is where the real magic happens.

The only exception to this is the PKG decryption tool I made. I load the storedownloadd binary into memory and call some of its internal methods to decrypt the PKG using CoreFP. So everything happens within my process and no remote connection is made.

But CoreFP is heavily obfuscated and so are the parts of commerce and storedownloadd that interface with it. It will probably need a significantly amount of effort and time to understand it.

Based on what I've seen, most people just gave up understanding it and just use some binary that already communicates with CoreFP to perform the desired action. For X-Apple-ActionSignature specifically, all implementations that I could find used some DLL from iTunes for Windows and called its methods. Just like I did with storedownloadd. This one seems to have successfully patched all Windows API calls to make it portable: https://github.com/main--/eye-oPEner.

[t0rr3sp3dr0]

For macOS, I just built this: https://github.com/t0rr3sp3dr0/SAPSigner. I was able to successfully authenticate as the App Store after using it to sign the request body and placing the signature in the X-Apple-ActionSignature header.

I still need to verify whether this header is also necessary for the buyProduct operation and if that operation also requires kbSync to be provided.

[t0rr3sp3dr0]

I just finish my tests and the buyProduct doesn't require X-Apple-ActionSignature or kbSync and returns the app receipt in its response.

That's all you need to do:

import requests

s = requests.session()
s.headers['Content-Type'] = 'application/x-apple-plist'
s.headers['User-Agent'] = 'MacAppStore/1.3 (Macintosh; OS X 10.9.5) AppleWebKit/537.78.2'

r = s.post('https://p54-buy.itunes.apple.com/WebObjects/MZFinance.woa/wa/authenticate', data=b'<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>appleId</key><string>***</string><key>attempt</key><string>1</string><key>guid</key><string>***</string><key>password</key><string>***</string><key>rmp</key><string>0</string><key>why</key><string>signIn</string></dict></plist>', headers={'X-Apple-ActionSignature': '***'})
print(r.text)

r = s.post('https://p54-buy.itunes.apple.com/WebObjects/MZBuy.woa/wa/buyProduct', data=b'<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>guid</key><string>***</string><key>price</key><string>0</string><key>pricingParameters</key><string>STDRDL</string><key>productType</key><string>C</string><key>salableAdamId</key><string>682658836</string></dict></plist>', headers={'X-Dsid': '***'})
print(r.text)

[t0rr3sp3dr0]

I forgot to mention, the server uses the macOS version from the User-Agent to determine if the app is compatible or not before proceeding.

If the OS version is not compatible but there is an older version of the app that is compatible with your OS, it expects you resend the requests with the appExtVrsId provided. This is the response it sends to you:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
<dict>
<key>pings</key>
<array></array>
<key>metrics</key>
<dict>
  <key>dialogId</key><string>MZCommerce.OsRequiresEarlierAppVersion.OSX</string>
  <key>message</key><string>Download an older version</string>
  <key>options</key>
  <array>
    <string>Download</string>
    <string>Cancel</string>
  </array>
  <key>subjectId</key><string>adamId:682658836</string>
  <key>actionUrl</key><string>p54-buy.itunes.apple.com/WebObjects/MZBuy.woa/wa/buyProduct</string>
  <key>asnState</key><integer>0</integer>
  <key>mtTopic</key><string>xp_its_main</string>
  <key>eventType</key><string>dialog</string>
</dict>
<key>failureType</key><string></string>
<key>customerMessage</key><string>Download an older version of GarageBand?</string>
<key>m-allowed</key><false/>
<key>dialog</key>
<dict><key>m-allowed</key><false/>
<key>isFree</key><true/>
<key>message</key><string>Download an older version of GarageBand?</string>
<key>explanation</key><string>The current version requires OS X 13.5 or later, but you can download the last compatible version.</string>
<key>defaultButton</key><string>ok</string>
<key>okButtonString</key><string>Download</string>
<key>okButtonAction</key><dict><key>kind</key><string>Buy</string>
<key>buyParams</key><string>pricingParameters=STDRDL&amp;salableAdamId=682658836&amp;guid=***&amp;appExtVrsId=813573506&amp;price=0&amp;productType=C</string>
<key>itemName</key><string>null</string>
</dict>
<key>cancelButtonString</key><string>Cancel</string>
<key>initialCheckboxValue</key><true/></dict>
<key>cancel-purchase-batch</key><true/>
</dict>
</plist>

If has never been a version of the app compatible with your system, this is what you get:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<plist version="1.0">
<dict>
<key>pings</key>
<array></array>
<key>metrics</key>
<dict>
  <key>dialogId</key><string>MZCommerce.ProductRequiresLaterStoreVersion</string>
  <key>message</key><string>This application requires</string>
  <key>options</key>
  <array>
    <string>OK</string>
  </array>
  <key>subjectId</key><string>adamId:682658836</string>
  <key>actionUrl</key><string>p54-buy.itunes.apple.com/WebObjects/MZBuy.woa/wa/buyProduct</string>
  <key>asnState</key><integer>0</integer>
  <key>mtTopic</key><string>xp_its_main</string>
  <key>eventType</key><string>dialog</string>
</dict>
<key>failureType</key><string></string>
<key>customerMessage</key><string>This application requires macOS 13.5 or later.</string>
<key>m-allowed</key><false/>
<key>dialog</key>
<dict><key>m-allowed</key><false/>
<key>isFree</key><true/>
<key>message</key><string>This application requires macOS 13.5 or later.</string>
<key>explanation</key><string>You must update to macOS 13.5 in order to download and use this application.</string>
<key>defaultButton</key><string>ok</string>
<key>okButtonString</key><string>OK</string>
<key>initialCheckboxValue</key><true/></dict>
<key>cancel-purchase-batch</key><true/>
</dict>
</plist>

I set the version of macOS to INT32_MAX and I was able to download any app I wanted. Even apps that are no longer supported by the most recent versions of macOS, like iPhoto. If we set the version to any number larger than 2147483647, the server returns 500.

s.headers['User-Agent'] = 'MacAppStore/1.3 (Macintosh; OS X 2147483647) AppleWebKit/537.78.2'

[blacktop]

@t0rr3sp3dr0 do you use Discord at all?

[t0rr3sp3dr0]

@blacktop, I do! We can chat over there if you want. I'm @t0rr3sp3dr0 there and pretty much everywhere else 😂

[t0rr3sp3dr0]

Btw, I just finished reversing the client-side interface of CoreFP. Based on the logs I found, this component seems to be called Mescal Client and is present on applications that talk with CoreFP.

I have reimplemented the SAPSigner in pure C with what I've learned: https://github.com/t0rr3sp3dr0/SAPSigner/blob/master/Sources/SAPSignerAlt/main.c. But it still relies on CommerceKit (from OS X Mavericks) or any library containing the Mescal Client. Newer versions of CommerceKit don't include the Mescal Client, but I was able to use a copy of the library from Mavericks on Sonoma without a problem, indicating that the CoreFP interface is super stable and hasn't changed since 10.9. On macOS Sonoma, I found the Mescal Client on the Osprey Framework, but its symbols are marked as local-only and we would need to use dlopen and dlsym to access those functions.

This C implementation will help a lot with the emulation idea, as we no longer need to worry about the Objective-C runtime and all async stuff that happens there. But I still would like to understand the interface of CoreFP library and completely bypass the Mescal Client.

[blacktop]

I think this is related to the same reason the ipa downloads stoped working?

[blacktop]

it might be nice to get the PR refactored to support both and merged? Would you have time to support that?

[blacktop]

I wouldn't mind this being macOS only and using dlopen. I think you might have solved your original goal another way by now? lmk if I'm wrong