Add experimental support for Clickteam installers

[sebamarynissen]

This PR adds experimental support for Clickteam installers (see #2) with the cicdec tool.

I am a total novice to Scala, so I had to search my way through the code a bit, but I've ended with something that works. I couldn't find a way to pass the type of the asset all the way down to the WrappedArchive.apply() method (as proposed in #2), so the solution I went for is to just try 7zip first, and if it fails try cicdec.

Currently this requires the cicdec.exe to be in the PATH variable, but it could be possible to bundle cicdec.exe with the tool as well of course.

I've tested this with the following yaml file:

group: aaron-graham
name: gracie-manor
version: 1.0.0
subfolder: 200-residential
info:
  summary: NYBT Gracie Manor - Residential
  author: Aaron Graham
  website: https://community.simtropolis.com/files/file/28981-nybt-gracie-manor/

variants:
  - variant: { nightmode: dark }
    dependencies: ["simfox:day-and-nite-mod"]
    assets:
      - assetId: aaron-graham-gracie-manor-darknite
  - variant: { nightmode: standard }
    assets:
      - assetId: aaron-graham-gracie-manor-maxisnite

---
assetId: aaron-graham-gracie-manor-darknite
version: 1.0.0
lastModified: "2016-10-06T23:02:05Z"
url: https://community.simtropolis.com/files/file/28981-nybt-gracie-manor/?do=download&r=163407
type: clickteam-installer

---
assetId: aaron-graham-gracie-manor-maxisnite
version: 1.0.0
lastModified: "2016-10-06T23:02:05Z"
url: https://community.simtropolis.com/files/file/28981-nybt-gracie-manor/?do=download&r=163406
type: clickteam-installer

Known issues:

• It looks like the cicdec tool does not work with filepaths longer than 260 characters. This is a bit problematic because the file paths can get quite long in the cache. I will see if I can find a workaround with using a different temp folder if it is detected that the file path is too long.

[memo33]

Apparently, cicdec works fine on Linux, as long as the mono .NET runtime is installed. This should work on macOS as well. I've updated the branch a bit with this in mind. In particular, I've added environment variables SC4PAC_MONO_CMD and SC4PAC_CICDEC_CMD.

I haven't been able to replicate the 260 character issue on Linux. Could you confirm that this is still a problem and is due to the length of the paths and not some other special characters? Does the limit apply to the input file, the output directory or the cicdec.exe itself?

I've also noticed another small problem. Cicdec doesn't detect that it runs non-interactively and may sometimes prompt for confirmation if there are duplicate or pre-existing files. This problem is minimized by extracting to a random temp folder. If a single installer file contains duplicates, the command should timeout and fail.

[sebamarynissen]

Sorry for the delay, had some other things on my mind last week. I can confirm that it is still a problem due to the length of the path. It seems that the problem is that the path cicdec tries to extract to is too long. I tried it for example with 2639 and 2641 jerome avenue and put the installer at the location C:\Users\sebam\Documents\temp\thisisaclickteaminstallerwithanextremelylongnamethatiswaytooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooolongforwindowstohandle.exe (a length of 247 characters). Windows is able to handle this, but as soon as I run

cicdec thisisaclickteaminstallerwithanextremelylongnamethatiswaytooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooolongforwindowstohandle.exe

in that folder, it gives the following error:

1/13    CS$$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
2/13    CS$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
3/13    NYBT 2639 And 2641 Jerome Avenue01.SC4Model
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
4/13    NYBT 2639 And 2641 Jerome Avenue02.SC4Model
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
5/13    NYBT 2639 And 2641 Jerome Avenue03.SC4Model
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
6/13    NYBT 2639 And 2641 Jerome Avenue04.SC4Model
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
7/13    R$$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
8/13    R$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
9/13    images\Dark Night.jpg
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
10/13   images\Day.jpg
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
11/13   images\Maxis Night.jpg
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
12/13   NYBT 2639 And 2641 Jerome Avenue Read Me.html
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
13/13   NYBT Banner03.png
[WARNING] Failed to decompress file, exception was Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens.
[ERROR] Extraction failed. The installer is either encrypted or a version, which is currently not supported.

My Windows is in Dutch, but the message "Het opgegeven pad, de bestandsnaam of beide zijn te lang. De volledig gekwalificeerde bestandsnaam moet minder dan 260 tekens bevatten en de mapnaam minder dan 248 tekens." means "The given path, the filename or both are too long. The fully qualified filename has to be less than 260 characters and the folder name has to be less than 248 characters".

[memo33]

Ok, thanks for testing. We're probably ok then in terms of path lengths. I've shortened one of the temporary paths, so they are not too long now, as long as sc4pac isn't installed in a very deep folder or an archive contains very long folder and file names.
From the debug output (with environment variable SC4PAC_DEBUG=1):

--> Attempting to extract Clickteam exe installer "/mnt/data/sc4files/temp/staging-process9826920896034249457/jars/19cfad4d/Install NYBT Gracie Manor (Maxis Nite).exe" to "/mnt/data/sc4files/temp/staging-process9826920896034249457/exe12182150406131884686"

You can test it with the updated sc4pac.bat file if you place cicdec at ./cicdec/cicdec.exe. I've force-pushed to your branch, so be sure to reset it before adding changes.

While the gracie-manor installer extracts flawlessly, the jerome-avenue one still fails to extract and I'm not sure if we can do much about it.

Starting extraction at offset 188422

Reading block 0x1435 UNKNOWN_FONT     with size 652
Reading block 0x1436 UNKNOWN_DATA     with size 102183
Reading block 0x143E STRINGS          with size 1850
Reading block 0x1445 REGISTRY_CHANGES with size 140
Reading block 0x143A FILE_LIST        with size 765
Reading block 0x143E STRINGS          with size 1862
Reading block 0x1445 REGISTRY_CHANGES with size 139
Reading block 0x143A FILE_LIST        with size 761
Reading block 0x7F7F FILE_DATA        with size 2853389
Reading block 0x8A0D 35341            with size 3050993665

13 files in installer


Starting extraction as installer version 40

1/13    ��S�CS$$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
2/13    ���S�CS$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
3/13    T��S�NYBT 2639 And 2641 Jerome Avenue01.SC4Model
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
4/13    C��S�NYBT 2639 And 2641 Jerome Avenue02.SC4Model
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
5/13    g��S�NYBT 2639 And 2641 Jerome Avenue03.SC4Model
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
6/13    ^��S�NYBT 2639 And 2641 Jerome Avenue04.SC4Model
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
7/13    *|�S�R$$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
Parameter name: offset
8/13    )�S�R$7_2x2_NYBT 2639 And 2641 Jerome Avenue.dat
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
9/13    �0�S�images\Dark Night.jpg
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
10/13   �׻S�images\Day.jpg
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
11/13   8B�S�images\Maxis Night.jpg
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
12/13   ��S�NYBT 2639 And 2641 Jerome Avenue Read Me.html
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
13/13   �/<�NYBT Banner03.png
[WARNING] Failed to decompress file, exception was The UTC time represented when the offset is applied must be between year 0 and 10,000.
Parameter name: offset
[ERROR] Extraction failed. The installer is either encrypted or a version, which is currently not supported.

[memo33]

It seems cicdec gets the installer version wrong. Manually extracting with the option cicdec.exe -v 35 works. Hmmm.

[sebamarynissen]

That's weird. When I tried it by putting the installer in a path that's not too long, it correctly extracted all the files and recognized the installer version was 35.

Cicdec itself states though that the version detection might not always work properly. As far as I'm aware, no version information is stored in the installer itself, so if no version is explicitly specified, cicdec just tries to parse it as all versions it knows, and then when it's reading stuff that it considers garbage, it assumes that this can't be the correct version. See https://github.com/Bioruebe/cicdec/blob/master/cicdec/Program.cs#L215 and https://github.com/Bioruebe/cicdec/blob/master/cicdec/FileInfo.cs#L57 for the logic. No clue why it does it correctly on Windows, but not on Linux.

It could be an option to add the installer version to the metadata of the asset. Something like

assetId: aaron-graham-2639-and-2641-jerome-avenue
url: https://community.simtropolis.com
version: 1.0.0
lastModified: ""
installer:
  type: clickteam
  version: 35

[memo33]

I've filed an upstream report at Bioruebe/cicdec/issues/10. It looks like cicdec uses a file-system-dependent .NET API to determine whether or not the extracted paths contain invalid characters. On Linux/ext4, it's just that more characters are valid.

Initially, I was considering to just try all the installer versions until extraction succeeds, but, seeing that this creates garbage files on the file system, this seems unsafe. Your proposal of extending the metadata might be the best approach, even though this adds more complexity to the process of writing metadata.

[memo33]

The current branch is ready for merging. I'm going to create a new release with the cicdec.exe included.

If you have the chance, could you give this .bat file a quick sanity check? Setting the environment variable here is something I haven't tested yet.

sc4pac-tools/src/scripts/sc4pac.bat

Lines 14 to 16 in 04435a7

	IF "%SC4PAC_CICDEC_CMD%"=="" SET "SC4PAC_CICDEC_CMD=%SCRIPTDIR%\cicdec\cicdec.exe"
	java -jar "%SCRIPTDIR%\sc4pac-cli.jar" %*

[sebamarynissen]

Tested the .bat file and it's working nicely. 👍

[memo33]

Awesome, thanks for confirming and for your contribution here. It's merged into 0.4.2.