Brainstorm about exchange specific channels

[sebamarynissen]

I was thinking a bit about how metadata can be manged more easily in the future, and realized that for me, the ideal situation would be that the big exchanges (Simtropolis and SC4E) host their own channels that update automatically when new content is posted. More specifically, when someone uploads a new file, they should get the option to immediately write the yaml metadata for it that when published immediately becomes available in the exchange's channel.

The exchanges might not even need to implement all of this themselves, I can imagine an approach where they just call a server endpoint with the new metadata that then compiles it for them and returns the compiled metadata somehow. The only thing then is that those channels might depend on one another (ST metadata referring to packages available on SC4 and vice versa). I'm definitely willing to help here. What's your take on this @memo33 ?

[memo33]

@sebamarynissen I'd love that. It would give the original plugin authors direct control over the metadata for their files.

My main concern is how much work this will require from the admins of those sites. They aren't necessarily programmers and I'm not sure if this is asking too much. The sites are mostly black boxes to me, so I'm not sure how much we can assist from the outside.

The second concern is that writing correctly formatted metadata seems to be quite difficult for many people. @noah-severyn's editor can be very helpful though. Perhaps combined with an automated check whether the metadata compiles (this could maybe be integrated into the GUI).

That said, a basic approach for such a system might even be:

• plugin authors upload an additional .yaml file on their STEX entries
• a server runs a nightly build script, finds those newly added .yaml files using the STEX API, downloads them and builds the channel.

(It's a bit more complicated due to download limits, the lack of my own server and the fact that uploading a .yaml file to the STEX would alter the timestamp for that entry which in turn cannot be part of the .yaml file itself, but those are details.)

[noah-severyn]

I would really love this idea too.

My main concern is how much work this will require from the admins of those sites. They aren't necessarily programmers and I'm not sure if this is asking too much. The sites are mostly black boxes to me, so I'm not sure how much we can assist from the outside.

Yep, I share the same concerns. From what I understand, ST is one or two major versions behind from the most recent board software releases, and upgrading to versions that could potentially make this a little easier would not be an easy task because of significant custom development CB has put in.

plugin authors upload an additional .yaml file on their STEX entries

I see this as a non-starter. I again have the same concerns here. SC4 is relatively unique in the video game modding scene that a majority of the mod creators are less technical and tend towards more artist/modeller/designer types. I am skeptical about requiring this for fear of increasing the friction of uploading, and that's the last thing a really niche community needs. But at the same time if it is strictly optional, it kind of defeats the purpose because we would need to go create the .yaml file ourselves for any author who didn't create it themselves.

Another problem is keeping them both in sync - what if an author updates the SC4 file but not the .yaml file. The .yaml file would need to have some parts automatically updated somehow. At that point, plus with the other drawbacks memo mentioned, it feels like watching the STEX API from the outside (assuming that's possible) would yield better and more controllable results. Something like querying the STEX API every 24h or so scooping up any recently updated or added items, and putting them through some external processes to generate the .yaml.

IMO, minimizing as much as possible the tweaks to ST is the optimal path for now.

[sebamarynissen]

Yes, I think the path forward here is indeed to require the minimum amount of work possible on the side of the exchanges itself, so indeed a script that runs every 24 hours and pulls in the new entries is probably the way to go. We could obviously use a hybrid approach as well: set up a cron job, but also allow exchanges to dispatch an event. I immediately think here of how GitHub actions can be trigger both by a cron job or a workflow_dispatch event via the api. In that case the exchange could decide to implement this or not, without it being catastrophic if it suddenly does not work anymore.

As for writing the metadata, I would definitely not require this, as as @noah-severyn says, this might be a hindrance to upload new content, though I think it should be heavily encouraged. I hope that once the gui has been released and much more packages have been added - I'm working really hard on this as you can all see - the community will realize that this is the way to go.

As for writing the metadata, there are definitely ways to make this easier. A lot of the metadata can probably be prefilled from what the user has already entered in the "normal" fields when uploading content and even for the dependencies there are ways to automate this - see for example my helper scripts where I'm automatically tracking dependencies by looking at all urls mentioned in the description and then matching them against a known database of sc4pac packages. For the moment it covers all dependencies when adding a package like 90% of the time.

As for minimizing the work for admins of the exchanges, I think it should be possible to compile a JavaScript file for this which can then simply be added as a plugin to the upload page. I'm no expert in Invision Community software, but I assume adding a JavaScript file to a page is easy.

I'd say we give ourselves some time to think about it a bit, and then we can maybe compile some kind of draft idea with the minimal amount of work needed and then check with CB what he thinks of it.

In any case, I really think that somehow the responsibility to write metadata for packages should be put in the hands of the creators itself somehow, obviously minimizing the amount of friction it creates for them.

[Zasco]

As I believe my #48 (comment) launched this whole thread, I simply want to clarifiy I didn't mean exchange specific, but channel specific. ST's packages depending on SC4E's packages is no big deal.

Yes, I think the path forward here is indeed to require the minimum amount of work possible on the side of the exchanges itself, so indeed a script that runs every 24 hours and pulls in the new entries is probably the way to go. We could obviously use a hybrid approach as well: set up a cron job, but also allow exchanges to dispatch an event. I immediately think here of how GitHub actions can be trigger both by a cron job or a workflow_dispatch event via the api. In that case the exchange could decide to implement this or not, without it being catastrophic if it suddenly does not work anymore.

This resumes well my position about the management of the metadata.

[sebamarynissen]

Coming back to this, now that @memo33 has moved the actions code to a separate repo (https://github.com/memo33/sc4pac-actions), I think I'm going to try and setup something in the coming days as a proof of concept for an exchange channel. My idea is to have the following flow:

• Run an action that pulls in data from the STEX api. Perhaps that I need to fake the STEX api endpoint initially, but we'll see. For the PoC, I will probably have the action be triggered manually via workflow_dispatch and a specific STEX file id, but in the future this can probably be changed to run at specific intervals, or even be dispatched by ST itself whenever a file is uploaded/modified.
• The response from the STEX api is parsed and a .yaml file is generated automatically from it.
• The generated yaml metadata is commited to the repository (automatically by the action).
• @memo33's actions run on every commit to main, which will subsequently deploy the updated channel.

This workflow eliminates the work needed by the exchanges as much as possible. The only problem I see with this is customizing the metadata. In the ideal case, Invision Community allows to easily add more fields in the file upload forms. For example, if a "metadata" field could be added, then the creator could write the metadata right away, and then we can pull in the metadata field from the STEX api.

If this is not possible, there might still be the possibility to leverage the "description" field somehow, for example by adding custom syntax. For example,

This is the description of the package that is visible for users

---
group: smf16
name: my-package-name
info: ...

Ideally this can be done in a way that does not show the metadata to the users, but is included in the STEX api response.

In order to make it easier for content creators, I would also "autocomplete" missing fields. For example, if the creator does not provide a group or package name, it gets parsed automatically from the api response. Same for description, url, whatever, but anything should be overridable.

As the final asset urls are unknown before the upload, I would introduce specific syntax to reference them in case non-standard behavior is needed:

# Group and name can be autocompleted
# group: smf16
# name: choose-nightmode

variants:
  - variant: { nightmode: standard }
    assets:
      - assetId: ${{assets.0.url}}
  - variant { nightmode: dark }
     assets:
       - assetId: ${{assets.1.url}}

Asset urls would simply never need to be specified, they can be generated by the actions code based on the STEX api response. For dll uploads, checksums could be added automatically. I would intentionally not support ClickTeam installers, there's no need for them anymore anyway.

Let's do this 🔥 🔥

[memo33]

Yes! This sounds like a solid plan. It reminds me a bit of how Homebrew operated I think.

It think at minimum there needs to be a way for STEX authors to say "yes, this plugin is compatible with sc4pac". Even if all the rest of the metadata might be auto-generated, someone needs to make a conscious decision about this at some point, to avoid ending up with metadata that doesn't work properly (e.g. load order, dependencies, inclusions/exclusions).

I've also been thinking about how we could display, for an entry on one of the exchanges, a link to the corresponding packages if there are any. In the simplest form, this could be entirely client-side if we can include a script on those sites. I've added an externalIds field for stex and sc4e IDs to the channel contents file which could be fetched from the browser (though, I'd probably generate a separate JSON file including proper URLs, for that purpose.)

[sebamarynissen]

Yup, I was thinking exactly the same. ST should probably add a checkbox to the file upload page that when checked allows the user to write the yaml metadata, which means the package will be added to sc4pac. Perhaps that this feature should initially be limited to a set of "trusted" users that know how to use and write metadata for sc4pac, but to be discussed with the ST admins if that is even possible.

I was thinking about creating some kind of client side script as well - which should be possible to include. It could help with writing the metadata. Such a script can probably also be used for displaying an "Add with sc4pac" button or someting, which could fire up the sc4pac gui automatically and handle everything under the hood. I suppose it's possible for sc4pac to register itself as a file handler for the sc4pac: protocol? Then we could make the button something like

<a href="sc4pac:memo:submenus-dll">Add with sc4pac</a>

Oh boy, so many ideas in my head right now. 😅

Anyway, I've started on the proof of concept, which is going fairly well: https://github.com/sebamarynissen/simtropolis-channel Currently I use my scraping approach as for some reason I can't access the STEX api. Can you verify if it still works with you? Even the examples that CB gave in the DM aren't working.

I've also hit a road block with your https://github.com/memo33/sc4pac-actions repo because I had to bypass the linting in order to be able to deploy. That's because the linking verifies that all referenced dependencies are present, but this is not the case. I think it should be possible for the linting action to take at least the default channel into account. I'll create an issue for it in your repo.

[Zasco]

Currently I use my scraping approach as for some reason I can't access the STEX api. Can you verify if it still works with you? Even the examples that CB gave in the DM aren't working.

It's working here. Any more details?

[Zasco]

memo33 said:

I've also been thinking about how we could display, for an entry on one of the exchanges, a link to the corresponding packages if there are any.

sebamarynissen said:

I was thinking about creating some kind of client side script as well - which should be possible to include. It could help with writing the metadata. Such a script can probably also be used for displaying an "Add with sc4pac" button or someting, which could fire up the sc4pac gui automatically and handle everything under the hood. I suppose it's possible for sc4pac to register itself as a file handler for the sc4pac: protocol? Then we could make the button something like

This could simply be a browser plugin? There could be an "add package" button added as an overlay somewhere logical on the page, like beside the plugin's name. This button could be a "create package from page" instead if the package is not found in any listed channel.

I've also hit a road block with your https://github.com/memo33/sc4pac-actions repo because I had to bypass the linting in order to be able to deploy. That's because the linking verifies that all referenced dependencies are present, but this is not the case. I think it should be possible for the linting action to take at least the default channel into account. I'll create an issue for it in your repo.

I think you need to add "https://memo33.github.io/sc4pac/channel/sc4pac-channel-contents.json" here: https://github.com/memo33/sc4pac-actions/blob/484b27a13ec4fbd0c3352abbf0d8e3a6283855ec/src/lint.py#L439
That's what I had to do with the version of that file on my channel: https://github.com/Zasco/sc4pac-channel/blob/6cb53187f6b762d689c8ca4734383cf90728c50d/.github/sc4pac-yaml-schema.py#L164-L167

[sebamarynissen]

@Zasco From CB's own examples, I tried this one: https://community.simtropolis.com/stex/files-api.php?key=[STEX_API_KEY]&mode=updated&type=downloads&days=365&query=DLL&category=122&sc4only=true&author=Null%2045&sort=desc (api key redacted because this repo is public). Then I just get to see this:

Screenshot

This could simply be a browser plugin?

Yes, a browser plugin is also possible, but I think it would be much simpler to just add a <script src=""> to the page that adds the same functionality.

[sebamarynissen]

Lol, suddenly it is working. Weird. Perhaps that there were some anti-bot measures in place.

[Zasco]

Yes, a browser plugin is also possible, but I think it would be much simpler to just add a <script src=""> to the page that adds the same functionality.

The browser plugin would do just that. It would be "overlayed" onto the page. An informal standard for the structure of the description field could be established to simplify parsing. These would have the upside to avoid ANY work from the exchange managers...

[sebamarynissen]

That's a good idea as a fallback indeed, but I guess CB can easily add a simple script to a page. We'll see when we get there. My plan is to improve my proof of concept a bit, and then present it to CB and the other ST staff to see what they have to say about it.

[Zasco]

@Zasco From CB's own examples, I tried this one: https://community.simtropolis.com/stex/files-api.php?key=[STEX_API_KEY]&mode=updated&type=downloads&days=365&query=DLL&category=122&sc4only=true&author=Null%2045&sort=desc (api key redacted because this repo is public). Then I just get to see this:

I get this error when I use uppercase "STEX" in the url as Cyclone Boom provided here (link to private conversation on Simtropolis about STEX API). Could it have been your case?

Warning

In the URL, "stex" absolutely needs to be lowercase!

See here and here (links to private conversation on Simtropolis about STEX API).

Edit: Uppercase is now supported as stated here (link to private conversation on Simtropolis about STEX API).

[sebamarynissen]

Yes, that must have been it. It pasted the url that didn't work in this is issue, and then GitHub lowercased the url, after which it worked when I clicked the url (although it gave me a 401 because [STEX_API_KEY] was invalid). Thanks for that!

[memo33]

I've also hit a road block with your https://github.com/memo33/sc4pac-actions repo because I had to bypass the linting in order to be able to deploy. That's because the linking verifies that all referenced dependencies are present, but this is not the case.

Alright, alright. Check the Readme of the actions repo again now. I think it should be usable and stable now. (Before that, it was mainly uploaded so I could test the actions.)

(@Zasco This avoids the need to import and modify any Python files, which in turn gives me more freedom for improving the linter.)

I suppose it's possible for sc4pac to register itself as a file handler for the sc4pac: protocol? Then we could make the button something like

Yes, it's possible. I've looked into it a few days ago. However, it wouldn't work with the cross-platform web app (I think). Instead, my idea was to link from an external exchange to the sc4pac website (for more control) and from there trigger the sc4pac API. This requires that the GUI is already running. I'll create a PR, as the details are not that relevant for this issue.

[sebamarynissen]

Thanks, that linting-config.yaml approach works perfect. The linting now properly passes, but apparently something's still wrong with the url-check action. Could you have a look at it? I don't really understand what that error message could mean.

[Zasco]

Yes, that must have been it. It pasted the url that didn't work in this is issue, and then GitHub lowercased the url, after which it worked when I clicked the url (although it gave me a 401 because [STEX_API_KEY] was invalid). Thanks for that!

I've mentionned it (link to private conversation on Simtropolis about STEX API) to Cyclone Boom.

[sebamarynissen]

@memo33 Looks like the problem with the url-check action is related to PREVIOUS_COMMIT not being set due to the way my actions are structured. I propose to move this discussion to here: sebamarynissen/simtropolis-channel#1