Should we add some sort of verified_working attribute or timestamp?

[pnadolny13]

We have a maintenance status attribute right now thats mostly based on repo commit frequency. It would be awesome to have a flag and probably a timestamp to go with it that can be updated based on usage stats or user input about whether the plugin has been working for people recently. As part of adding a bunch of variants to the hub I was able to verify that they all were installable and could at least run --about or --help, thats useful information to know "As of 12-05-2022 this seems to be working" vs something like https://hub.meltano.com/extractors/tap-appstore where maintenance is unknown/no usage to go off. It could help fill the gap for taps that are lightly used by projects with telemetry off, so theyre good but we have nothing to show for them.

How we could do it:

1. It would be awesome if we could get some live attributes from users kind of like how https://downdetector.com/ does. If they land on the page, they can click a button that fires us a snowplow event saying "I verified this worked". We could use that in combination with telemetry data on adds + successful runs to show a verified flag with timestamp. Passed back from Squared to the hub via the daily metrics data.
2. Alternatively or in addition we could add an attribute to the plugin definition with this info. Although thats harder to update and could go stale quickly, so we'd have to have logic that stops showing it if its >6 months old or something like that.

What do people think? Open to other solutions but wanted to get the ideas flowing 😄

[aaronsteers]

I agree; I this is super important - thanks for suggesting!

One path that doesn't require re-deploying the Hub when metrics change is to publish badge images to a public S3 bucket.

For instance, we could publish badge images to paths like these:

s3://public-meltano-bucket/badges/extractors/tap-gitlab--meltanolabs.svg
s3://public-meltano-bucket/badges/extractors/tap-github--meltanolabs.svg
s3://public-meltano-bucket/badges/extractors/tap-github--variant2.svg
s3://public-meltano-bucket/badges/extractors/tap-github--variant3.svg
s3://public-meltano-bucket/badges/extractors/tap-github--variant4.svg
...

Each plugin detail page can then get its badge dynamically based on a naming convention, without parsing the meaning of the status from the image.

We could then render all of the badges on the "Other variants" section of the plugin detail pages, so that whenever you are evaluating switching variants, you can some confirmation that

We'd ideally want to fail gracefully in the web UI if a badge is missing, but as status changes occur on these repos, they could be picked up dynamically by the website.

Now, that said, I think we're currently in a place where we need not just badges but also category changes, and we may very soon want to render hints about stability on the plugin discovery and search screens - which likely would not perform well if pulling in dynamically for n distinct images and n is >400. There could be advantages to simply setting and maintaining this just in the same way we're doing for other metrics so that use cases where we need the quality evaluation on 400-700 variants can be performed quickly.

Whatever way is simplest I think something this would be valuable.