feat(cli): added cli option for ingestion source

[kevinkarchacryl]

I'm adding a new CLI command called "datahub ingest list-source-runs" (name pending) that will give this result

Can also filter by URN and/or source name:

Do we like the resulting columns? Are there any we should add or get rid of?

I attempted to filter by platform but it seems like platform is always null in the graphql response.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[sgomezvillamor]

These are output columns for datahub ingest list-runs as of today: runId, rows and created at.
In your proposal, columns are: runId, source, startTime, status and URN.
Assuming startTime is the same as created at, so this is adding source, status and URN.

Instead of adding a new option to the command line, what about extending table in list-runs with the new columns?

I hope this helps:

• datahub ingest list-runs is resolved by

  datahub/metadata-io/src/main/java/com/linkedin/metadata/systemmetadata/ElasticSearchSystemMetadataService.java

  Line 200 in 84de42d

  public List<IngestionRunSummary> listRuns(

• and modelled in

  datahub/metadata-models/src/main/pegasus/com/linkedin/metadata/run/IngestionRunSummary.pdl

  Line 3 in 84de42d

  record IngestionRunSummary {

FYI: @hsheth2

[kevinkarchacryl]

@sgomezvillamor The problem with the current datahub ingest list-runs today is that it's not working properly. The runId often shows 'no-run-id-provided' and the number of rows is off. This new command uses the graphql endpoint instead of elasticsearch to always get the proper runId. I think we decided that making a new working command was better than changing an old one.

[sgomezvillamor]

I think we decided that making a new working command was better than changing an old one.

I was not aware this had been discussed before, sorry for the confusion I may have caused.

If you discussed this with someone else, feel free to invite them to join the review here.

[hsheth2]

Going to weight in with my two cents here

Main question is if we consider the old behavior to have any real use cases, or if we'd just say the old behavior was buggy.

If the former -> let's do two commands as done here, and we can mark the old one as deprecated
If the latter -> we can just replace the old command, since there's no real reason to keep it around

[asikowitz]

Talked to Gabe about this separately. I could see the old behavior having use cases, e.g. if you are disciplined about setting run ids while using the API, that is not represented by the list of custom ingestion sources. In general I think run ids and ingestion source runs are different enough that we can have CLI commands for both, as long as they're distinguished esp in docs.

[asikowitz]

Another use case: a user produces data from an ingestion source, then hard deletes it. If they want to clean up data from that deleted ingestion source, they need to look at run ids, because the ingestion source no longer exists.

[kevinkarchacryl]

So it looks like we should keep both options. Do we want to change the help output for either one to make it more clear what they're doing? Any changes to docs necessary like adding deprecated?

[sgomezvillamor]

So it looks like we should keep both options. Do we want to change the help output for either one to make it more clear what they're doing? Any changes to docs necessary like adding deprecated?

Keeping both options 👌

My suggestion:

• Renaming list-runs to list-run-ids would be a breaking change, which I'd prefer to avoid. Let's maintain list-runs and introduce list-source-runs.
• In addition to documenting the new list-source-runs, let's also add the missing documentation for list-runs to help users understand the differences.

[sgomezvillamor]

LGTM