Tweak mermaid dataflow arrow heuristics for downstream pipelines

[ryannedolan]

Summary

Tweaked the heuristics used for rendering the dataflow arrows in mermaid diagrams via the !mermaid command.

Details

Previously, any downstream resources were fully connected to an upstream resource, even when downstream resources were themselves connected in series.

With this change, downstream resources are only connected to the upstream resource if they aren't already connected to something upstream.

As this is a heuristic, it's likely to break for some topologies, but it seems to work for existing use-cases. Keep in mind that this is only for visually inspecting pipelines -- the arrows are non-functional.

Testing

This was tested with an internal (non-public) adapter, with results along these lines:

flowchart
  subgraph SqlJob
  R-299574215["pipeline.sql: CREATE DATABASE I..."]
  end
  subgraph EtlJob
  R769688330["catalog: redacted
database: tracking
kafkaPipeline: tracking
kafkaTopic: REDACTED1
table: REDACTED1"]
  end
  subgraph KafkaTopic
  R-1619492592["topicName: REDACTED1"]
  end
  
  subgraph ReverseEtlJob
  R1874360940["kafkaPipeline: tracking
kafkaTopic: MyPinotTable
table: MyPinotTable"]
  end
  subgraph BrooklinDatastream
  R-1748512199["brooklinCluster: https://brooklin....
connectorType: BrooklinFunction
name: brooklinsampledb1...
sourceDatabase: BrooklinSampleDb1
sourceTable: SampleTable1"]
  end
  subgraph IcebergTable
  R1232766216["
database: tracking
table: REDACTED1"]
  end
  subgraph REDACTED
  R1207860856["
database: tracking
table: REDACTED1"]
  R-1812063814["catalog: pinot
database: tracking
table: MyPinotTable"]
  end
  subgraph PinotTable
  R1460909616["table: MyPinotTable"]
  end
  subgraph EspressoTable
  R-809211083["database: BrooklinSampleDb1
name: sampletable1
table: SampleTable1"]
  end
  
  subgraph SqlJob
    R1207860856 --> R-299574215
    R-1748512199 --> R-299574215
  end
  subgraph EtlJob
    R-1619492592 --> R769688330
  end
  subgraph KafkaTopic
  end
  
  subgraph ReverseEtlJob
    R-1812063814 --> R1874360940
  end
  subgraph BrooklinDatastream
    R-809211083 --> R-1748512199
  end
  subgraph IcebergTable
    R769688330 --> R1232766216
  end
  subgraph REDACTED
    R1232766216 --> R1207860856
    R-299574215 --> R-1812063814
  end
  subgraph PinotTable
    R1874360940 --> R1460909616
  end
  subgraph EspressoTable
  end
  

Loading