Limit the number of shapes trip_distance_exceeds_shape_distance supports

[davidgamez]

Description

As part of the investigation of #1587, we came across a feed to ~40M of shapes to validate. The current validator's implementation needs to be optimized to handle large feeds; see #1358. The recently merged #1553 introduced validation on shapes that expands resource usage on the validator. As a temporary solution until #1358 provides a better way to manage large files, we propose limiting the number of the shapes that trip_distance_exceeds_shape_distance's validator supports.

Tasks:

• Investigate an approximate higher number of shapes that trip_distance_exceeds_shape_distance can support.
• Implement logic to skip the validator when the shape count is above the limit and log why the validator is being skipped.
• Add the validator to the list of validators that didn't run due to limit

[emmambd]

@cka-y Is this needed after the work done in #1676?

[cka-y]

@davidgamez could you share the feed containing the 40M shapes, so I can provide a well-informed answer to @emmambd's question, please?

[emmambd]

@cka-y It's this one: https://storage.googleapis.com/storage/v1/b/mdb-latest/o/de-unknown-ulmer-eisenbahnfreunde-gtfs-1081.zip?alt=media

[cka-y]

From the master branch, here’s what I found regarding TripAndShapeDistanceValidator performance for the example feed:

• With the validator, validation takes about 199 seconds.
• Without it, it's around 183 seconds.

The difference is roughly 16 seconds, which seems reasonable for the volume of shapes (~40M) we're handling. I believe we don’t currently have data on much larger sets of shapes, but so far, it doesn’t seem necessary to make further adjustments.

Thoughts?

[emmambd]

Closing given there were significant improvements in the last pre release analytics: #1703