[pydrake] Fix incompatibility with Mypy

[jwnimmer-tri]

When parsing the docstrings of a function to find its set of overloads, C++ code samples that mention the function name were at risk of matching the C++ scoping operator :: as a type annotation, leading to malformed output. For example MyFunc(foo : int) -> bool is a valid signature, but the sample code MyFunc(Eigen::Lower) was being mis-parsed. Fix this by patching stubgen to reset in case it sees a double colon.

Enable the stubgen regression for all of pydrake; no more caveats.

Revert da58ae0 and 2cf04fd header file changes, which were work-arounds for this parsing bug and are no longer needed.

Closes #18580.

---

This change is 

[jwnimmer-tri]

+@SeanCurtis-TRI for feature review, please.

[jwnimmer-tri]

Reviewable status: 1 unresolved discussion, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers (waiting on @jwnimmer-tri and @SeanCurtis-TRI)

a discussion (no related file):
Working

This patch changed a bit more of the pyi output than I expected:
https://gist.github.com/jwnimmer-tri/b391107de9ae4484a3950abc97d1ae39

I'm going to double-check to see if the changes are all improvements.

---

[jwnimmer-tri]

Reviewable status: 1 unresolved discussion, LGTM missing from assignee SeanCurtis-TRI(platform), needs at least two assigned reviewers (waiting on @SeanCurtis-TRI)

a discussion (no related file):

Previously, jwnimmer-tri (Jeremy Nimmer) wrote…

Working

This patch changed a bit more of the pyi output than I expected:
https://gist.github.com/jwnimmer-tri/b391107de9ae4484a3950abc97d1ae39

I'm going to double-check to see if the changes are all improvements.

OK I am happy with these changes. All of them are either:

(1) Removing untyped argument names from signatures where the type is a C++ type -- because our bindings are not correctly topologically ordered and the C++ name is leaking into the docs (~90%).

(2) Removing totally buggy and wrong signatures like an argument named std which is really a mis-understood std:: (~10%).

---

[SeanCurtis-TRI]

+@ggould-tri for platform review, please.

Reviewed 5 of 5 files at r1, all commit messages.
Reviewable status: 1 unresolved discussion, LGTM missing from assignee ggould-tri(platform) (waiting on @ggould-tri and @jwnimmer-tri)

a discussion (no related file):

Previously, jwnimmer-tri (Jeremy Nimmer) wrote…

OK I am happy with these changes. All of them are either:

(1) Removing untyped argument names from signatures where the type is a C++ type -- because our bindings are not correctly topologically ordered and the C++ name is leaking into the docs (~90%).

(2) Removing totally buggy and wrong signatures like an argument named std which is really a mis-understood std:: (~10%).

I see. So, the apparent degradation (substitution of parameter names with *args, **kwargs and a return type of Any simply points to defects in the organization of binding. So, fixing those defects will restore the .pyi files to a better state?

---

[jwnimmer-tri]

Reviewable status: LGTM missing from assignee ggould-tri(platform) (waiting on @ggould-tri)

a discussion (no related file):

Previously, SeanCurtis-TRI (Sean Curtis) wrote…

I see. So, the apparent degradation (substitution of parameter names with *args, **kwargs and a return type of Any simply points to defects in the organization of binding. So, fixing those defects will restore the .pyi files to a better state?

Yes, exactly. I have a hope that down the road we'll have a build-system-automated interlock that all Drake C++ types used in Drake function signatures must resolve correctly in the docs.

---

[ggould-tri]

Reviewed 5 of 5 files at r1, all commit messages.
Reviewable status: 1 unresolved discussion (waiting on @jwnimmer-tri)

---

tools/workspace/mypy_internal/patches/reject_double_colon.patch line 9 at r1 (raw file):

the sample code "MyFunc(Eigen::Lower)" was being mis-parsed. Fix this by
patching stubgen to reset in case it sees a double colon.

minor: Has this been communicated upstream; can this reference an upstream issue number?

[jwnimmer-tri]

Reviewable status: 1 unresolved discussion (waiting on @ggould-tri)

---

tools/workspace/mypy_internal/patches/reject_double_colon.patch line 9 at r1 (raw file):

Previously, ggould-tri wrote…

minor: Has this been communicated upstream; can this reference an upstream issue number?

My hypothesis is that they would not be interested in a bug report.

The premise of stubgen is that it outputs a starting point, which a human is supposed to review and amend by hand and then commit into source control. Given our scale, we've chosen not to do that.

It's also very unusual (and wrong) to have C++ sample code in Python docstrings (#12591). I don't think most users of stubgen would hit this problem, either.

You could imagine that upstream might accept a pull request (not issue) from us with this patch, but generally my thought is to wait until all of our pydrake documentation epic is done, and then circle back and submit anything upstream. It's plausible that we might change how this all works, so this patch is in some sense "in beta" for a while before we might submit it.