Signatures may expand multiple lines

[paulinus]

When a function has default arguments, the repr() of those arguments appear in the function signature in the __doc__. Sometimes, the repr() of those default arguments may expand more than one line. For example, if the default argument is a numpy matrix, the repr() of that matrix can have multiple lines.

__init__(*args, **kwargs)
Overloaded function.

1. __init__(self: Pose, rotation: numpy.ndarray[float64[3, 3]]=array([[1., 0., 0.],
       [0., 1., 0.],
       [0., 0., 1.]]), translation: numpy.ndarray[float64[3, 1]]=array([0., 0., 0.])) -> None

These seems to break the parsing code here, which enters an infinite loop.

For the particular case of numpy arrays, one can use

import numpy as np
np.set_string_function(lambda x: " ".join(np.array_repr(x).split()))

to avoid the multiply lines but a more general solution would be to actually detect and properly parse signatures with multiple lines.

My regex abilities are not good enough to provide a fix. Sorry and thanks for building such a useful tool!

[paulinus]

I've just now read #31 So, i'm closing this to reduce the noise.

[sizmailov]

Hi, thanks for report!

Usage of repr in place of default value is the most troubling challenge for parser.
repr has not obligation to return a correct python expression, unlike default argument expression. Basically repr can return anything which will break any parser at some point. At the same time repr probably is the best candidate to provide a human-readable value of arbitrary python object.

I think the only way to handle this complexity is delegating it to user of the library via custom docstring preprocessing hook.

I'll reopen it as this issue can be solved with a little effort. #31 exists to encourage myself (and others) to contribute to mypy.

[sizmailov]

Here is simple snippet that should help to avoid the problem. Here I register a hook that replaces troubling default value. Note that it requires just updated master branch to work (I'll release it as 0.8.3 a bit later).

import pybind11_stubgen
import re

space_chars = re.compile(r"\s+")
array_repr = re.compile(r"array\(\[[^()]*]\)", flags=re.MULTILINE)


def space_chars_to_whitespaces(match):
    return "numpy." + space_chars.sub(" ", match.group(0))


def replace_mulitline_ndarray(docstring: str):
    return array_repr.sub(space_chars_to_whitespaces, docstring)


pybind11_stubgen.function_docstring_preprocessing_hooks.append(
    replace_mulitline_ndarray
)

pybind11_stubgen.main(args="cpp_library_bindings".split())

output

def accept_3x3_float64(transform: numpy.ndarray[numpy.float64, _Shape[3, 3]] = numpy.array([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]])) -> None:
    pass

The regex loop is not infinite. It has only exponential time of input string which is just as bad as infinite. Probably I need to give up regexes in this case and write all by hand... once again. sigh.

I'll keep this issue open until I solve the regex "hanging" problem if you don't mind

[sizmailov]

The remaining narrow problem is reported in #51, closing.