Wrong surrogate gradient in notebook 5 ?

[OliFaure]

Hello, it seems to me that the formula given in notebook 5 for the surrogate gradient is wrong.

The function which approximates the Heavisde function is
$$\frac{1}{\pi}\cdot \arctan(\pi \cdot U) + \frac{1}{2}$$
and therefore the derivative wrt U should be:
$$\frac{1}{(1+(\pi\cdot U)^2)} $$

Which is consistent with the backward specified in the LeakySurrogate.

[rmazzier]

I agree, I also found out about this and arrived at your same conclusion.