ChainRules types in FiniteDifferences to replace to_vec

[willtebbutt]

As suggested in #90 (comment) , we might want to consider moving away from to_vec towards defining operations on ChainRules's types directly.

In particular #91 implements the difference operation, which is the only operation that we're missing to let us approximately compute tangents. Still TODO is

1. determine how to compute cotangents within this framework + implement it, and
2. determine whether or not we require some notion of a Jacobian (probably we do) and implement it, and
3. define the norm of a differential, from which we get isapprox for free if we also define subtraction, which is really easy because we've already defined addition and scalar multiplication. I think we can do this because I think we can always derive an appropriate norm for a differential, that is, I think we can treat any given differential as being an element of an appropriate normed vector space. (Note that we've not defined inner products between differentials, and I don't think we need to. Doing so would be one way to go about defining a norm, but it probably makes sense to go straight to a norm if we can't think of a reason why would need to define inner products. I'm open to suggestions here.) This is something we'll need to sort out in ChainRulesCore, as per Approximate Equality of Differentials ChainRulesCore.jl#184.

Implementing this will immediately resolve:

1. j′vp errors on empty array primal input #92
2. Overload FiniteDifferences.to_vec for Composite ChainRulesTestUtils.jl#24 as it is covered by difference already

but will generally significantly improve the compatibility between ChainRuless and FiniteDifferences, and should make one's experience testing differentials a much more pleasant experience.

[oxinabox]

Its weird to me when finitedifferences returns a non-valid differential like:

julia> jvp(central_fdm(3,1), x->(x,x), (1, 1.0))
(1.0000000000000002, 1.0000000000000002)

[willtebbutt]

Sorry, could you expand? In what way is the differential invalid? Because derivatives aren't defined on the integers?

[oxinabox]

Because differentials need to support addition.
Or really they need to be the potential result of subtraction -- a difference.
And Tuples don't -- Composites do

(sorry bad example, I normally ty and avoid examples with integers)

[willtebbutt]

Oh I see. Yeah, it's a bit frustrating.

I think we're going to need some bespoke functionality to convert a primal into a valid tangent vector for this kind of thing.

[oxinabox]

I am surprised we even get a tuple in the first place.
I guess its is because of how map works.
I think we don't need a function for converting a primal into a valid tangent vector
I think we should be able to get this out of just using substraction,
and having a fallback for if substraction is not defined that creates a Composite