is state-machine testing the right approach?

[mwotton]

This is the underlying problem making #3 and #4 hard: as soon as we have to consider the possibility of digging into sum types for components, things get a bit complicated.

Consider this API:

data Branchy = Branch1 B1 | Branch2 B2 | .... | Branch100 B100

type Api = "getBranchy" :> Get '[JSON] Branchy 
            <|> "b1" :> Capture "one" B1 :> Get '[JSON] Int
            <|> "b2" :> Capture "one" B2 :> Get '[JSON] Int
...
            <|> "b99" :> Capture "one" B99 :> Get '[JSON] Int

This is obviously a pathological case, but illustrates the problem: Because we never get any information back from the concrete side about which arm of the sum the concrete server returned, we're in a bit of a pickle: we need that information to decide which commands can be generated! This wasn't a problem before we were breaking values down, because we statically knew the type coming back, but now it's up to the concrete server.

Options:

1. We could explicitly model every possible state the model could be in.

• this would at a minimum have to include information about whether we've got a response containing B${n} for each n from 1 to 100
• a sum with 2^100 arms isn't that practical, and that's just on this toy problem.

2. We could fly completely blind

• Keep all the real info on the concrete side, and then the generator can produce commands like "given a datatype Foo = Foo A B C, using the 3rd, 7th and 1 millionth A, B and C available to you (mod the number of As, Bs and Cs you have respectively), construct a value of type Foo."
• this fails when there are zero instances of some components: in this case, there'd be a state-machine step recorded that doesn't actually do anything. This doesn't completely break it, it can be shrunk away later, but it seems like it could easily lead to a case where you very rarely actually manage to put together a working call.

3. We could modify Hedgehog

• Huw in the hedgehog slack suggested it might be possible to pass some information about actual concrete control flow back. This would be great, but I have absolutely no idea how to go about this without breaking invariants.

4. We could abandon hedgehog entirely.

• I can explain to a human fairly easily the process roboservant takes, it's not uncomputable or anything. it just seems hard to jam into this testing paradigm: perhaps we need a different substrate.

[mwotton]

homebrew solution works.