Batch IO support

[QwertGold]

Hi Committers

Long time QuickFixJ user here, thanks for all the code.
Have the committers ever discussed changes to the Session logic to allow batching of incoming and outgoing messages?

Currently every outgoing message requires two calls to the message store, storing the message and incrementing the sender sequence number.
likewise, every incoming message requires you handle (persist) the received message in Application.fromApp after which the target sequence number will be incremented.
When running with the JdbcStore this type of individual IO operation, is very costly. If you are running in the cloud even the FileStore is slow, as a redundant filesystem is reached via the network. When I say slow I mean ~5ms per IO operation.

To add some numbers, when the stock market opens at 9:30 in New York, we sometimes get 20.000 ExecutionReports within seconds. Having two IO operations per ExecutionReport create a huge backlog, that takes minutes to process.
We have worked around this by creating a custom version of the JdbcStore, so we can create a batch of messages for processing, and increment the target sequence number afterwards.
However, I don't see a reason why the framework could not offer that feature (a callback with a list of available messages, instead of one at the time) - I'm aware that this is not an easy thing to do, you have to maintain backwards compatibility, keep the protocol state in sync, etc. Has anything like this ever been discussed?

[chrjohn]

Hi @QwertGold ,
there was a similar brief discussion some months ago on the mailing list: https://sourceforge.net/p/quickfixj/mailman/quickfixj-users/thread/CAMv3%2B_-CS9d3JAff9nCpsghtb1PTKMDNBpJ55idKSKnH1jps-Q%40mail.gmail.com/#msg37225209
It seems that some people also do it as you describe, i.e. only increment the seqnum every n messages.
Cheers,
Chris.

[QwertGold]

Thanks Chris
I have looked at some of these solutions, and I think they have a 'hidden' flaw.

You can delay the storage of the target sequence number, because the protocol has a built-in resend mechanism. However, the solution I have seen for batching sent messages have violated the following protocol principle regarding resending:
"Messages MUST be persisted before they are sent"!

The solution I have seen, work like this Hibernate message store from Marketcetera, https://github.com/Marketcetera/marketcetera/blob/3c01d2244bd4562756c7a1898645a031e7779c9f/fix/fix-core/src/main/java/org/marketcetera/fix/store/HibernateMessageStore.java#L381
They add a storage event to a queue, so another thread can perform batch storage of the outgoing messages and sequence number - and this violates aforementioned principle.

I know crashes are rare, but in the worst case scenario messages are lost, and you will not be able to reconnect to the session.

1. Session.send gets called by an application thread
2. Session.persist is called, and calls are made to the message store to persist the message and increment the sender seqnum
3. The calling thread proceeds to 'send' the message (enqueue it with the Mina Responder)
4. The Persister thread eventually stores the message and sequence number.
   Since both 3 and 4 are async, they can happen in any order, and this is dangerous.

If the persister thread becomes slower than the Mina sending layer, it is possible to have sent message which have not yet been stored. If you crash in that state, you have lost the message. If the other side has received the messages and incremented its target sequence number, then your sequence numbers are out-of-sync and you will not be able to reconnect, because you sent the message and your sequence number is lower than the receiver's, which should never happen.

I know the chance of this happening is very low, but I don't feel comfortable with these delayed storage solutions, because the entire resend mechanism is built around the principle, that the client persists messages before sending.

I really think the framework should have a method which allows the application to send a list of messages, which can be handled by the message store in a transactional manner. I would even volunteer some free time to try to write it, if I could get some guidance on whether JNI compatibility affect which changes can be made, and other backwards compatibility concerns.

[chrjohn]

Hi @QwertGold
you are absolutely right that there is a gap in this processing. You can only aim to make the gap smaller. For example if the messages were put to a Hazelcast (or similar) grid which was distributed over several nodes. That way you don't need to persist to disk which is fast but of course everything is broken when all nodes go down.
Moreover, there are two things involved in this: storage of the sequence number and storage of the message itself. When you can store the sequence number in a fast manner then you can most likely avoid the "sequence too low" scenario but of course you can only send a gap-fill message on a resend in case you haven't already persisted the message.

As to the last part of the question: I don't think we still should have to deal with JNI- or backward compatibility anymore about 15 years after QFJ's initial release. :) I would be happy to see a PR, if you need help just ask.

Cheers,
Chris.