XSOR: Incomplete state transition in case of conflicting states

[csc-gip]

Node 1: CONNECTED
Node 2: DISCONNECTED but reachable (NAd died on Node 1)

Update on Node 1
Restart of Node 1
Start NAd

Node 1 is updated with all data from Node 2 (SYNC_M => SYNC_S)
Node 2 gets (old) updated state from Node 1 which is still in its incomming queue somehow

Update on Node 1: => Conflict as the states are different
Node 1 stays in state 'M'

The state should not stay in 'M' but return to 'S' and the data should be considered as invalid (for updates).

[csc-gip]

If there is a conflict grabDoubleCheckAndReturnRereadPayload throws an exception

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/XynaScalableObjectRepositoryImpl.java

Line 1316 in 971b722

throw new CollisionWithRemoteRequestException();

which is not handled in search

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/XynaScalableObjectRepositoryImpl.java

Line 1236 in 971b722

XSORPayload grabbedPayload = grabDoubleCheckAndReturnRereadPayload(transactionLockedPair.getInternalId(), transactionLockedPair.getPayload(),

As the state has been changed to 'E' it should be added to the grabbed elements

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/XynaScalableObjectRepositoryImpl.java

Line 1298 in 971b722

transactionContext.grabbed(internalId, searchRequest.getTablename());

as e.g. in the case of PENDING_ACTIONS_LOCAL_CHANGES and the payload needs to be released again.

            case CONFLICTING_REQUEST_FROM_OTHER_NODE:
              // state is 'E' but the grab failed due to inconsitency.
              transactionContext.grabbed(internalId, searchRequest.getTablename());
              releaseAndUnlockFromTransaction(internalId, payload, correspondingMemory, strictlyCoherent,
                  transactionContext);
              throw new CollisionWithRemoteRequestException();

[csc-gip]

The serverSocket is never closed, when pausing the server.

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/interconnect/InterconnectServer.java

Lines 196 to 210 in 971b722

	private void closeSocket() {
	Socket s = socket;
	if (s != null) {
	socket = null;
	socketClosedForWriting=true;
	if (logger.isDebugEnabled()) {
	logger.debug("closing open connection.");
	}
	try {
	s.close();
	} catch (IOException e) {
	logger.warn("could not close connection", e);
	}
	}
	}

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/interconnect/InterconnectServer.java

Lines 347 to 354 in 971b722

	public void pauseWorking() {
	synchronized (notifyObject) {
	paused = true;
	notifyObject.notifyAll();
	}
	//sicherstellen, dass nicht weiter vom socket-inputstream gelesen wird.
	closeSocket();
	}

So the server will accept connections from clients while being paused and keep the messages from the client until accept is called again.

xyna-factory/components/xnwh/xcs/xsor/src/com/gip/xyna/xsor/interconnect/InterconnectServer.java

Lines 76 to 86 in 971b722

	ServerSocket s = serverSocket;
	while (running && !paused) {
	if (s != null) {
	Socket localSocket = s.accept();
	if (logger.isInfoEnabled()) {
	logger.info("got new socket from " + localSocket.getInetAddress() + ":" + localSocket.getPort());
	}
	socket = localSocket;
	socketClosedForWriting=false;
	InputStream is = localSocket.getInputStream();
	while (running && !paused) {

Server closes its connection to the client:

04-12-2024 13:34:04 XYNA_001  INFO [XynaClusteringServicesManagementChangeHandlerThread0] (DHCPv4ServicesImpl.java:632) clusterstate changed to DISJOINED_RUNNING. oldstate = CONNECTED
04-12-2024 13:34:04 XYNA_001 TRACE [StateChangeHandlerExecutorThread-1] (XSORMemory.java:1103) changeQueueModeToMerging()
04-12-2024 13:34:04 XYNA_001  INFO [XynaClusteringServicesManagementChangeHandlerThread0] (DHCPv4ServicesImpl.java:632) clusterstate changed to DISJOINED_RUNNING. oldstate = CONNECTED
04-12-2024 13:34:04 XYNA_001 DEBUG [StateChangeHandlerExecutorThread-1] (InterconnectServer.java:202) closing open connection.
04-12-2024 13:34:04 XYNA_001 ERROR [InterconnectServer-Thread] (InterconnectServer.java:187) IOException reading data
java.net.SocketException: Socket closed
        at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
        at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
        at com.gip.xyna.xsor.interconnect.InterconnectServer.readInBuffer(InterconnectServer.java:281) ~[xsor.jar:?]
        at com.gip.xyna.xsor.interconnect.InterconnectServer.run(InterconnectServer.java:87) ~[xsor.jar:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

Client immediately opens a new connection and (tries) to send updates

04-12-2024 13:34:04 XYNA_001  INFO [InterconnectSender-Thread] (InterconnectSender.java:106) opening socket to 192.168.178.133:1712
04-12-2024 13:34:04 XYNA_001  INFO [InterconnectSender-Thread] (InterconnectSender.java:111) try to set extended Options
04-12-2024 13:34:04 XYNA_001 DEBUG [InterconnectSender-Thread] (InterconnectSender.java:144) Interconnectsender connected to 192.168.178.133:1712

04-12-2024 13:34:10 XYNA_001 DEBUG [InterconnectSender-Thread] (InterconnectSender.java:156) CORRECT sending message 17121987I 45

04-12-2024 13:34:10 XYNA_001 DEBUG [InterconnectSender-Thread] (InterconnectSender.java:156) CORRECT sending message 17121988S 141

After synchronization the server accepts the connection and handles the (old) messages.

04-12-2024 13:40:33 XYNA_001  INFO [InterconnectSender-Thread] (InterconnectSender.java:90) opening socket to 192.168.178.132:1712
04-12-2024 13:40:33 XYNA_001  INFO [InterconnectServer-Thread] (InterconnectServer.java:81) got new socket from /192.168.178.132:51118
04-12-2024 13:40:33 XYNA_001 DEBUG [InterconnectSender-Thread] (InterconnectSender.java:116) Interconnectsender connected to 192.168.178.132:1712

04-12-2024 13:40:33 XYNA_001 DEBUG [InterconnectServer-Thread] (XSORProcess.java:805) newState=I, type=43, cid=17121987, checksumRemoteBackup=995033974, checkSumNewCalculatedRemotely=169220220, checkSumNew=0, modTime=1733318823495, relTi
meRemoteBackup=1733318823497, relTimeNew=1733318823497, objectID=10.22.24.124

04-12-2024 13:40:33 XYNA_001 DEBUG [InterconnectServer-Thread] (XSORProcess.java:805) newState=S, type=43, cid=17121988, checksumRemoteBackup=995033974, checkSumNewCalculatedRemotely=-943311168, checkSumNew=-943311168, modTime=1733319250919, relTimeRemoteBackup=1733318823497, relTimeNew=1733319250920, objectID=10.22.24.124