mirrored queue does not mirror replication queues #25
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Charlie-chenchrl
force-pushed
the
mirroredQueueBug
branch
from
fa90e1b to
1d5f5db
Compare
NikitaShupletsov
approved these changes
Jan 24, 2024
Charlie-chenchrl
merged commit daff596
into
feature/replicationPlugin
4 of 6 checks passed
Charlie-chenchrl
added a commit
that referenced
this pull request
Jan 29, 2024
AMQ-133425 revert CR-95394111 CR-95398045 CR-96843644 Fixed formatting. cr: https://code.amazon.com/reviews/CR-99954905
Charlie-chenchrl
added a commit
that referenced
this pull request
Jan 30, 2024
Charlie-chenchrl
added a commit
that referenced
this pull request
Jan 30, 2024
Charlie-chenchrl
added a commit
that referenced
this pull request
Jan 30, 2024
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 14, 2024
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 14, 2024
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 19, 2024
[AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-9235] Update Jenkins JDK 19 -> JDK 20 (apache#994) JDK 20 compile succeeds [AMQ-9234] Remove JournalJDBC store and activeio dependency (apache#993) Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-9239] Host joram jms unit tests ahead of converting to jakarta.jms (apache#1002) Merging now for jakarta work, may backout later Removing 5.16.x from list of supported versions fix typo in activemq launcher script [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixes after rebase * Removing a duplicate dispatchNotification method from Queue * Added getConnectedTimestamp implementation to DummyConnection since this has been added to the interface recently Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 20, 2024
[AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. fix typo in activemq launcher script [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixes after rebase * Removing a duplicate dispatchNotification method from Queue * Added getConnectedTimestamp implementation to DummyConnection since this has been added to the interface recently Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 20, 2024
[AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. fix typo in activemq launcher script [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixes after rebase * Removing a duplicate dispatchNotification method from Queue * Added getConnectedTimestamp implementation to DummyConnection since this has been added to the interface recently Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 20, 2024
[AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. fix typo in activemq launcher script [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixes after rebase * Removing a duplicate dispatchNotification method from Queue * Added getConnectedTimestamp implementation to DummyConnection since this has been added to the interface recently Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 22, 2024
[AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 22, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup rebase/squash discrepancies Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
that referenced
this pull request
Mar 22, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
to kartg/upstreaming-activemq
that referenced
this pull request
Mar 25, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (amazon-mq#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
that referenced
this pull request
Apr 10, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
that referenced
this pull request
Apr 10, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
that referenced
this pull request
Apr 29, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kenliao94
pushed a commit
that referenced
this pull request
Apr 30, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kartg
pushed a commit
that referenced
this pull request
May 1, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kenliao94
pushed a commit
that referenced
this pull request
May 1, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kenliao94
pushed a commit
that referenced
this pull request
May 1, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
kenliao94
pushed a commit
that referenced
this pull request
May 1, 2024
[AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
When mirroredQueue option is enabled, messages replication queues got mirrored, which then being replicated again by replication queue, and caused stackoverflow issue eventually.
Tests:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.