From 45cfefcbc8d498bf5ce437a092622512264b4f51 Mon Sep 17 00:00:00 2001 From: Nikita Shupletsov Date: Tue, 7 Jun 2022 20:16:40 -0700 Subject: [PATCH] Squashed commit for replica plugin [AMQ-8354] Add initial plugin skeleton [AMQ-8354] Replica Broker can connect to source broker [AMQ-8354] Replication of destination creation and removal [AMQ-8354] Replication of message send [AMQ-8354] Replication of purge queue and acknowledge message [AMQ-8354] Replication of transaction begin, prepare, forget, rollback, commit [AMQ-8354] stop taskRunner on broker stop [AMQ-8354] Ensure destinations are replicated on startup [AMQ-8354] Isolate the replication queue by letting to add consumers and producers only via the replication transport. [AMQ-8354] Support of durable subscribers [AMQ-8354] Test of replication of virtual topics and expired messages. [AMQ-8354] More tests for ReplicaSourceBroker AMQ-8354 Add scheduled messages support to the replica plugin [AMQ-8354] Move ReplicaBroker before ReplicaSourceBroker. Add missed try-catch in replicateDestinationCreation. Bug fix: not deleting destination from the list of destination to replicate when deleting the destination. [AMQ-8354] Atomic send message on primary broker [AMQ-8354] Different implementation of ack replication. [AMQ-8354] Add the license [AMQ-8354] Add exponential replica retries. [AMQ-8354] Add an intermediate replication queue. Make the main replication queue non-persistent. Add logic for adding and checking sequences on both sides. [AMQ-8354] Add batches for main replication queue. [AMQ-8354] Add compaction logic for send and ack events that cancel each other out. [AMQ-8354] Do not send messages to mainQueue if there are no consumers to apply message compaction on the main queue. [AMQ-8354] Increased the size of batches. Made batches idempotent. [AMQ-8354] Add message compaction when there is no consumer. [AMQ-8354] Add replica batch acknowledge. [AMQ-8354] Split threads for ack and for send. Added logic to ignore Delivered and Unmatched acks. [AMQ-8354] Improve Batcher to make batches bigger. [AMQ-8354] Reduce the delay between acks. Reduce the amount of messages in ack batches. [AMQ-8354] Replace ReplicaStorage with Sequence.Queue in ReplicaSequencer. [AMQ-8354] Small refactoring. [AMQ-8354] Add missed licence. [AMQ-8354] Add missed licence. [AMQ-8354] Add helper methods needed for replica plugin. Reimplement getAllMessageIds method to avoid pontential out of memory errors. Convert AdvisoryBroker to MutableBrokerFilter Revert "Convert AdvisoryBroker to MutableBrokerFilter" This reverts commit a32ab41903abd286b758a6bb7ab61c2d156111e5. Rollback the conversion of SchedulerBroker to MutableBrokerFilter. Remove getAllMessageIds. Rework processDispatchNotification. [AMQ-8354] Fix replica sequence saving and parsing. Fix BrokerStoppedException handling. [AMQ-8354] NoB support. [AMQ-8354] Extract constants to parameters. [AMQ-8354] Fix errors during DLQ messages replication. [AMQ-8354] Add JMX controllers for the failover [AMQ-8354] Fix btoker.stop() when there is a blocking operation on replica. [AMQ-8354] ReplicaBroker stop: close listener before close event consumer [AMQ-8354] Small refactoring. [AMQ-8354] Block consumption from replication queues when wildcard selector used. Block replication queue deletion. [AMQ-8354] Add deinitialization of ReplicaSequencer. [AMQ-8354] Disable replication of non-persistent messages. [AMQ-8354] Add failover support for replication [AMQ-8354] Fix compaction with additional messages. Now when we look for additional messages, we gracefully handle the processed messages. [AMQ-8354] Enable replication queue protection for replica. Remove source role as it's no longer supported. [AMQ-8354] ReplicationPlugin consumer is aborted by abortSlowAckConsumerStrategy [AMQ-8354] Fixed sequence recovery. [AMQ-8354] topic unsubscribe durable subscription event is not replicated [AMQ-8354] Replication queues purge notification [AMQ-8354] fix acknowledge replication message from AMQP protocol [AMQ-8354] Soft failover implementation [AMQ-8354] Change batch ack to individual acks. [AMQ-8354] Fix transactions in compactor. Simplify the logic to make it more error prone. [AMQ-8354] Replication event is out of order. Current sequence 1 belongs to message with id [AMQ-8354] Isolate context to prevent concurrent modifications. [AMQ-8354] Add web console access control. [AMQ-8354] Save Broker failover state and make failover more resilient to failure [AMQ-8354] Fix send to main queue when there is only one message in the batch. Fix sequence validation for FAIL_OVER messages. [AMQ-8354] Remove scheduled messages replication support. [AMQ-8354] bug fix - Virtual destinations replication works incorrectly [AMQ-8354] Fix acks when they are a single message in a batch. [AMQ-8354] Ignore MessageAck and Transactional replication events if corresponding entities do not exist [AMQ-8354] Redelivery plugin support. [AMQ-8354] Refactoring. The goals of it was: 1. make sure the role field in ReplicaRoleManagementBroker and the role saved in the storage align. 2. force and not force failovers follow the same flow as much as possible. 3. The failover flows for source and replica look as similar as possible. 4. existing code is reused as much as possible Bugs that were fixed: 1. connections are not blocked on start 2. after failover JMX returns wrong role 3. during soft failover there is a chance that the failover message is acked, but the role is not updated. 4. race condition during soft failover that may lead to incorrect role change [AMQ-8354] Add handling failures during failover on replica side. [AMQ-8354] Fix Not authorized to access destination: topic://ActiveMQ.Plugin.Replication.Role.Advisory.Topic [AMQ-8354] Fix role switching when there is an ongoing failover already. [AMQ-8354] Less invasive implementation of advisory suppresor. [AMQ-8354] Small refactoring. [AMQ-8354] Fix and add tests. [AMQ-8354] Replication plugin basic functionality tests - part 1. [AMQ-8354] Replication plugin basic function tests - replication event handling [AMQ-8354] Replication plugin basic function test - Replication event Ack [AMQ-8354] plugin test: queue operations [AMQ-8354] plugin test: Topic operations [AMQ-8354] add message property replication test [AMQ-8354] replication NetworkConnector tests [AMQ-8354] plugin Connection level protocol connection tests [AMQ-8354] added Connection mode tests and amqp connection test [AMQ-8354] Add Replication Queue Operations Tests [AMQ-8354] Enable testDurableSubscribers [AMQ-8354] Replication test: hard failover [AMQ-8354] refactor integration tests [AMQ-8354] Replication Tests: soft failover tests [AMQ-8354] refactor integration tests [AMQ-8354] Fix integration tests. [AMQ-8354] add replication redelivery plugin test [AMQ-8354] fix Replica Plugin Queue Test [AMQ-8354] Remove unused imports. [AMQ-8354] Add missing licenses. [AMQ-8354] Fix classloader issue. Improve failover logs. [AMQ-8354] Add heart beat messages. [AMQ-8354] Add versioning. [AMQ-8354] Throw exception on replication errors. [AMQ-8354] Delete TODOs and FIXMEs [AMQ-8354] add replication lag and wait time metrics. [AMQ-8394] Don't disable jolokia server. add replication plugin test profile: replica-plugin Fix slow ack replication. [AMQ-8354] Fix Replication event is out of order on broker restart. [AMQ-8354] Force producer flow control. [AMQ-8354] Add replication flow control. [AMQ-8354] Add an error log if a replication message is being sent to DLQ. [AMQ-8354] Add JMX metric to monitor replication flow control. mirrored queue does not mirror replication queues (#25) Message expired failure when the destination doesn't exist Fix Formatting fix flaky Replication Integration tests fix flaky test: ReplicaAcknowledgeReplicationEventTest Fixup after rebase/squash Signed-off-by: Kartik Ganesh --- .../amqp/ReplicaPluginAmqpConnectionTest.java | 191 ++++ .../amqp/transport-protocol-test-primary.xml | 47 + .../amqp/transport-protocol-test-replica.xml | 47 + activemq-broker/pom.xml | 11 + .../apache/activemq/broker/region/Queue.java | 1 + .../replica/ActiveMQReplicaException.java | 28 + .../replica/DestinationExtractor.java | 50 + .../activemq/replica/DummyConnection.java | 138 +++ .../activemq/replica/MutativeRoleBroker.java | 97 ++ .../activemq/replica/PeriodAcknowledge.java | 77 ++ .../activemq/replica/ReplicaAckHelper.java | 84 ++ .../replica/ReplicaAdvisorySuppressor.java | 69 ++ .../replica/ReplicaAuthorizationBroker.java | 140 +++ .../activemq/replica/ReplicaBatcher.java | 94 ++ .../activemq/replica/ReplicaBroker.java | 328 ++++++ .../replica/ReplicaBrokerEventListener.java | 652 +++++++++++ .../activemq/replica/ReplicaCompactor.java | 442 ++++++++ .../replica/ReplicaDestinationFilter.java | 79 ++ .../ReplicaDestinationInterceptor.java | 47 + .../apache/activemq/replica/ReplicaEvent.java | 97 ++ .../activemq/replica/ReplicaEventRetrier.java | 62 + .../replica/ReplicaEventSerializer.java | 94 ++ .../activemq/replica/ReplicaEventType.java | 40 + .../ReplicaInternalMessageProducer.java | 50 + .../ReplicaMirroredDestinationFilter.java | 43 + ...ReplicaMirroredDestinationInterceptor.java | 45 + .../activemq/replica/ReplicaPlugin.java | 244 ++++ .../activemq/replica/ReplicaPolicy.java | 163 +++ .../ReplicaReplicationQueueSupplier.java | 200 ++++ .../apache/activemq/replica/ReplicaRole.java | 40 + .../replica/ReplicaRoleManagement.java | 31 + .../replica/ReplicaRoleManagementBroker.java | 271 +++++ .../activemq/replica/ReplicaSequencer.java | 676 +++++++++++ .../activemq/replica/ReplicaSourceBroker.java | 785 +++++++++++++ .../activemq/replica/ReplicaStatistics.java | 148 +++ .../activemq/replica/ReplicaSupport.java | 99 ++ .../replica/ReplicationMessageProducer.java | 75 ++ .../replica/WebConsoleAccessController.java | 109 ++ .../replica/jmx/ReplicationJmxHelper.java | 41 + .../activemq/replica/jmx/ReplicationView.java | 83 ++ .../replica/jmx/ReplicationViewMBean.java | 49 + .../storage/ReplicaBaseSequenceStorage.java | 61 + .../replica/storage/ReplicaBaseStorage.java | 135 +++ .../ReplicaRecoverySequenceStorage.java | 56 + .../replica/storage/ReplicaRoleStorage.java | 57 + .../storage/ReplicaSequenceStorage.java | 54 + .../replica/DestinationExtractorTest.java | 52 + .../activemq/replica/ReplicaBatcherTest.java | 310 +++++ .../ReplicaBrokerEventListenerTest.java | 1010 +++++++++++++++++ .../replica/ReplicaCompactorTest.java | 185 +++ .../replica/ReplicaEventSerializerTest.java | 283 +++++ .../ReplicaInternalMessageProducerTest.java | 65 ++ .../ReplicaPluginInstallationTest.java | 85 ++ .../activemq/replica/ReplicaPluginTest.java | 197 ++++ .../activemq/replica/ReplicaPolicyTest.java | 43 + .../ReplicaReplicationQueueSupplierTest.java | 80 ++ .../ReplicaRoleManagementBrokerTest.java | 228 ++++ .../replica/ReplicaSequencerTest.java | 491 ++++++++ .../ReplicaSourceAuthorizationBrokerTest.java | 152 +++ .../replica/ReplicaSourceBrokerTest.java | 565 +++++++++ .../ReplicaRecoverySequenceStorageTest.java | 95 ++ .../storage/ReplicaSequenceStorageTest.java | 166 +++ .../org.mockito.plugins.MockMaker | 18 + activemq-unit-tests/pom.xml | 27 + ...eplicaAcknowledgeReplicationEventTest.java | 287 +++++ ...licaConnectionLevelMQTTConnectionTest.java | 268 +++++ .../replica/ReplicaConnectionModeTest.java | 168 +++ .../replica/ReplicaHardFailoverTest.java | 240 ++++ .../replica/ReplicaMessagePropertyTest.java | 273 +++++ .../replica/ReplicaNetworkConnectorTest.java | 353 ++++++ ...orsOnTwoPairsOfReplicationBrokersTest.java | 170 +++ .../replica/ReplicaPluginFunctionsTest.java | 210 ++++ .../replica/ReplicaPluginMirrorQueueTest.java | 207 ++++ ...icaPluginPersistentBrokerFunctionTest.java | 197 ++++ .../replica/ReplicaPluginQueueTest.java | 539 +++++++++ .../replica/ReplicaPluginTestSupport.java | 270 +++++ .../replica/ReplicaPluginTopicTest.java | 479 ++++++++ .../ReplicaPluginVirtualDestinationTest.java | 237 ++++ .../ReplicaProtocolConnectionTest.java | 180 +++ .../ReplicaProtocolStompConnectionTest.java | 188 +++ .../replica/ReplicaRedeliveryPluginTest.java | 196 ++++ .../replica/ReplicaSoftFailoverTest.java | 358 ++++++ .../replica/ReplicationEventHandlingTest.java | 272 +++++ .../ReplicationQueueOperationsTest.java | 169 +++ .../transport-protocol-test-primary.xml | 58 + .../transport-protocol-test-replica.xml | 58 + 86 files changed, 15859 insertions(+) create mode 100644 activemq-amqp/src/test/java/org/apache/activemq/transport/amqp/ReplicaPluginAmqpConnectionTest.java create mode 100644 activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-primary.xml create mode 100644 activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-replica.xml create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ActiveMQReplicaException.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/DestinationExtractor.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/DummyConnection.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/MutativeRoleBroker.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/PeriodAcknowledge.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAckHelper.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAdvisorySuppressor.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAuthorizationBroker.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBatcher.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBroker.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBrokerEventListener.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaCompactor.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationFilter.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationInterceptor.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEvent.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventRetrier.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventSerializer.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventType.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaInternalMessageProducer.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationFilter.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationInterceptor.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPlugin.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPolicy.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplier.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRole.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagement.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagementBroker.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSequencer.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSourceBroker.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaStatistics.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSupport.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/ReplicationMessageProducer.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/WebConsoleAccessController.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationJmxHelper.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationView.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationViewMBean.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseSequenceStorage.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseStorage.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorage.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRoleStorage.java create mode 100644 activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaSequenceStorage.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/DestinationExtractorTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBatcherTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBrokerEventListenerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaCompactorTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaEventSerializerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaInternalMessageProducerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginInstallationTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPolicyTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplierTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaRoleManagementBrokerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSequencerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceAuthorizationBrokerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceBrokerTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorageTest.java create mode 100644 activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaSequenceStorageTest.java create mode 100644 activemq-broker/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaAcknowledgeReplicationEventTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionLevelMQTTConnectionTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionModeTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaHardFailoverTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaMessagePropertyTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorsOnTwoPairsOfReplicationBrokersTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginFunctionsTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginMirrorQueueTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginPersistentBrokerFunctionTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginQueueTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTestSupport.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTopicTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginVirtualDestinationTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolConnectionTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolStompConnectionTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaRedeliveryPluginTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaSoftFailoverTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationEventHandlingTest.java create mode 100644 activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationQueueOperationsTest.java create mode 100644 activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-primary.xml create mode 100644 activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-replica.xml diff --git a/activemq-amqp/src/test/java/org/apache/activemq/transport/amqp/ReplicaPluginAmqpConnectionTest.java b/activemq-amqp/src/test/java/org/apache/activemq/transport/amqp/ReplicaPluginAmqpConnectionTest.java new file mode 100644 index 00000000000..feedb8f6903 --- /dev/null +++ b/activemq-amqp/src/test/java/org/apache/activemq/transport/amqp/ReplicaPluginAmqpConnectionTest.java @@ -0,0 +1,191 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.transport.amqp; + +import org.apache.activemq.broker.BrokerFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.spring.SpringSslContext; +import org.apache.activemq.test.TestSupport; +import org.apache.activemq.transport.amqp.protocol.AmqpConnection; +import org.apache.qpid.jms.JmsConnection; +import org.apache.qpid.jms.JmsConnectionFactory; +import org.junit.After; +import org.junit.Before; +import org.junit.Ignore; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.net.ssl.KeyManager; +import javax.net.ssl.SSLContext; +import javax.net.ssl.TrustManager; +import java.io.File; +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.security.SecureRandom; +import java.util.Arrays; +import java.util.Collection; + +@RunWith(Parameterized.class) +public class ReplicaPluginAmqpConnectionTest extends TestSupport { + + private static final Logger LOG = LoggerFactory.getLogger(ReplicaPluginAmqpConnectionTest.class); + public static final String KEYSTORE_TYPE = "jks"; + public static final String PASSWORD = "password"; + private final SpringSslContext sslContext = new SpringSslContext(); + private static final long LONG_TIMEOUT = 15000; + + public static final String PRIMARY_BROKER_CONFIG = "org/apache/activemq/transport/amqp/transport-protocol-test-primary.xml"; + public static final String REPLICA_BROKER_CONFIG = "org/apache/activemq/transport/amqp/transport-protocol-test-replica.xml"; + private final String protocol; + protected BrokerService firstBroker; + protected BrokerService secondBroker; + private JmsConnection firstBrokerConnection; + private JmsConnection secondBrokerConnection; + protected ActiveMQDestination destination; + + @Before + public void setUp() throws Exception { + SSLContext ctx = SSLContext.getInstance("TLS"); + ctx.init(new KeyManager[0], new TrustManager[]{new DefaultTrustManager()}, new SecureRandom()); + SSLContext.setDefault(ctx); + final File classesDir = new File(AmqpConnection.class.getProtectionDomain().getCodeSource().getLocation().getFile()); + File keystore = new File(classesDir, "../../src/test/resources/keystore"); + final SpringSslContext sslContext = new SpringSslContext(); + sslContext.setKeyStore(keystore.getCanonicalPath()); + sslContext.setKeyStorePassword("password"); + sslContext.setTrustStore(keystore.getCanonicalPath()); + sslContext.setTrustStorePassword("password"); + sslContext.afterPropertiesSet(); + System.setProperty("javax.net.ssl.trustStore", keystore.getCanonicalPath()); + System.setProperty("javax.net.ssl.trustStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.trustStoreType", KEYSTORE_TYPE); + System.setProperty("javax.net.ssl.keyStore", keystore.getCanonicalPath()); + System.setProperty("javax.net.ssl.keyStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.keyStoreType", KEYSTORE_TYPE); + + firstBroker = setUpBrokerService(PRIMARY_BROKER_CONFIG); + secondBroker = setUpBrokerService(REPLICA_BROKER_CONFIG); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + destination = new ActiveMQQueue(getClass().getName()); + } + + @After + public void tearDown() throws Exception { + firstBrokerConnection.close(); + secondBrokerConnection.close(); + if (firstBroker != null) { + try { + firstBroker.stop(); + firstBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + if (secondBroker != null) { + try { + secondBroker.stop(); + secondBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + } + + @Parameterized.Parameters(name="protocol={0}") + public static Collection getTestParameters() { + return Arrays.asList(new String[][] { + {"amqp"}, {"amqp+ssl"}, {"amqp+nio+ssl"}, {"amqp+nio"}, + }); + } + + @Test + @Ignore + public void messageSendAndReceive() throws Exception { + JmsConnectionFactory firstBrokerFactory = createConnectionFactory(firstBroker.getTransportConnectorByScheme(protocol)); + firstBrokerConnection = (JmsConnection) firstBrokerFactory.createConnection(); + firstBrokerConnection.setClientID("testMessageSendAndReceive-" + System.currentTimeMillis()); + secondBrokerConnection = (JmsConnection) createConnectionFactory(secondBroker.getTransportConnectorByScheme(protocol)).createConnection(); + secondBrokerConnection.setClientID("testMessageSendAndReceive-" + System.currentTimeMillis()); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + + Connection firstBrokerConsumerConnection = JMSClientContext.INSTANCE.createConnection(URI.create(protocol + "://localhost:" + firstBroker.getTransportConnectorByScheme(protocol).getConnectUri().getPort())); + firstBrokerConsumerConnection.start(); + Session firstBrokerConsumerSession = firstBrokerConsumerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer firstBrokerConsumer = firstBrokerConsumerSession.createConsumer(destination); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + receivedMessage.acknowledge(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + private JmsConnectionFactory createConnectionFactory(TransportConnector connector) throws IOException, URISyntaxException { + return new JmsConnectionFactory(protocol + "://localhost:" + connector.getConnectUri().getPort()); + } + + public ReplicaPluginAmqpConnectionTest(String protocol) { + this.protocol = protocol; + } + + protected BrokerService setUpBrokerService(String configurationUri) throws Exception { + BrokerService broker = createBroker(configurationUri); + broker.setPersistent(false); + broker.setSslContext(sslContext); + return broker; + } + + protected BrokerService createBroker(String uri) throws Exception { + LOG.info("Loading broker configuration from the classpath with URI: " + uri); + return BrokerFactory.createBroker(new URI("xbean:" + uri)); + } +} diff --git a/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-primary.xml b/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-primary.xml new file mode 100644 index 00000000000..04c011c6e94 --- /dev/null +++ b/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-primary.xml @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-replica.xml b/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-replica.xml new file mode 100644 index 00000000000..3a0d800e8fc --- /dev/null +++ b/activemq-amqp/src/test/resources/org/apache/activemq/transport/amqp/transport-protocol-test-replica.xml @@ -0,0 +1,47 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/activemq-broker/pom.xml b/activemq-broker/pom.xml index 0f34b1d486c..a25ce73ddae 100644 --- a/activemq-broker/pom.xml +++ b/activemq-broker/pom.xml @@ -89,6 +89,17 @@ com.fasterxml.jackson.core jackson-databind + + org.assertj + assertj-core + 3.11.1 + test + + + org.mockito + mockito-core + test + diff --git a/activemq-broker/src/main/java/org/apache/activemq/broker/region/Queue.java b/activemq-broker/src/main/java/org/apache/activemq/broker/region/Queue.java index 493a3b12a94..912008113b1 100644 --- a/activemq-broker/src/main/java/org/apache/activemq/broker/region/Queue.java +++ b/activemq-broker/src/main/java/org/apache/activemq/broker/region/Queue.java @@ -2484,6 +2484,7 @@ private QueueMessageReference getMatchingMessage(MessageDispatchNotification mes do { doPageIn(true, false, getMaxPageSize()); pagedInMessagesLock.readLock().lock(); + List list = new ArrayList<>(); try { if (pagedInMessages.size() == size) { // nothing new to check - mem constraint on page in diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ActiveMQReplicaException.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ActiveMQReplicaException.java new file mode 100644 index 00000000000..7dffabc0102 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ActiveMQReplicaException.java @@ -0,0 +1,28 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +public class ActiveMQReplicaException extends RuntimeException { + + public ActiveMQReplicaException(String message) { + super(message); + } + + public ActiveMQReplicaException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/DestinationExtractor.java b/activemq-broker/src/main/java/org/apache/activemq/replica/DestinationExtractor.java new file mode 100644 index 00000000000..85cc273f034 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/DestinationExtractor.java @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.region.BaseDestination; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.Topic; + +public class DestinationExtractor { + + public static Queue extractQueue(Destination destination) { + return extract(destination, Queue.class); + } + + static Topic extractTopic(Destination destination) { + return extract(destination, Topic.class); + } + + static BaseDestination extractBaseDestination(Destination destination) { + return extract(destination, BaseDestination.class); + } + + private static T extract(Destination destination, Class clazz) { + Destination result = destination; + while (result != null && !clazz.isInstance(result)) { + if (result instanceof DestinationFilter) { + result = ((DestinationFilter) result).getNext(); + } else { + return null; + } + } + return (T) result; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/DummyConnection.java b/activemq-broker/src/main/java/org/apache/activemq/replica/DummyConnection.java new file mode 100644 index 00000000000..fa45d0e67f9 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/DummyConnection.java @@ -0,0 +1,138 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Connection; +import org.apache.activemq.broker.Connector; +import org.apache.activemq.broker.region.ConnectionStatistics; +import org.apache.activemq.command.Command; +import org.apache.activemq.command.ConnectionControl; +import org.apache.activemq.command.Response; + +import java.io.IOException; + +class DummyConnection implements Connection { + @Override + public Connector getConnector() { + return null; + } + + @Override + public void dispatchSync(Command message) { + } + + @Override + public void dispatchAsync(Command command) { + } + + @Override + public Response service(Command command) { + return null; + } + + @Override + public void serviceException(Throwable error) { + } + + @Override + public boolean isSlow() { + return false; + } + + @Override + public boolean isBlocked() { + return false; + } + + @Override + public boolean isConnected() { + return false; + } + + @Override + public boolean isActive() { + return false; + } + + @Override + public int getDispatchQueueSize() { + return 0; + } + + @Override + public ConnectionStatistics getStatistics() { + return null; + } + + @Override + public boolean isManageable() { + return false; + } + + @Override + public String getRemoteAddress() { + return null; + } + + @Override + public void serviceExceptionAsync(IOException e) { + + } + + @Override + public String getConnectionId() { + return null; + } + + @Override + public boolean isNetworkConnection() { + return false; + } + + @Override + public boolean isFaultTolerantConnection() { + return false; + } + + @Override + public void updateClient(ConnectionControl control) { + + } + + @Override + public int getActiveTransactionCount() { + return 0; + } + + @Override + public Long getOldestActiveTransactionDuration() { + return null; + } + + @Override + public Long getConnectedTimestamp() { + return null; + } + + @Override + public void start() throws Exception { + } + + @Override + public void stop() throws Exception { + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/MutativeRoleBroker.java b/activemq-broker/src/main/java/org/apache/activemq/replica/MutativeRoleBroker.java new file mode 100644 index 00000000000..72fcad41c29 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/MutativeRoleBroker.java @@ -0,0 +1,97 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerFilter; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.TransactionId; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.ConcurrentHashMap; + +public abstract class MutativeRoleBroker extends BrokerFilter { + + private final Logger logger = LoggerFactory.getLogger(MutativeRoleBroker.class); + + private final ReplicaRoleManagement management; + + public MutativeRoleBroker(Broker broker, ReplicaRoleManagement management) { + super(broker); + this.management = management; + } + + public abstract void start(ReplicaRole role) throws Exception; + + abstract void stopBeforeRoleChange(boolean force) throws Exception; + + abstract void startAfterRoleChange() throws Exception; + + abstract void brokerServiceStarted(ReplicaRole role); + + void updateBrokerState(ReplicaRole role) throws Exception { + ConnectionContext connectionContext = createConnectionContext(); + LocalTransactionId tid = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + + super.beginTransaction(connectionContext, tid); + try { + updateBrokerState(connectionContext, tid, role); + super.commitTransaction(connectionContext, tid, true); + } catch (Exception e) { + super.rollbackTransaction(connectionContext, tid); + logger.error("Failed to ack fail over message", e); + throw e; + } + } + + void updateBrokerState(ConnectionContext connectionContext, TransactionId tid, ReplicaRole role) throws Exception { + management.updateBrokerState(connectionContext, tid, role); + } + + void stopAllConnections() { + management.stopAllConnections(); + } + + void startAllConnections() throws Exception { + management.startAllConnections(); + } + + void removeReplicationQueues() throws Exception { + for (String queueName : ReplicaSupport.DELETABLE_REPLICATION_DESTINATION_NAMES) { + super.removeDestination(createConnectionContext(), new ActiveMQQueue(queueName), 1000); + } + } + + void onStopSuccess() throws Exception { + management.onStopSuccess(); + } + + ConnectionContext createConnectionContext() { + ConnectionContext connectionContext = getAdminConnectionContext().copy(); + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + + return connectionContext; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/PeriodAcknowledge.java b/activemq-broker/src/main/java/org/apache/activemq/replica/PeriodAcknowledge.java new file mode 100644 index 00000000000..cdf1d011f41 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/PeriodAcknowledge.java @@ -0,0 +1,77 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.ActiveMQSession; + +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.AtomicReference; + +public class PeriodAcknowledge { + + private boolean safeToAck = true; + private final AtomicLong lastAckTime = new AtomicLong(); + private final AtomicInteger pendingAckCount = new AtomicInteger(); + private final AtomicReference connection = new AtomicReference<>(); + private final AtomicReference connectionSession = new AtomicReference<>(); + private final ReplicaPolicy replicaPolicy; + private final Object periodicCommitLock = new Object(); + + + public PeriodAcknowledge(ReplicaPolicy replicaPolicy) { + this.replicaPolicy = replicaPolicy; + } + + public void setConnection(ActiveMQConnection activeMQConnection) { + connection.set(activeMQConnection); + } + + public void setConnectionSession(ActiveMQSession activeMQSession) { + connectionSession.set(activeMQSession); + } + + public void setSafeToAck(boolean safeToAck) { + this.safeToAck = safeToAck; + } + + private boolean shouldPeriodicallyCommit() { + return System.currentTimeMillis() - lastAckTime.get() >= replicaPolicy.getReplicaAckPeriod(); + } + + private boolean reachedMaxAckBatchSize() { + return pendingAckCount.incrementAndGet() >= replicaPolicy.getReplicaMaxAckBatchSize(); + } + + public void acknowledge() throws Exception { + acknowledge(false); + } + public void acknowledge(boolean forceAcknowledge) throws Exception { + if (connection.get() == null || connectionSession.get() == null || !safeToAck) { + return; + } + + synchronized (periodicCommitLock) { + if (reachedMaxAckBatchSize() || shouldPeriodicallyCommit() || forceAcknowledge) { + connectionSession.get().acknowledge(); + lastAckTime.set(System.currentTimeMillis()); + pendingAckCount.set(0); + } + } + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAckHelper.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAckHelper.java new file mode 100644 index 00000000000..bd260e8cdb5 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAckHelper.java @@ -0,0 +1,84 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; + +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +public class ReplicaAckHelper { + + private final Broker broker; + + public ReplicaAckHelper(Broker broker) { + this.broker = broker; + } + + public List getMessagesToAck(MessageAck ack, Destination destination) { + PrefetchSubscription prefetchSubscription = getPrefetchSubscription(destination, ack.getConsumerId()); + if (prefetchSubscription == null) { + return null; + } + + return getMessagesToAck(ack, prefetchSubscription); + } + + public List getMessagesToAck(MessageAck ack, PrefetchSubscription subscription) { + List dispatched = subscription.getDispatched(); + if (ack.isStandardAck() || ack.isExpiredAck() || ack.isPoisonAck()) { + boolean inAckRange = false; + List removeList = new ArrayList<>(); + for (final MessageReference node : dispatched) { + MessageId messageId = node.getMessageId(); + if (ack.getFirstMessageId() == null || ack.getFirstMessageId().equals(messageId)) { + inAckRange = true; + } + if (inAckRange) { + removeList.add(node); + if (ack.getLastMessageId().equals(messageId)) { + break; + } + } + } + + return removeList; + } + + if (ack.isIndividualAck()) { + return dispatched.stream() + .filter(mr -> mr.getMessageId().equals(ack.getLastMessageId())) + .collect(Collectors.toList()); + } + + return null; + } + + private PrefetchSubscription getPrefetchSubscription(Destination destination, ConsumerId consumerId) { + return destination.getConsumers().stream() + .filter(c -> c.getConsumerInfo().getConsumerId().equals(consumerId)) + .findFirst().filter(PrefetchSubscription.class::isInstance).map(PrefetchSubscription.class::cast) + .orElse(null); + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAdvisorySuppressor.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAdvisorySuppressor.java new file mode 100644 index 00000000000..454ede29bbb --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAdvisorySuppressor.java @@ -0,0 +1,69 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.advisory.AdvisorySupport; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.Message; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class ReplicaAdvisorySuppressor implements DestinationInterceptor { + + private final Logger logger = LoggerFactory.getLogger(ReplicaAdvisorySuppressor.class); + + @Override + public Destination intercept(Destination destination) { + if (AdvisorySupport.isAdvisoryTopic(destination.getActiveMQDestination())) { + return new ReplicaAdvisorySuppressionFilter(destination); + } + return destination; + } + + @Override + public void remove(Destination destination) { + } + + @Override + public void create(Broker broker, ConnectionContext context, ActiveMQDestination destination) throws Exception { + } + + private static class ReplicaAdvisorySuppressionFilter extends DestinationFilter { + + public ReplicaAdvisorySuppressionFilter(Destination next) { + super(next); + } + + @Override + public void send(ProducerBrokerExchange producerExchange, Message messageSend) throws Exception { + if (messageSend.isAdvisory()) { + if (messageSend.getDestination().getPhysicalName().contains(ReplicaSupport.REPLICATION_QUEUE_PREFIX)) { + // NoB relies on advisory messages for AddConsumer. + // Suppress these messages for replication queues so that the replication queues are ignored by NoB. + return; + } + } + super.send(producerExchange, messageSend); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAuthorizationBroker.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAuthorizationBroker.java new file mode 100644 index 00000000000..e0ede793cc8 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaAuthorizationBroker.java @@ -0,0 +1,140 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerFilter; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.ProducerInfo; +import org.apache.activemq.security.SecurityContext; + +import java.util.Arrays; + +public class ReplicaAuthorizationBroker extends BrokerFilter { + + public ReplicaAuthorizationBroker(Broker next) { + super(next); + // add DestinationInterceptor + final RegionBroker regionBroker = (RegionBroker) next.getAdaptor(RegionBroker.class); + final CompositeDestinationInterceptor compositeInterceptor = (CompositeDestinationInterceptor) regionBroker.getDestinationInterceptor(); + DestinationInterceptor[] interceptors = compositeInterceptor.getInterceptors(); + interceptors = Arrays.copyOf(interceptors, interceptors.length + 1); + interceptors[interceptors.length - 1] = new ReplicaAuthorizationDestinationInterceptor(); + compositeInterceptor.setInterceptors(interceptors); + } + + @Override + public Subscription addConsumer(ConnectionContext context, ConsumerInfo info) throws Exception { + assertAuthorized(context, info.getDestination()); + return super.addConsumer(context, info); + } + + @Override + public void addProducer(ConnectionContext context, ProducerInfo producerInfo) throws Exception { + // JMS allows producers to be created without first specifying a destination. In these cases, every send + // operation must specify a destination. Because of this, we only authorize 'addProducer' if a destination is + // specified. If not specified, the authz check in the 'send' method below will ensure authorization. + if (producerInfo.getDestination() != null) { + assertAuthorized(context, producerInfo.getDestination()); + } + super.addProducer(context, producerInfo); + } + + @Override + public void removeDestination(ConnectionContext context, ActiveMQDestination destination, long timeout) throws Exception { + if (ReplicaSupport.isReplicationDestination(destination)) { + throw new ActiveMQReplicaException(createUnauthorizedMessage(destination)); + } + super.removeDestination(context, destination, timeout); + } + + private void assertAuthorized(ConnectionContext context, ActiveMQDestination destination) { + if (isAuthorized(context, destination)) { + return; + } + + throw new ActiveMQReplicaException(createUnauthorizedMessage(destination)); + } + + private static boolean isAuthorized(ConnectionContext context, ActiveMQDestination destination) { + boolean replicationQueue = ReplicaSupport.isReplicationDestination(destination); + boolean replicationTransport = ReplicaSupport.isReplicationTransport(context.getConnector()); + + if (isSystemBroker(context)) { + return true; + } + if (replicationTransport && (replicationQueue || ReplicaSupport.isAdvisoryDestination(destination))) { + return true; + } + if (!replicationTransport && !replicationQueue) { + return true; + } + return false; + } + + private static boolean isSystemBroker(ConnectionContext context) { + SecurityContext securityContext = context.getSecurityContext(); + return securityContext != null && securityContext.isBrokerContext(); + } + + private static String createUnauthorizedMessage(ActiveMQDestination destination) { + return "Not authorized to access destination: " + destination; + } + + private static class ReplicaAuthorizationDestinationInterceptor implements DestinationInterceptor { + + @Override + public Destination intercept(Destination destination) { + if (ReplicaSupport.isReplicationDestination(destination.getActiveMQDestination())) { + return new ReplicaAuthorizationDestinationFilter(destination); + } + return destination; + } + + @Override + public void remove(Destination destination) { + } + + @Override + public void create(Broker broker, ConnectionContext context, ActiveMQDestination destination) throws Exception { + } + } + + private static class ReplicaAuthorizationDestinationFilter extends DestinationFilter { + + + public ReplicaAuthorizationDestinationFilter(Destination next) { + super(next); + } + + @Override + public void addSubscription(ConnectionContext context, Subscription sub) throws Exception { + if (!isAuthorized(context, getActiveMQDestination())) { + throw new SecurityException(createUnauthorizedMessage(getActiveMQDestination())); + } + super.addSubscription(context, sub); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBatcher.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBatcher.java new file mode 100644 index 00000000000..4c916c98094 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBatcher.java @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.command.ActiveMQMessage; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +public class ReplicaBatcher { + + private ReplicaPolicy replicaPolicy; + + public ReplicaBatcher(ReplicaPolicy replicaPolicy) { + this.replicaPolicy = replicaPolicy; + } + + @SuppressWarnings("unchecked") + List> batches(List list) throws Exception { + List> result = new ArrayList<>(); + + Map> destination2eventType = new HashMap<>(); + List batch = new ArrayList<>(); + int batchSize = 0; + for (MessageReference reference : list) { + ActiveMQMessage message = (ActiveMQMessage) reference.getMessage(); + String originalDestination = message.getStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY); + ReplicaEventType currentEventType = + ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)); + + if (currentEventType == ReplicaEventType.FAIL_OVER) { + if (batch.size() > 0) { + result.add(batch); + batch = new ArrayList<>(); + batchSize = 0; + } + batch.add(reference); + result.add(batch); + batch = new ArrayList<>(); + continue; + } + + boolean eventTypeSwitch = false; + if (originalDestination != null) { + Set sends = destination2eventType.computeIfAbsent(originalDestination, k -> new HashSet<>()); + if (currentEventType == ReplicaEventType.MESSAGE_SEND) { + sends.add(message.getStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY)); + } + if (currentEventType == ReplicaEventType.MESSAGE_ACK) { + List stringProperty = (List) message.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY); + if (sends.stream().anyMatch(stringProperty::contains)) { + destination2eventType.put(originalDestination, new HashSet<>()); + eventTypeSwitch = true; + } + } + } + + boolean exceedsLength = batch.size() + 1 > replicaPolicy.getMaxBatchLength(); + boolean exceedsSize = batchSize + reference.getSize() > replicaPolicy.getMaxBatchSize(); + if (batch.size() > 0 && (exceedsLength || exceedsSize || eventTypeSwitch)) { + result.add(batch); + batch = new ArrayList<>(); + batchSize = 0; + } + + batch.add(reference); + batchSize += reference.getSize(); + } + if (batch.size() > 0) { + result.add(batch); + } + + return result; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBroker.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBroker.java new file mode 100644 index 00000000000..acec9e13ba2 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBroker.java @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQMessageConsumer; +import org.apache.activemq.ActiveMQPrefetchPolicy; +import org.apache.activemq.ActiveMQSession; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.MessageDispatch; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.JMSException; +import javax.jms.MessageConsumer; +import java.lang.reflect.Method; +import java.text.MessageFormat; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; + + +public class ReplicaBroker extends MutativeRoleBroker { + + private final Logger logger = LoggerFactory.getLogger(ReplicaBroker.class); + private final ScheduledExecutorService brokerConnectionPoller = Executors.newSingleThreadScheduledExecutor(); + private final ScheduledExecutorService periodicAckPoller = Executors.newSingleThreadScheduledExecutor(); + private final AtomicBoolean isConnecting = new AtomicBoolean(); + private final AtomicReference connection = new AtomicReference<>(); + private final AtomicReference connectionSession = new AtomicReference<>(); + private final AtomicReference eventConsumer = new AtomicReference<>(); + private final ReplicaReplicationQueueSupplier queueProvider; + private final ReplicaPolicy replicaPolicy; + private final PeriodAcknowledge periodAcknowledgeCallBack; + private final ReplicaStatistics replicaStatistics; + private ReplicaBrokerEventListener messageListener; + private ScheduledFuture replicationScheduledFuture; + private ScheduledFuture ackPollerScheduledFuture; + + public ReplicaBroker(Broker broker, ReplicaRoleManagement management, ReplicaReplicationQueueSupplier queueProvider, + ReplicaPolicy replicaPolicy, ReplicaStatistics replicaStatistics) { + super(broker, management); + this.queueProvider = queueProvider; + this.replicaPolicy = replicaPolicy; + this.periodAcknowledgeCallBack = new PeriodAcknowledge(replicaPolicy); + this.replicaStatistics = replicaStatistics; + } + + @Override + public void start(ReplicaRole role) throws Exception { + init(role); + + logger.info("Starting replica broker." + + (role == ReplicaRole.ack_processed ? " Ack has been processed. Checking the role of the other broker." : "")); + } + + @Override + public void brokerServiceStarted(ReplicaRole role) { + stopAllConnections(); + } + + @Override + public void stop() throws Exception { + logger.info("Stopping Source broker"); + deinitialize(); + super.stop(); + } + + @Override + public void stopBeforeRoleChange(boolean force) throws Exception { + if (!force) { + return; + } + logger.info("Stopping broker replication. Forced: [{}]", force); + + updateBrokerState(ReplicaRole.source); + completeBeforeRoleChange(); + } + + @Override + public void startAfterRoleChange() throws Exception { + logger.info("Starting Replica broker"); + init(ReplicaRole.replica); + } + + void completeBeforeRoleChange() throws Exception { + messageListener.deinitialize(); + removeReplicationQueues(); + deinitialize(); + onStopSuccess(); + } + + private void init(ReplicaRole role) { + logger.info("Initializing Replica broker"); + queueProvider.initializeSequenceQueue(); + replicationScheduledFuture = brokerConnectionPoller.scheduleAtFixedRate(() -> beginReplicationIdempotent(role), 5, 5, TimeUnit.SECONDS); + ackPollerScheduledFuture = periodicAckPoller.scheduleAtFixedRate(() -> { + synchronized (periodAcknowledgeCallBack) { + try { + periodAcknowledgeCallBack.acknowledge(); + } catch (Exception e) { + logger.error("Failed to Acknowledge replication Queue message {}", e.getMessage()); + } + } + }, replicaPolicy.getReplicaAckPeriod(), replicaPolicy.getReplicaAckPeriod(), TimeUnit.MILLISECONDS); + messageListener = new ReplicaBrokerEventListener(this, queueProvider, periodAcknowledgeCallBack, replicaPolicy, replicaStatistics); + } + + private void deinitialize() throws JMSException { + if (replicationScheduledFuture != null) { + replicationScheduledFuture.cancel(true); + } + if (ackPollerScheduledFuture != null) { + ackPollerScheduledFuture.cancel(true); + } + + ActiveMQMessageConsumer consumer = eventConsumer.get(); + ActiveMQSession session = connectionSession.get(); + ActiveMQConnection brokerConnection = connection.get(); + if (messageListener != null) { + messageListener.close(); + } + if (consumer != null) { + consumer.stop(); + consumer.close(); + } + if (session != null) { + session.close(); + } + if (brokerConnection != null && brokerConnection.isStarted()) { + brokerConnection.stop(); + brokerConnection.close(); + } + + eventConsumer.set(null); + connectionSession.set(null); + connection.set(null); + replicationScheduledFuture = null; + ackPollerScheduledFuture = null; + } + + @Override + public boolean sendToDeadLetterQueue(ConnectionContext context, MessageReference messageReference, Subscription subscription, Throwable poisonCause) { + // suppressing actions on the replica side. Expecting them to be replicated + return false; + } + + @Override + public boolean isExpired(MessageReference messageReference) { + // suppressing actions on the replica side. Expecting them to be replicated + return false; + } + + private void beginReplicationIdempotent(ReplicaRole initialRole) { + if (connectionSession.get() == null) { + logger.debug("Establishing inter-broker replication connection"); + establishConnectionSession(); + } + if (eventConsumer.get() == null) { + try { + logger.debug("Creating replica event consumer"); + consumeReplicationEvents(initialRole); + } catch (Exception e) { + logger.error("Could not establish replication consumer", e); + } + } + } + + private void establishConnectionSession() { + if (isConnecting.compareAndSet(false, true)) { + logger.debug("Trying to connect to replica source"); + try { + establishConnection(); + ActiveMQSession session = (ActiveMQSession) connection.get().createSession(false, ActiveMQSession.CLIENT_ACKNOWLEDGE); + session.setAsyncDispatch(false); // force the primary broker to block if we are slow + connectionSession.set(session); + periodAcknowledgeCallBack.setConnectionSession(session); + } catch (RuntimeException | JMSException e) { + logger.warn("Failed to establish connection to replica", e); + } finally { + if (connectionSession.get() == null) { + logger.info("Closing connection session after unsuccessful connection establishment"); + connection.getAndUpdate(conn -> { + try { + if (conn != null) { + conn.close(); + } + } catch (JMSException e) { + logger.error("Failed to close connection after session establishment failed", e); + } + return null; + }); + } + isConnecting.weakCompareAndSetPlain(true, false); + } + } + } + + private void establishConnection() throws JMSException { + ActiveMQConnectionFactory replicaSourceConnectionFactory = replicaPolicy.getOtherBrokerConnectionFactory(); + logger.trace("Replica connection URL {}", replicaSourceConnectionFactory.getBrokerURL()); + ActiveMQConnection newConnection = (ActiveMQConnection) replicaSourceConnectionFactory.createConnection(); + newConnection.setSendAcksAsync(false); + newConnection.start(); + connection.set(newConnection); + periodAcknowledgeCallBack.setConnection(newConnection); + logger.debug("Established connection to replica source: {}", replicaSourceConnectionFactory.getBrokerURL()); + } + + private void consumeReplicationEvents(ReplicaRole initialRole) throws Exception { + if (connectionUnusable() || sessionUnusable()) { + return; + } + if (initialRole == ReplicaRole.ack_processed) { + if (isReadyToFailover()) { + updateBrokerState(ReplicaRole.source); + completeBeforeRoleChange(); + return; + } + } + + ActiveMQQueue replicationSourceQueue = connection.get() + .getDestinationSource() + .getQueues() + .stream() + .filter(d -> ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME.equals(d.getPhysicalName())) + .findFirst() + .orElseThrow(() -> new IllegalStateException( + MessageFormat.format("There is no replication queue on the source broker {0}", replicaPolicy.getOtherBrokerConnectionFactory().getBrokerURL()) + )); + logger.info("Plugin will mirror events from queue {}", replicationSourceQueue.getPhysicalName()); + messageListener.initialize(); + ActiveMQPrefetchPolicy prefetchPolicy = connection.get().getPrefetchPolicy(); + Method getNextConsumerId = ActiveMQSession.class.getDeclaredMethod("getNextConsumerId"); + getNextConsumerId.setAccessible(true); + eventConsumer.set(new ActiveMQMessageConsumer(connectionSession.get(), (ConsumerId) getNextConsumerId.invoke(connectionSession.get()), replicationSourceQueue, null, null, prefetchPolicy.getQueuePrefetch(), + prefetchPolicy.getMaximumPendingMessageLimit(), false, false, connectionSession.get().isAsyncDispatch(), messageListener) { + @Override + public void dispatch(MessageDispatch md) { + synchronized (periodAcknowledgeCallBack) { + super.dispatch(md); + try { + periodAcknowledgeCallBack.acknowledge(); + } catch (Exception e) { + logger.error("Failed to acknowledge replication message [{}]", e); + } + } + } + }); + } + + private boolean isReadyToFailover() throws JMSException { + ActiveMQTopic replicationRoleAdvisoryTopic = connection.get() + .getDestinationSource() + .getTopics() + .stream() + .filter(d -> ReplicaSupport.REPLICATION_ROLE_ADVISORY_TOPIC_NAME.equals(d.getPhysicalName())) + .findFirst() + .orElseThrow(() -> new IllegalStateException( + MessageFormat.format("There is no replication role advisory topic on the source broker {0}", + replicaPolicy.getOtherBrokerConnectionFactory().getBrokerURL()) + )); + MessageConsumer advisoryConsumer = connectionSession.get().createConsumer(replicationRoleAdvisoryTopic); + ActiveMQTextMessage message = (ActiveMQTextMessage) advisoryConsumer.receive(5000); + if (message == null) { + throw new IllegalStateException("There is no replication role in the role advisory topic on the source broker {0}" + + replicaPolicy.getOtherBrokerConnectionFactory().getBrokerURL()); + } + advisoryConsumer.close(); + return ReplicaRole.valueOf(message.getText()) == ReplicaRole.replica; + } + + + private boolean connectionUnusable() { + if (isConnecting.get()) { + logger.trace("Will not consume events because we are still connecting"); + return true; + } + ActiveMQConnection conn = connection.get(); + if (conn == null) { + logger.trace("Will not consume events because we don't have a connection"); + return true; + } + if (conn.isClosed() || conn.isClosing()) { + logger.trace("Will not consume events because the connection is not open"); + return true; + } + return false; + } + + private boolean sessionUnusable() { + ActiveMQSession session = connectionSession.get(); + if (session == null) { + logger.trace("Will not consume events because we don't have a session"); + return true; + } + if (session.isClosed()) { + logger.trace("Will not consume events because the session is not open"); + return true; + } + return false; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBrokerEventListener.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBrokerEventListener.java new file mode 100644 index 00000000000..aeb696e2fd0 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaBrokerEventListener.java @@ -0,0 +1,652 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ScheduledMessage; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerStoppedException; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.TransactionBroker; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DurableTopicSubscription; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageDispatchNotification; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.RemoveSubscriptionInfo; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.storage.ReplicaSequenceStorage; +import org.apache.activemq.transaction.Transaction; +import org.apache.activemq.usage.MemoryUsage; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.JMSException; +import javax.jms.Message; +import javax.jms.MessageListener; +import javax.transaction.xa.XAException; +import java.io.IOException; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicReference; + +import static java.util.Objects.requireNonNull; + +public class ReplicaBrokerEventListener implements MessageListener { + + private static final String REPLICATION_CONSUMER_CLIENT_ID = "DUMMY_REPLICATION_CONSUMER"; + private static final String SEQUENCE_NAME = "replicaSeq"; + private final Logger logger = LoggerFactory.getLogger(ReplicaBrokerEventListener.class); + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final ReplicaBroker replicaBroker; + private final Broker broker; + private final ConnectionContext connectionContext; + private final ReplicaInternalMessageProducer replicaInternalMessageProducer; + private final PeriodAcknowledge acknowledgeCallback; + private final ReplicaPolicy replicaPolicy; + private final ReplicaStatistics replicaStatistics; + private final MemoryUsage memoryUsage; + private final AtomicReference replicaEventRetrier = new AtomicReference<>(); + final ReplicaSequenceStorage sequenceStorage; + private final TransactionBroker transactionBroker; + + BigInteger sequence; + MessageId sequenceMessageId; + + ReplicaBrokerEventListener(ReplicaBroker replicaBroker, ReplicaReplicationQueueSupplier queueProvider, + PeriodAcknowledge acknowledgeCallback, ReplicaPolicy replicaPolicy, ReplicaStatistics replicaStatistics) { + this.replicaBroker = requireNonNull(replicaBroker); + this.broker = requireNonNull(replicaBroker.getNext()); + this.acknowledgeCallback = requireNonNull(acknowledgeCallback); + this.replicaPolicy = replicaPolicy; + this.replicaStatistics = replicaStatistics; + connectionContext = broker.getAdminConnectionContext().copy(); + connectionContext.setUserName(ReplicaSupport.REPLICATION_PLUGIN_USER_NAME); + connectionContext.setClientId(REPLICATION_CONSUMER_CLIENT_ID); + connectionContext.setConnection(new DummyConnection()); + replicaInternalMessageProducer = new ReplicaInternalMessageProducer(broker); + + createTransactionMapIfNotExist(); + + this.sequenceStorage = new ReplicaSequenceStorage(broker, + queueProvider, replicaInternalMessageProducer, SEQUENCE_NAME); + this.transactionBroker = (TransactionBroker) broker.getAdaptor(TransactionBroker.class); + + memoryUsage = broker.getBrokerService().getSystemUsage().getMemoryUsage(); + } + + public void initialize() throws Exception { + String savedSequence = sequenceStorage.initialize(connectionContext); + if (savedSequence == null) { + return; + } + + String[] split = savedSequence.split("#"); + if (split.length != 2) { + throw new IllegalStateException("Unknown sequence message format: " + savedSequence); + } + sequence = new BigInteger(split[0]); + + sequenceMessageId = new MessageId(split[1]); + } + + public void deinitialize() throws Exception { + sequenceStorage.deinitialize(connectionContext); + } + + @Override + public void onMessage(Message jmsMessage) { + logger.trace("Received replication message from replica source"); + ActiveMQMessage message = (ActiveMQMessage) jmsMessage; + + if (replicaPolicy.isReplicaReplicationFlowControl()) { + long start = System.currentTimeMillis(); + long nextWarn = start; + try { + while (!memoryUsage.waitForSpace(1000, 90)) { + replicaStatistics.setReplicaReplicationFlowControl(true); + long now = System.currentTimeMillis(); + if (now >= nextWarn) { + logger.warn("High memory usage. Pausing replication (paused for: {}s)", (now - start) / 1000); + nextWarn = now + 30000; + } + } + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + } + replicaStatistics.setReplicaReplicationFlowControl(false); + + try { + processMessageWithRetries(message, null); + } catch (BrokerStoppedException bse) { + logger.warn("The broker has been stopped"); + } catch (InterruptedException ie) { + logger.warn("Retrier interrupted: {}", ie.toString()); + } + } + + public void close() { + ReplicaEventRetrier retrier = replicaEventRetrier.get(); + if (retrier != null) { + retrier.stop(); + } + } + + private synchronized void processMessageWithRetries(ActiveMQMessage message, TransactionId transactionId) throws InterruptedException { + ReplicaEventRetrier retrier = new ReplicaEventRetrier(() -> { + boolean commit = false; + TransactionId tid = transactionId; + ReplicaEventType eventType = getEventType(message); + + if (tid == null && eventType != ReplicaEventType.FAIL_OVER) { + tid = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + + broker.beginTransaction(connectionContext, tid); + + commit = true; + } + + try { + if (eventType == ReplicaEventType.BATCH) { + processBatch(message, tid); + } else { + processMessage(message, eventType, tid); + } + + if (commit) { + sequenceStorage.enqueue(connectionContext, tid, sequence.toString() + "#" + sequenceMessageId); + + broker.commitTransaction(connectionContext, tid, true); + acknowledgeCallback.setSafeToAck(true); + } + } catch (Exception e) { + if (commit) { + broker.rollbackTransaction(connectionContext, tid); + } + acknowledgeCallback.setSafeToAck(false); + throw e; + } + return null; + }); + replicaEventRetrier.set(retrier); + try { + retrier.process(); + } finally { + replicaEventRetrier.set(null); + } + } + + private void processMessage(ActiveMQMessage message, ReplicaEventType eventType, TransactionId transactionId) throws Exception { + int messageVersion = message.getIntProperty(ReplicaSupport.VERSION_PROPERTY); + if (messageVersion > ReplicaSupport.CURRENT_VERSION) { + throw new IllegalStateException("Unsupported version of replication event: " + messageVersion + ". Maximum supported version: " + ReplicaSupport.CURRENT_VERSION); + } + Object deserializedData = eventSerializer.deserializeMessageData(message.getContent()); + BigInteger newSequence = new BigInteger(message.getStringProperty(ReplicaSupport.SEQUENCE_PROPERTY)); + + long sequenceDifference = sequence == null ? 0 : newSequence.subtract(sequence).longValue(); + MessageId messageId = message.getMessageId(); + if (sequence == null || sequenceDifference == 1) { + processMessage(message, eventType, deserializedData, transactionId); + + sequence = newSequence; + sequenceMessageId = messageId; + + } else if (sequenceDifference > 0) { + throw new IllegalStateException(String.format( + "Replication event is out of order. Current sequence: %s, the sequence of the event: %s", + sequence, newSequence)); + } else if (sequenceDifference < 0) { + logger.info(String.format( + "Replication message duplicate. Current sequence: %s, the sequence of the event: %s", + sequence, newSequence)); + } else if (!sequenceMessageId.equals(messageId)) { + throw new IllegalStateException(String.format( + "Replication event is out of order. Current sequence %s belongs to message with id %s," + + "but the id of the event is %s", sequence, sequenceMessageId, messageId)); + } + + long currentTime = System.currentTimeMillis(); + replicaStatistics.setReplicationLag(currentTime - message.getTimestamp()); + replicaStatistics.setReplicaLastProcessedTime(currentTime); + } + + private void processMessage(ActiveMQMessage message, ReplicaEventType eventType, Object deserializedData, + TransactionId transactionId) throws Exception { + switch (eventType) { + case DESTINATION_UPSERT: + logger.trace("Processing replicated destination"); + upsertDestination((ActiveMQDestination) deserializedData); + return; + case DESTINATION_DELETE: + logger.trace("Processing replicated destination deletion"); + deleteDestination((ActiveMQDestination) deserializedData); + return; + case MESSAGE_SEND: + logger.trace("Processing replicated message send"); + sendMessage((ActiveMQMessage) deserializedData, transactionId); + return; + case MESSAGE_ACK: + logger.trace("Processing replicated messages dropped"); + try { + messageAck((MessageAck) deserializedData, + (List) message.getObjectProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY), transactionId); + } catch (JMSException e) { + logger.error("Failed to extract property to replicate messages dropped [{}]", deserializedData, e); + throw new Exception(e); + } + return; + case QUEUE_PURGED: + logger.trace("Processing queue purge"); + purgeQueue((ActiveMQDestination) deserializedData); + return; + case TRANSACTION_BEGIN: + logger.trace("Processing replicated transaction begin"); + beginTransaction((TransactionId) deserializedData); + return; + case TRANSACTION_PREPARE: + logger.trace("Processing replicated transaction prepare"); + prepareTransaction((TransactionId) deserializedData); + return; + case TRANSACTION_FORGET: + logger.trace("Processing replicated transaction forget"); + forgetTransaction((TransactionId) deserializedData); + return; + case TRANSACTION_ROLLBACK: + logger.trace("Processing replicated transaction rollback"); + rollbackTransaction((TransactionId) deserializedData); + return; + case TRANSACTION_COMMIT: + logger.trace("Processing replicated transaction commit"); + try { + commitTransaction( + (TransactionId) deserializedData, + message.getBooleanProperty(ReplicaSupport.TRANSACTION_ONE_PHASE_PROPERTY)); + } catch (JMSException e) { + logger.error("Failed to extract property to replicate transaction commit with id [{}]", deserializedData, e); + throw new Exception(e); + } + return; + case ADD_DURABLE_CONSUMER: + logger.trace("Processing replicated add consumer"); + try { + addDurableConsumer((ConsumerInfo) deserializedData, + message.getStringProperty(ReplicaSupport.CLIENT_ID_PROPERTY)); + } catch (JMSException e) { + logger.error("Failed to extract property to replicate add consumer [{}]", deserializedData, e); + throw new Exception(e); + } + return; + case REMOVE_DURABLE_CONSUMER: + logger.trace("Processing replicated remove consumer"); + removeDurableConsumer((ConsumerInfo) deserializedData); + return; + case MESSAGE_EXPIRED: + logger.trace("Processing replicated message expired"); + messageExpired((ActiveMQMessage) deserializedData); + return; + case REMOVE_DURABLE_CONSUMER_SUBSCRIPTION: + logger.trace("Processing replicated remove durable consumer subscription"); + removeDurableConsumerSubscription((RemoveSubscriptionInfo) deserializedData); + return; + case FAIL_OVER: + failOver(); + return; + case HEART_BEAT: + logger.trace("Heart beat message received"); + return; + default: + throw new IllegalStateException( + String.format("Unhandled event type \"%s\" for replication message id: %s", + eventType, message.getJMSMessageID())); + } + } + + private boolean isDestinationExisted(ActiveMQDestination destination) throws Exception { + try { + return Arrays.stream(broker.getDestinations()) + .anyMatch(d -> d.getQualifiedName().equals(destination.getQualifiedName())); + } catch (Exception e) { + logger.error("Unable to determine if [{}] is an existing destination", destination, e); + throw e; + } + } + + private boolean isTransactionExisted(TransactionId transactionId) throws Exception { + try { + Transaction transaction = transactionBroker.getTransaction(connectionContext, transactionId, false); + return transaction != null; + } + catch (XAException e) { + logger.error("Transaction cannot be found - non-existing transaction [{}]", transactionId, e); + return false; + } + } + + private boolean isExceptionDueToNonExistingMessage(JMSException exception) { + if (exception.getMessage().contains("Slave broker out of sync with master - Message:") + && exception.getMessage().contains("does not exist among pending")) { + return true; + } + + return false; + } + + private void processBatch(ActiveMQMessage message, TransactionId tid) throws Exception { + List objects = eventSerializer.deserializeListOfObjects(message.getContent().getData()); + for (Object o : objects) { + processMessageWithRetries((ActiveMQMessage) o, tid); + } + } + + private void upsertDestination(ActiveMQDestination destination) throws Exception { + if (isDestinationExisted(destination)) { + logger.debug("Destination [{}] already exists, no action to take", destination); + return; + } + try { + broker.addDestination(connectionContext, destination, true); + } catch (Exception e) { + logger.error("Unable to add destination [{}]", destination, e); + throw e; + } + } + + private void deleteDestination(ActiveMQDestination destination) throws Exception { + if (!isDestinationExisted(destination)) { + logger.debug("Destination [{}] does not exist, no action to take", destination); + return; + } + try { + broker.removeDestination(connectionContext, destination, 1000); + } catch (Exception e) { + logger.error("Unable to remove destination [{}]", destination, e); + throw e; + } + } + + private ReplicaEventType getEventType(ActiveMQMessage message) throws JMSException { + return ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)); + } + + private void sendMessage(ActiveMQMessage message, TransactionId transactionId) throws Exception { + try { + if (message.getTransactionId() == null || !message.getTransactionId().isXATransaction()) { + message.setTransactionId(transactionId); + } + removeScheduledMessageProperties(message); + + if(message.getExpiration() > 0 && System.currentTimeMillis() + 1000 > message.getExpiration()) { + message.setExpiration(System.currentTimeMillis() + 1000); + } + + replicaInternalMessageProducer.sendForcingFlowControl(connectionContext, message); + } catch (Exception e) { + logger.error("Failed to process message {} with JMS message id: {}", message.getMessageId(), message.getJMSMessageID(), e); + throw e; + } + } + + private void messageDispatch(ConsumerId consumerId, ActiveMQDestination destination, String messageId) throws Exception { + MessageDispatchNotification mdn = new MessageDispatchNotification(); + mdn.setConsumerId(consumerId); + mdn.setDestination(destination); + mdn.setMessageId(new MessageId(messageId)); + broker.processDispatchNotification(mdn); + } + + private void removeScheduledMessageProperties(ActiveMQMessage message) throws IOException { + message.removeProperty(ScheduledMessage.AMQ_SCHEDULED_PERIOD); + message.removeProperty(ScheduledMessage.AMQ_SCHEDULED_REPEAT); + message.removeProperty(ScheduledMessage.AMQ_SCHEDULED_CRON); + message.removeProperty(ScheduledMessage.AMQ_SCHEDULED_DELAY); + } + + private void purgeQueue(ActiveMQDestination destination) throws Exception { + try { + Optional queue = broker.getDestinations(destination).stream() + .findFirst().map(DestinationExtractor::extractQueue); + if (queue.isPresent()) { + queue.get().purge(connectionContext); + } + } catch (Exception e) { + logger.error("Unable to replicate queue purge {}", destination, e); + throw e; + } + } + + private void beginTransaction(TransactionId xid) throws Exception { + try { + createTransactionMapIfNotExist(); + broker.beginTransaction(connectionContext, xid); + } catch (Exception e) { + logger.error("Unable to replicate begin transaction [{}]", xid, e); + throw e; + } + } + + private void prepareTransaction(TransactionId xid) throws Exception { + try { + if (xid.isXATransaction() && !isTransactionExisted(xid)) { + logger.warn("Skip processing transaction event - non-existing XA transaction [{}]", xid); + return; + } + createTransactionMapIfNotExist(); + broker.prepareTransaction(connectionContext, xid); + } catch (Exception e) { + logger.error("Unable to replicate prepare transaction [{}]", xid, e); + throw e; + } + } + + private void forgetTransaction(TransactionId xid) throws Exception { + try { + if (xid.isXATransaction() && !isTransactionExisted(xid)) { + logger.warn("Skip processing transaction event - non-existing XA transaction [{}]", xid); + return; + } + createTransactionMapIfNotExist(); + broker.forgetTransaction(connectionContext, xid); + } catch (Exception e) { + logger.error("Unable to replicate forget transaction [{}]", xid, e); + throw e; + } + } + + private void rollbackTransaction(TransactionId xid) throws Exception { + try { + if (xid.isXATransaction() && !isTransactionExisted(xid)) { + logger.warn("Skip processing transaction event - non-existing XA transaction [{}]", xid); + return; + } + createTransactionMapIfNotExist(); + broker.rollbackTransaction(connectionContext, xid); + } catch (Exception e) { + logger.error("Unable to replicate rollback transaction [{}]", xid, e); + throw e; + } + } + + private void commitTransaction(TransactionId xid, boolean onePhase) throws Exception { + try { + if (xid.isXATransaction() && !isTransactionExisted(xid)) { + logger.warn("Skip processing transaction event - non-existing XA transaction [{}]", xid); + return; + } + broker.commitTransaction(connectionContext, xid, onePhase); + } catch (Exception e) { + logger.error("Unable to replicate commit transaction [{}]", xid, e); + throw e; + } + } + + private void addDurableConsumer(ConsumerInfo consumerInfo, String clientId) throws Exception { + try { + consumerInfo.setPrefetchSize(0); + ConnectionContext context = connectionContext.copy(); + context.setClientId(clientId); + context.setConnection(new DummyConnection()); + DurableTopicSubscription durableTopicSubscription = (DurableTopicSubscription) broker.addConsumer(context, consumerInfo); + // We don't want to keep it active to be able to connect to it on the other side when needed + // but we want to have keepDurableSubsActive to be able to acknowledge + durableTopicSubscription.deactivate(true, 0); + } catch (Exception e) { + logger.error("Unable to replicate add durable consumer [{}]", consumerInfo, e); + throw e; + } + } + + private void removeDurableConsumer(ConsumerInfo consumerInfo) throws Exception { + try { + ConnectionContext context = broker.getDestinations(consumerInfo.getDestination()).stream() + .findFirst() + .map(Destination::getConsumers) + .stream().flatMap(Collection::stream) + .filter(v -> v.getConsumerInfo().getClientId().equals(consumerInfo.getClientId())) + .findFirst() + .map(Subscription::getContext) + .orElse(null); + if (context == null || !ReplicaSupport.REPLICATION_PLUGIN_USER_NAME.equals(context.getUserName())) { + // a real consumer had stolen the context before we got the message + return; + } + + broker.removeConsumer(context, consumerInfo); + } catch (Exception e) { + logger.error("Unable to replicate remove durable consumer [{}]", consumerInfo, e); + throw e; + } + } + + private void removeDurableConsumerSubscription(RemoveSubscriptionInfo subscriptionInfo) throws Exception { + try { + ConnectionContext context = connectionContext.copy(); + context.setClientId(subscriptionInfo.getClientId()); + broker.removeSubscription(context, subscriptionInfo); + } catch (Exception e) { + logger.error("Unable to replicate remove durable consumer subscription [{}]", subscriptionInfo, e); + throw e; + } + } + + private void messageAck(MessageAck ack, List messageIdsToAck, TransactionId transactionId) throws Exception { + ActiveMQDestination destination = ack.getDestination(); + MessageAck messageAck = new MessageAck(); + try { + if (!isDestinationExisted(destination)) { + logger.warn("Skip MESSAGE_ACK processing event due to non-existing destination [{}]", destination.getPhysicalName()); + return; + } + ConsumerInfo consumerInfo = null; + if (destination.isQueue()) { + consumerInfo = new ConsumerInfo(); + consumerInfo.setConsumerId(ack.getConsumerId()); + consumerInfo.setPrefetchSize(0); + consumerInfo.setDestination(destination); + broker.addConsumer(connectionContext, consumerInfo); + } + + List existingMessageIdsToAck = new ArrayList(); + for (String messageId : messageIdsToAck) { + try { + messageDispatch(ack.getConsumerId(), destination, messageId); + existingMessageIdsToAck.add(messageId); + } catch (JMSException e) { + if (isExceptionDueToNonExistingMessage(e)) { + logger.warn("Skip MESSAGE_ACK processing event due to non-existing message [{}]", messageId); + } else { + throw e; + } + } + } + + if (existingMessageIdsToAck.size() == 0) { + return; + } + + ack.copy(messageAck); + + messageAck.setMessageCount(existingMessageIdsToAck.size()); + messageAck.setFirstMessageId(new MessageId(existingMessageIdsToAck.get(0))); + messageAck.setLastMessageId(new MessageId(existingMessageIdsToAck.get(existingMessageIdsToAck.size() - 1))); + + if (messageAck.getTransactionId() == null || !messageAck.getTransactionId().isXATransaction()) { + messageAck.setTransactionId(transactionId); + } + + if (messageAck.isPoisonAck()) { + messageAck.setAckType(MessageAck.STANDARD_ACK_TYPE); + } + + ConsumerBrokerExchange consumerBrokerExchange = new ConsumerBrokerExchange(); + consumerBrokerExchange.setConnectionContext(connectionContext); + broker.acknowledge(consumerBrokerExchange, messageAck); + + if (consumerInfo != null) { + broker.removeConsumer(connectionContext, consumerInfo); + } + } catch (Exception e) { + logger.error("Unable to ack messages [{} <-> {}] for consumer {}", + ack.getFirstMessageId(), + ack.getLastMessageId(), + ack.getConsumerId(), e); + throw e; + } + } + + private void messageExpired(ActiveMQMessage message) { + try { + Destination destination = broker.getDestinations(message.getDestination()).stream() + .findFirst().map(DestinationExtractor::extractBaseDestination).orElseThrow(); + message.setRegionDestination(destination); + destination.messageExpired(connectionContext, null, new IndirectMessageReference(message)); + } catch (Exception e) { + logger.error("Unable to replicate message expired [{}]", message.getMessageId(), e); + throw e; + } + } + + private void failOver() throws Exception { + replicaBroker.updateBrokerState(ReplicaRole.ack_processed); + acknowledgeCallback.acknowledge(true); + replicaBroker.updateBrokerState(ReplicaRole.source); + replicaBroker.completeBeforeRoleChange(); + } + + private void createTransactionMapIfNotExist() { + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaCompactor.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaCompactor.java new file mode 100644 index 00000000000..34f4e5166c0 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaCompactor.java @@ -0,0 +1,442 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.MessageReferenceFilter; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.QueueMessageReference; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.util.JMSExceptionSupport; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.JMSException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Function; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class ReplicaCompactor { + private static final Logger logger = LoggerFactory.getLogger(ReplicaCompactor.class); + + private final Broker broker; + private final ReplicaReplicationQueueSupplier queueProvider; + private final PrefetchSubscription subscription; + private final int additionalMessagesLimit; + private final ReplicaStatistics replicaStatistics; + + private final Queue intermediateQueue; + + public ReplicaCompactor(Broker broker, ReplicaReplicationQueueSupplier queueProvider, PrefetchSubscription subscription, + int additionalMessagesLimit, ReplicaStatistics replicaStatistics) { + this.broker = broker; + this.queueProvider = queueProvider; + this.subscription = subscription; + this.additionalMessagesLimit = additionalMessagesLimit; + this.replicaStatistics = replicaStatistics; + + intermediateQueue = broker.getDestinations(queueProvider.getIntermediateQueue()).stream().findFirst() + .map(DestinationExtractor::extractQueue).orElseThrow(); + } + + List compactAndFilter(ConnectionContext connectionContext, List list, + boolean withAdditionalMessages) throws Exception { + List toProcess = list.stream() + .map(DeliveredMessageReference::new) + .collect(Collectors.toList()); + + int prefetchSize = subscription.getPrefetchSize(); + int maxPageSize = intermediateQueue.getMaxPageSize(); + int maxExpirePageSize = intermediateQueue.getMaxExpirePageSize(); + try { + if (withAdditionalMessages) { + subscription.setPrefetchSize(0); + intermediateQueue.setMaxPageSize(0); + intermediateQueue.setMaxExpirePageSize(0); + toProcess.addAll(getAdditionalMessages(connectionContext, list)); + } + + List processed = compactAndFilter0(connectionContext, toProcess); + + Set messageIds = list.stream().map(MessageReference::getMessageId).collect(Collectors.toSet()); + + return processed.stream() + .map(dmr -> dmr.messageReference) + .filter(mr -> messageIds.contains(mr.getMessageId())) + .collect(Collectors.toList()); + } finally { + subscription.setPrefetchSize(prefetchSize); + intermediateQueue.setMaxPageSize(maxPageSize); + intermediateQueue.setMaxExpirePageSize(maxExpirePageSize); + } + } + + private List getAdditionalMessages(ConnectionContext connectionContext, + List toProcess) throws Exception { + List dispatched = subscription.getDispatched(); + Set dispatchedMessageIds = dispatched.stream() + .map(MessageReference::getMessageId) + .map(MessageId::toString) + .collect(Collectors.toSet()); + + Set toProcessIds = toProcess.stream() + .map(MessageReference::getMessageId) + .map(MessageId::toString) + .collect(Collectors.toSet()); + + Set ignore = new HashSet<>(); + for (int i = 0; i < ReplicaSupport.INTERMEDIATE_QUEUE_PREFETCH_SIZE / additionalMessagesLimit + 1; i++) { + List acks = + intermediateQueue.getMatchingMessages(connectionContext, + new AckMessageReferenceFilter(toProcessIds, dispatchedMessageIds, ignore, dispatched), + additionalMessagesLimit); + if (acks.isEmpty()) { + return new ArrayList<>(); + } + + Set ackedMessageIds = getAckedMessageIds(acks); + List sends = intermediateQueue.getMatchingMessages(connectionContext, + new SendMessageReferenceFilter(toProcessIds, dispatchedMessageIds, ackedMessageIds), ackedMessageIds.size()); + if (sends.isEmpty()) { + acks.stream().map(MessageReference::getMessageId).map(MessageId::toString) + .forEach(ignore::add); + continue; + } + + return Stream.concat(acks.stream(), sends.stream().filter(mr -> !toProcessIds.contains(mr.getMessageId().toString()))) + .map(mr -> new DeliveredMessageReference(mr, false)) + .collect(Collectors.toList()); + } + return new ArrayList<>(); + } + + private List compactAndFilter0(ConnectionContext connectionContext, + List list) throws Exception { + List result = new ArrayList<>(list); + + List sendsAndAcksList = combineByDestination(list); + + List toDelete = compact(sendsAndAcksList); + + if (toDelete.isEmpty()) { + return result; + } + + acknowledge(connectionContext, toDelete); + + Set messageIds = toDelete.stream().map(dmid -> dmid.messageReference.getMessageId()).collect(Collectors.toSet()); + result.removeIf(reference -> messageIds.contains(reference.messageReference.getMessageId())); + + replicaStatistics.increaseTpsCounter(toDelete.size()); + + return result; + } + + private void acknowledge(ConnectionContext connectionContext, List list) throws Exception { + List notDelivered = list.stream() + .filter(dmr -> !dmr.delivered) + .map(DeliveredMessageReference::getReference) + .collect(Collectors.toList()); + if (!notDelivered.isEmpty()) { + intermediateQueue.dispatchNotification(subscription, notDelivered); + } + + TransactionId transactionId = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + boolean rollbackOnFail = false; + + try { + broker.beginTransaction(connectionContext, transactionId); + rollbackOnFail = true; + + ConsumerBrokerExchange consumerExchange = new ConsumerBrokerExchange(); + consumerExchange.setConnectionContext(connectionContext); + consumerExchange.setSubscription(subscription); + + for (DeliveredMessageReference dmr : list) { + MessageAck messageAck = new MessageAck(); + messageAck.setMessageID(dmr.messageReference.getMessageId()); + messageAck.setTransactionId(transactionId); + messageAck.setMessageCount(1); + messageAck.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + messageAck.setDestination(queueProvider.getIntermediateQueue()); + + broker.acknowledge(consumerExchange, messageAck); + } + + broker.commitTransaction(connectionContext, transactionId, true); + } catch (Exception e) { + logger.error("Failed to persist messages in the main replication queue", e); + if (rollbackOnFail) { + try { + broker.rollbackTransaction(connectionContext, transactionId); + } catch (Exception ex) { + logger.error("Could not rollback transaction", ex); + } + } + throw e; + } + } + + private static List combineByDestination(List list) throws Exception { + Map result = new HashMap<>(); + for (DeliveredMessageReference reference : list) { + ActiveMQMessage message = (ActiveMQMessage) reference.messageReference.getMessage(); + + ReplicaEventType eventType = + ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)); + if (eventType != ReplicaEventType.MESSAGE_SEND && eventType != ReplicaEventType.MESSAGE_ACK) { + continue; + } + + if (!message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY) + || message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY)) { + continue; + } + + SendsAndAcks sendsAndAcks = + result.computeIfAbsent(message.getStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY), + SendsAndAcks::new); + + if (eventType == ReplicaEventType.MESSAGE_SEND) { + List sends = sendsAndAcks.sendMap + .computeIfAbsent(message.getStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY), o -> new ArrayList<>()); + sends.add(new DeliveredMessageReference(message, reference.delivered)); + } + if (eventType == ReplicaEventType.MESSAGE_ACK) { + List messageIds = getAckMessageIds(message); + sendsAndAcks.acks.add(new Ack(messageIds, message, reference.delivered)); + } + } + + return new ArrayList<>(result.values()); + } + + private List compact(List sendsAndAcksList) throws IOException { + List result = new ArrayList<>(); + Set sendMessageIds = new HashSet<>(); + for (SendsAndAcks sendsAndAcks : sendsAndAcksList) { + for (Ack ack : sendsAndAcks.acks) { + List sends = new ArrayList<>(); + List sendIds = new ArrayList<>(); + for (String id : ack.messageIdsToAck) { + if (!sendsAndAcks.sendMap.containsKey(id)) { + continue; + } + List sendMessages = sendsAndAcks.sendMap.get(id); + DeliveredMessageReference message = null; + for (DeliveredMessageReference dmr : sendMessages) { + if (!sendMessageIds.contains(dmr.messageReference.getMessageId().toString())) { + message = dmr; + break; + } + } + if (message == null) { + continue; + } + sendIds.add(id); + sends.add(message); + sendMessageIds.add(message.messageReference.getMessageId().toString()); + } + if (sendIds.size() == 0) { + continue; + } + + if (ack.messageIdsToAck.size() == sendIds.size() && new HashSet<>(ack.messageIdsToAck).containsAll(sendIds)) { + result.addAll(sends); + result.add(ack); + } + } + } + return result; + } + + private Set getAckedMessageIds(List ackMessages) throws IOException { + Set messageIds = new HashSet<>(); + for (QueueMessageReference messageReference : ackMessages) { + ActiveMQMessage message = (ActiveMQMessage) messageReference.getMessage(); + + messageIds.addAll(getAckMessageIds(message)); + } + return messageIds; + } + + @SuppressWarnings("unchecked") + private static List getAckMessageIds(ActiveMQMessage message) throws IOException { + return (List) message.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY); + } + + private static class DeliveredMessageReference { + final MessageReference messageReference; + final boolean delivered; + + public DeliveredMessageReference(MessageReference messageReference) { + this(messageReference, true); + } + + public DeliveredMessageReference(MessageReference messageReference, boolean delivered) { + this.messageReference = messageReference; + this.delivered = delivered; + } + + public QueueMessageReference getReference() { + if (messageReference instanceof QueueMessageReference) { + return (QueueMessageReference) messageReference; + } + return new IndirectMessageReference(messageReference.getMessage()); + } + } + + private static class SendsAndAcks { + final String destination; + final Map> sendMap = new LinkedHashMap<>(); + final List acks = new ArrayList<>(); + + private SendsAndAcks(String destination) { + this.destination = destination; + } + } + + private static class Ack extends DeliveredMessageReference { + final List messageIdsToAck; + final ActiveMQMessage message; + + public Ack(List messageIdsToAck, ActiveMQMessage message, boolean needsDelivery) { + super(message, needsDelivery); + this.messageIdsToAck = messageIdsToAck; + this.message = message; + } + } + + static class AckMessageReferenceFilter extends InternalMessageReferenceFilter { + + private final Map existingSendsAndAcks; + + public AckMessageReferenceFilter(Set toProcess, Set dispatchedMessageIds, + Set ignore, List dispatched) throws Exception { + super(toProcess, dispatchedMessageIds, ignore, ReplicaEventType.MESSAGE_ACK); + List list = dispatched.stream() + .filter(mr -> !toProcess.contains(mr.getMessageId().toString())) + .map(DeliveredMessageReference::new) + .collect(Collectors.toList()); + existingSendsAndAcks = combineByDestination(list).stream().collect(Collectors.toMap(o -> o.destination, Function.identity())); + } + + @Override + public boolean evaluate(ActiveMQMessage message) throws JMSException { + if (!message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY) + || message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY)) { + return false; + } + + String destination = message.getStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY); + SendsAndAcks sendsAndAcks = existingSendsAndAcks.get(destination); + if (sendsAndAcks == null) { + return true; + } + + List messageIds; + try { + messageIds = (List) message.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY); + } catch (IOException e) { + throw JMSExceptionSupport.create(e); + } + + return !sendsAndAcks.sendMap.keySet().containsAll(messageIds); + } + } + + static class SendMessageReferenceFilter extends InternalMessageReferenceFilter { + + private final Set ackedMessageIds; + + public SendMessageReferenceFilter(Set toProcess, Set dispatchedMessageIds, + Set ackedMessageIds) { + super(toProcess, dispatchedMessageIds, new HashSet<>(), ReplicaEventType.MESSAGE_SEND); + this.ackedMessageIds = ackedMessageIds; + } + + @Override + public boolean evaluate(ActiveMQMessage message) throws JMSException { + if (!message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY) + || message.getBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY)) { + return false; + } + + String messageId = message.getStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY); + return ackedMessageIds.contains(messageId); + } + } + + private static abstract class InternalMessageReferenceFilter implements MessageReferenceFilter { + + private final Set toProcess; + private final Set dispatchedMessageIds; + private final Set ignore; + private final ReplicaEventType eventType; + + public InternalMessageReferenceFilter(Set toProcess, Set dispatchedMessageIds, + Set ignore, ReplicaEventType eventType) { + this.toProcess = toProcess; + this.dispatchedMessageIds = dispatchedMessageIds; + this.ignore = ignore; + this.eventType = eventType; + } + + @Override + public boolean evaluate(ConnectionContext context, MessageReference messageReference) throws JMSException { + String messageId = messageReference.getMessageId().toString(); + if (ignore.contains(messageId)) { + return false; + } + + if (dispatchedMessageIds.contains(messageId) && !toProcess.contains(messageId)) { + return false; + } + ActiveMQMessage message = (ActiveMQMessage) messageReference.getMessage(); + + ReplicaEventType eventType = + ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)); + + if (eventType != this.eventType) { + return false; + } + return evaluate(message); + } + + public abstract boolean evaluate(ActiveMQMessage message) throws JMSException; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationFilter.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationFilter.java new file mode 100644 index 00000000000..aef8f2c1211 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationFilter.java @@ -0,0 +1,79 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.broker.region.virtual.CompositeDestinationFilter; +import org.apache.activemq.command.Message; +import org.apache.activemq.command.TransactionId; + +public class ReplicaDestinationFilter extends DestinationFilter { + private final boolean nextIsComposite; + private final ReplicaSourceBroker sourceBroker; + private final ReplicaRoleManagementBroker roleManagementBroker; + + public ReplicaDestinationFilter(Destination next, ReplicaSourceBroker sourceBroker, ReplicaRoleManagementBroker roleManagementBroker) { + super(next); + this.nextIsComposite = this.next != null && this.next instanceof CompositeDestinationFilter; + this.sourceBroker = sourceBroker; + this.roleManagementBroker = roleManagementBroker; + } + + @Override + public void send(ProducerBrokerExchange producerExchange, Message messageSend) throws Exception { + if(ReplicaRole.source == roleManagementBroker.getRole()) { + super.send(producerExchange, messageSend); + if(!nextIsComposite) { + // don't replicate composite destination + replicateSend(producerExchange, messageSend); + } + } else { + if(nextIsComposite) { + // we jump over CompositeDestinationFilter as we don't want to fan out composite destinations on the replica side + ((CompositeDestinationFilter) getNext()).getNext().send(producerExchange, messageSend); + } else { + super.send(producerExchange, messageSend); + } + } + } + + @Override + public boolean canGC() { + if (ReplicaRole.source == roleManagementBroker.getRole()) { + return super.canGC(); + } + return false; + } + + private void replicateSend(ProducerBrokerExchange producerExchange, Message messageSend) throws Exception { + final ConnectionContext connectionContext = producerExchange.getConnectionContext(); + if (!sourceBroker.needToReplicateSend(connectionContext, messageSend)) { + return; + } + + TransactionId transactionId = null; + if (messageSend.getTransactionId() != null && !messageSend.getTransactionId().isXATransaction()) { + transactionId = messageSend.getTransactionId(); + } + + sourceBroker.replicateSend(connectionContext, messageSend, transactionId); + } + +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationInterceptor.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationInterceptor.java new file mode 100644 index 00000000000..5d34e2de645 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaDestinationInterceptor.java @@ -0,0 +1,47 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.command.ActiveMQDestination; + +public class ReplicaDestinationInterceptor implements DestinationInterceptor { + + private final ReplicaSourceBroker sourceBroker; + private final ReplicaRoleManagementBroker roleManagementBroker; + + public ReplicaDestinationInterceptor(ReplicaSourceBroker sourceBroker, ReplicaRoleManagementBroker roleManagementBroker) { + this.sourceBroker = sourceBroker; + this.roleManagementBroker = roleManagementBroker; + } + + @Override + public Destination intercept(Destination destination) { + return new ReplicaDestinationFilter(destination, sourceBroker, roleManagementBroker); + } + + @Override + public void remove(Destination destination) { + } + + @Override + public void create(Broker broker, ConnectionContext context, ActiveMQDestination destination) throws Exception { + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEvent.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEvent.java new file mode 100644 index 00000000000..48209b5c696 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEvent.java @@ -0,0 +1,97 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.util.ByteSequence; + +import java.util.HashMap; +import java.util.Map; + +import static java.text.MessageFormat.format; +import static java.util.Objects.requireNonNull; + +public class ReplicaEvent { + + private TransactionId transactionId; + private ReplicaEventType eventType; + private byte[] eventData; + private Map replicationProperties = new HashMap<>(); + + private Integer version; + private Long timestamp; + + ReplicaEvent setTransactionId(TransactionId transactionId) { + this.transactionId = transactionId; + return this; + } + + public ReplicaEvent setEventType(final ReplicaEventType eventType) { + this.eventType = requireNonNull(eventType); + return this; + } + + public ReplicaEvent setEventData(final byte[] eventData) { + this.eventData = requireNonNull(eventData); + return this; + } + + ReplicaEvent setReplicationProperty(String propertyKey, Object propertyValue) { + requireNonNull(propertyKey); + requireNonNull(propertyValue); + if (replicationProperties.containsKey(propertyKey)) { + throw new IllegalStateException(format("replication property ''{0}'' already has value ''{1}''", propertyKey, propertyValue)); + } + replicationProperties.put(propertyKey, propertyValue); + return this; + } + + ReplicaEvent setVersion(int version) { + this.version = version; + return this; + } + + ReplicaEvent setTimestamp(long timestamp) { + this.timestamp = timestamp; + return this; + } + + + TransactionId getTransactionId() { + return transactionId; + } + + public ByteSequence getEventData() { + return new ByteSequence(eventData); + } + + public ReplicaEventType getEventType() { + return eventType; + } + + public Map getReplicationProperties() { + return replicationProperties; + } + + public Integer getVersion() { + return version; + } + + public Long getTimestamp() { + return timestamp; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventRetrier.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventRetrier.java new file mode 100644 index 00000000000..4511162bde2 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventRetrier.java @@ -0,0 +1,62 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.BrokerStoppedException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.Callable; +import java.util.concurrent.atomic.AtomicBoolean; + +public class ReplicaEventRetrier { + + private final Logger logger = LoggerFactory.getLogger(ReplicaEventRetrier.class); + + private final int INITIAL_SLEEP_RETRY_INTERVAL_MS = 10; + private final int MAX_SLEEP_RETRY_INTERVAL_MS = 10000; + + private final Callable task; + private final AtomicBoolean running = new AtomicBoolean(true); + + public ReplicaEventRetrier(Callable task) { + this.task = task; + } + + public void process() throws InterruptedException { + long attemptNumber = 0; + while (running.get()) { + try { + task.call(); + return; + } catch (BrokerStoppedException | InterruptedException bse) { + throw bse; + } catch (Exception e) { + logger.error("Caught exception while processing a replication event.", e); + int sleepInterval = Math.min((int) (INITIAL_SLEEP_RETRY_INTERVAL_MS * Math.pow(2.0, attemptNumber)), + MAX_SLEEP_RETRY_INTERVAL_MS); + attemptNumber++; + logger.info("Retry attempt number {}. Sleeping for {} ms.", attemptNumber, sleepInterval); + Thread.sleep(sleepInterval); + } + } + } + + public void stop() { + running.set(false); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventSerializer.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventSerializer.java new file mode 100644 index 00000000000..6411ed856e2 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventSerializer.java @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.command.DataStructure; +import org.apache.activemq.command.Message; +import org.apache.activemq.openwire.OpenWireFormatFactory; +import org.apache.activemq.util.ByteSequence; +import org.apache.activemq.util.ByteSequenceData; +import org.apache.activemq.util.DataByteArrayInputStream; +import org.apache.activemq.util.DataByteArrayOutputStream; +import org.apache.activemq.util.IOExceptionSupport; +import org.apache.activemq.wireformat.WireFormat; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +public class ReplicaEventSerializer { + + private final WireFormat wireFormat = new OpenWireFormatFactory().createWireFormat(); + + public byte[] serializeReplicationData(DataStructure object) throws IOException { + try { + ByteSequence packet = wireFormat.marshal(object); + return ByteSequenceData.toByteArray(packet); + } catch (IOException e) { + throw IOExceptionSupport.create("Failed to serialize data: " + object.toString() + " in container: " + e, e); + } + } + + public byte[] serializeMessageData(Message message) throws IOException { + try { + ByteSequence packet = wireFormat.marshal(message); + return ByteSequenceData.toByteArray(packet); + } catch (IOException e) { + throw IOExceptionSupport.create("Failed to serialize message: " + message.getMessageId() + " in container: " + e, e); + } + } + + Object deserializeMessageData(ByteSequence sequence) throws IOException { + return wireFormat.unmarshal(sequence); + } + + byte[] serializeListOfObjects(List list) throws IOException { + List listOfByteArrays = new ArrayList<>(); + for (DataStructure dataStructure : list) { + listOfByteArrays.add(serializeReplicationData(dataStructure)); + } + + int listSize = listOfByteArrays.stream().map(a -> a.length).reduce(0, Integer::sum); + + DataByteArrayOutputStream dbaos = new DataByteArrayOutputStream(4 + 2 * listOfByteArrays.size() + listSize); + + dbaos.writeInt(listOfByteArrays.size()); + for (byte[] b : listOfByteArrays) { + dbaos.writeInt(b.length); + dbaos.write(b); + } + + return ByteSequenceData.toByteArray(dbaos.toByteSequence()); + } + + List deserializeListOfObjects(byte[] bytes) throws IOException { + List result = new ArrayList<>(); + + DataByteArrayInputStream dbais = new DataByteArrayInputStream(bytes); + + int listSize = dbais.readInt(); + for (int i = 0; i < listSize; i++) { + int size = dbais.readInt(); + + byte[] b = new byte[size]; + dbais.readFully(b); + result.add(deserializeMessageData(new ByteSequence(b))); + } + + return result; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventType.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventType.java new file mode 100644 index 00000000000..5039f39fb2c --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaEventType.java @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +public enum ReplicaEventType { + DESTINATION_UPSERT, + DESTINATION_DELETE, + MESSAGE_SEND, + MESSAGE_ACK, + QUEUE_PURGED, + TRANSACTION_BEGIN, + TRANSACTION_PREPARE, + TRANSACTION_ROLLBACK, + TRANSACTION_COMMIT, + TRANSACTION_FORGET, + ADD_DURABLE_CONSUMER, + REMOVE_DURABLE_CONSUMER, + MESSAGE_EXPIRED, + BATCH, + REMOVE_DURABLE_CONSUMER_SUBSCRIPTION, + FAIL_OVER, + HEART_BEAT, + ; + + public static final String EVENT_TYPE_PROPERTY = "ActiveMQReplicationEventType"; +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaInternalMessageProducer.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaInternalMessageProducer.java new file mode 100644 index 00000000000..fd1632590c9 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaInternalMessageProducer.java @@ -0,0 +1,50 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ProducerInfo; +import org.apache.activemq.state.ProducerState; + +import static java.util.Objects.requireNonNull; + +public class ReplicaInternalMessageProducer { + + private final Broker broker; + + ReplicaInternalMessageProducer(Broker broker) { + this.broker = requireNonNull(broker); + } + + public void sendForcingFlowControl(ConnectionContext connectionContext, ActiveMQMessage eventMessage) throws Exception { + ProducerBrokerExchange producerExchange = new ProducerBrokerExchange(); + producerExchange.setConnectionContext(connectionContext); + producerExchange.setMutable(true); + producerExchange.setProducerState(new ProducerState(new ProducerInfo())); + + boolean originalFlowControl = connectionContext.isProducerFlowControl(); + try { + connectionContext.setProducerFlowControl(true); + broker.send(producerExchange, eventMessage); + } finally { + connectionContext.setProducerFlowControl(originalFlowControl); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationFilter.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationFilter.java new file mode 100644 index 00000000000..3ddde277cc8 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationFilter.java @@ -0,0 +1,43 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.replica; + +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.command.Message; + +public class ReplicaMirroredDestinationFilter extends DestinationFilter { + + private final ReplicaRoleManagementBroker roleManagementBroker; + + public ReplicaMirroredDestinationFilter(Destination next, ReplicaRoleManagementBroker roleManagementBroker) { + super(next); + this.roleManagementBroker = roleManagementBroker; + } + + @Override + public void send(ProducerBrokerExchange producerExchange, Message messageSend) throws Exception { + if (ReplicaSupport.isReplicationDestination(messageSend.getDestination()) || + (roleManagementBroker.getRole() != ReplicaRole.source && getNext() instanceof DestinationFilter)) { + ((DestinationFilter) getNext()).getNext().send(producerExchange, messageSend); + } else { + super.send(producerExchange, messageSend); + } + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationInterceptor.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationInterceptor.java new file mode 100644 index 00000000000..8255b5f087b --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaMirroredDestinationInterceptor.java @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + *

+ * http://www.apache.org/licenses/LICENSE-2.0 + *

+ * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.command.ActiveMQDestination; + +public class ReplicaMirroredDestinationInterceptor implements DestinationInterceptor { + + private final ReplicaRoleManagementBroker roleManagementBroker; + + public ReplicaMirroredDestinationInterceptor(ReplicaRoleManagementBroker roleManagementBroker) { + this.roleManagementBroker = roleManagementBroker; + } + + @Override + public Destination intercept(Destination destination) { + return new ReplicaMirroredDestinationFilter(destination, roleManagementBroker); + } + + @Override + public void remove(Destination destination) { + } + + @Override + public void create(Broker broker, ConnectionContext context, ActiveMQDestination destination) throws Exception { + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPlugin.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPlugin.java new file mode 100644 index 00000000000..97410e032e7 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPlugin.java @@ -0,0 +1,244 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerPluginSupport; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.AnnotatedMBean; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.broker.region.policy.PolicyEntry; +import org.apache.activemq.broker.region.policy.PolicyMap; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.replica.jmx.ReplicationJmxHelper; +import org.apache.activemq.replica.jmx.ReplicationView; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.net.URI; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +import static java.util.Objects.requireNonNull; + +/** + * A Broker plugin to replicate core messaging events from one broker to another. + * + * @org.apache.xbean.XBean element="replicaPlugin" + */ +public class ReplicaPlugin extends BrokerPluginSupport { + + private final Logger logger = LoggerFactory.getLogger(ReplicaPlugin.class); + + protected ReplicaRole role = ReplicaRole.source; + + protected ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + + private ReplicationView replicationView; + + private ReplicaRoleManagementBroker replicaRoleManagementBroker; + + public ReplicaPlugin() { + super(); + } + + @Override + public Broker installPlugin(final Broker broker) throws Exception { + if (role != ReplicaRole.source && role != ReplicaRole.replica) { + throw new IllegalArgumentException(String.format("Unsupported role [%s]", role.name())); + } + + logger.info("{} installed, running as {}", ReplicaPlugin.class.getName(), role); + + ReplicaStatistics replicaStatistics = new ReplicaStatistics(); + + BrokerService brokerService = broker.getBrokerService(); + if (brokerService.isUseJmx()) { + replicationView = new ReplicationView(this, replicaStatistics); + AnnotatedMBean.registerMBean(brokerService.getManagementContext(), replicationView, ReplicationJmxHelper.createJmxName(brokerService)); + } + + List policyEntries = new ArrayList<>(); + for (String queue : ReplicaSupport.REPLICATION_QUEUE_NAMES) { + PolicyEntry newPolicy = getPolicyEntry(new ActiveMQQueue(queue)); + newPolicy.setMaxPageSize(ReplicaSupport.INTERMEDIATE_QUEUE_PREFETCH_SIZE); + policyEntries.add(newPolicy); + } + for (String topic : ReplicaSupport.REPLICATION_TOPIC_NAMES) { + policyEntries.add(getPolicyEntry(new ActiveMQTopic(topic))); + } + if (brokerService.getDestinationPolicy() == null) { + brokerService.setDestinationPolicy(new PolicyMap()); + } + brokerService.getDestinationPolicy().setPolicyEntries(policyEntries); + + RegionBroker regionBroker = (RegionBroker) broker.getAdaptor(RegionBroker.class); + CompositeDestinationInterceptor compositeInterceptor = (CompositeDestinationInterceptor) regionBroker.getDestinationInterceptor(); + DestinationInterceptor[] interceptors = compositeInterceptor.getInterceptors(); + interceptors = Arrays.copyOf(interceptors, interceptors.length + 1); + interceptors[interceptors.length - 1] = new ReplicaAdvisorySuppressor(); + compositeInterceptor.setInterceptors(interceptors); + + replicaRoleManagementBroker = new ReplicaRoleManagementBroker(broker, replicaPolicy, role, replicaStatistics); + + return new ReplicaAuthorizationBroker(replicaRoleManagementBroker); + } + + private PolicyEntry getPolicyEntry(ActiveMQDestination destination) { + PolicyEntry newPolicy = new PolicyEntry(); + newPolicy.setGcInactiveDestinations(false); + newPolicy.setDestination(destination); + return newPolicy; + } + + public ReplicaPlugin setRole(ReplicaRole role) { + this.role = requireNonNull(role); + return this; + } + + public ReplicaPlugin connectedTo(URI uri) { + this.setOtherBrokerUri(requireNonNull(uri).toString()); + return this; + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setRole(String role) { + this.role = Arrays.stream(ReplicaRole.values()) + .filter(roleValue -> roleValue.name().equalsIgnoreCase(role)) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException(role + " is not a known " + ReplicaRole.class.getSimpleName())); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setOtherBrokerUri(String uri) { + replicaPolicy.setOtherBrokerUri(uri); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setTransportConnectorUri(String uri) { + replicaPolicy.setTransportConnectorUri(URI.create(uri)); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setUserName(String userName) { + replicaPolicy.setUserName(userName); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setPassword(String password) { + replicaPolicy.setPassword(password); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setSourceSendPeriod(int period) { + replicaPolicy.setSourceSendPeriod(period); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setCompactorAdditionalMessagesLimit(int limit) { + replicaPolicy.setCompactorAdditionalMessagesLimit(limit); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setMaxBatchLength(int length) { + replicaPolicy.setMaxBatchLength(length); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setMaxBatchSize(int size) { + replicaPolicy.setMaxBatchSize(size); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setReplicaAckPeriod(int period) { + replicaPolicy.setReplicaAckPeriod(period); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setReplicaMaxAckBatchSize(int size) { + replicaPolicy.setReplicaMaxAckBatchSize(size); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setControlWebConsoleAccess(boolean controlWebConsoleAccess) { + replicaPolicy.setControlWebConsoleAccess(controlWebConsoleAccess); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setHeartBeatPeriod(int heartBeatPeriod) { + replicaPolicy.setHeartBeatPeriod(heartBeatPeriod); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setSourceReplicationFlowControl(boolean sourceReplicationFlowControl) { + replicaPolicy.setSourceReplicationFlowControl(sourceReplicationFlowControl); + } + + /** + * @org.apache.xbean.Property propertyEditor="com.sun.beans.editors.StringEditor" + */ + public void setReplicaReplicationFlowControl(boolean replicaReplicationFlowControl) { + replicaPolicy.setReplicaReplicationFlowControl(replicaReplicationFlowControl); + } + + public ReplicaRole getRole() { + return replicaRoleManagementBroker.getRole().getExternalRole(); + } + + public void setReplicaRole(ReplicaRole role, boolean force) throws Exception { + logger.debug("Called switch role for broker. Params: [{}], [{}]", role.name(), force); + + if (role != ReplicaRole.replica && role != ReplicaRole.source) { + throw new RuntimeException(String.format("Can't switch role from [%s] to [%s]", this.role.name(), role.name())); + } + + replicaRoleManagementBroker.switchRole(role, force); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPolicy.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPolicy.java new file mode 100644 index 00000000000..2d9802da281 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaPolicy.java @@ -0,0 +1,163 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; + +import java.net.URI; +import java.util.Objects; + + +public class ReplicaPolicy { + + private final ActiveMQConnectionFactory otherBrokerConnectionFactory = new ActiveMQConnectionFactory(); + private URI transportConnectorUri = null; + + private int sourceSendPeriod = 5_000; + private int compactorAdditionalMessagesLimit = 10_000; + private int maxBatchLength = 500; + private int maxBatchSize = 5_000_000; + private int replicaAckPeriod = 5_000; + private int replicaMaxAckBatchSize = 100; + private boolean controlWebConsoleAccess = true; + + private int heartBeatPeriod = 60_000; + + private boolean sourceReplicationFlowControl = true; + private boolean replicaReplicationFlowControl = true; + + public URI getTransportConnectorUri() { + return Objects.requireNonNull(transportConnectorUri, "Need replication transport connection URI for this broker"); + } + + public void setTransportConnectorUri(URI uri) { + transportConnectorUri = uri; + } + + public ActiveMQConnectionFactory getOtherBrokerConnectionFactory() { + Objects.requireNonNull(otherBrokerConnectionFactory, "Need connection details of replica source for this broker"); + Objects.requireNonNull(otherBrokerConnectionFactory.getBrokerURL(), "Need connection URI of replica source for this broker"); + validateUser(otherBrokerConnectionFactory); + return otherBrokerConnectionFactory; + } + + public void setOtherBrokerUri(String uri) { + otherBrokerConnectionFactory.setBrokerURL(uri); // once to validate + otherBrokerConnectionFactory.setBrokerURL( + uri.toLowerCase().startsWith("failover:(") + ? uri + : "failover:("+ uri +")" + ); + } + + public void setUserName(String userName) { + otherBrokerConnectionFactory.setUserName(userName); + } + + public void setPassword(String password) { + otherBrokerConnectionFactory.setPassword(password); + } + + public int getSourceSendPeriod() { + return sourceSendPeriod; + } + + public void setSourceSendPeriod(int period) { + sourceSendPeriod = period; + } + + public int getCompactorAdditionalMessagesLimit() { + return compactorAdditionalMessagesLimit; + } + + public void setCompactorAdditionalMessagesLimit(int limit) { + compactorAdditionalMessagesLimit = limit; + } + + public int getMaxBatchLength() { + return maxBatchLength; + } + + public void setMaxBatchLength(int maxBatchLength) { + this.maxBatchLength = maxBatchLength; + } + + public int getMaxBatchSize() { + return maxBatchSize; + } + + public void setMaxBatchSize(int maxBatchSize) { + this.maxBatchSize = maxBatchSize; + } + + public int getReplicaAckPeriod() { + return replicaAckPeriod; + } + + public void setReplicaAckPeriod(int period) { + replicaAckPeriod = period; + } + + public int getReplicaMaxAckBatchSize() { + return replicaMaxAckBatchSize; + } + + public void setReplicaMaxAckBatchSize(int replicaMaxAckBatchSize) { + this.replicaMaxAckBatchSize = replicaMaxAckBatchSize; + } + + public boolean isControlWebConsoleAccess() { + return controlWebConsoleAccess; + } + + public void setControlWebConsoleAccess(boolean controlWebConsoleAccess) { + this.controlWebConsoleAccess = controlWebConsoleAccess; + } + + public int getHeartBeatPeriod() { + return heartBeatPeriod; + } + + public void setHeartBeatPeriod(int heartBeatPeriod) { + this.heartBeatPeriod = heartBeatPeriod; + } + + public boolean isSourceReplicationFlowControl() { + return sourceReplicationFlowControl; + } + + public void setSourceReplicationFlowControl(boolean enableSourceReplicationFlowControl) { + this.sourceReplicationFlowControl = enableSourceReplicationFlowControl; + } + + public boolean isReplicaReplicationFlowControl() { + return replicaReplicationFlowControl; + } + + public void setReplicaReplicationFlowControl(boolean enableReplicaReplicationFlowControl) { + this.replicaReplicationFlowControl = enableReplicaReplicationFlowControl; + } + + private void validateUser(ActiveMQConnectionFactory replicaSourceConnectionFactory) { + if (replicaSourceConnectionFactory.getUserName() != null) { + Objects.requireNonNull(replicaSourceConnectionFactory.getPassword(), "Both userName and password or none of them should be configured for replica broker"); + } + if (replicaSourceConnectionFactory.getPassword() != null) { + Objects.requireNonNull(replicaSourceConnectionFactory.getUserName(), "Both userName and password or none of them should be configured for replica broker"); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplier.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplier.java new file mode 100644 index 00000000000..8cb5e2bf985 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplier.java @@ -0,0 +1,200 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Optional; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +import static java.util.Objects.requireNonNull; + +public class ReplicaReplicationQueueSupplier { + + private final Logger logger = LoggerFactory.getLogger(ReplicaSourceBroker.class); + private final CountDownLatch initializationLatch = new CountDownLatch(1); + private final CountDownLatch sequenceInitializationLatch = new CountDownLatch(1); + private final CountDownLatch roleInitializationLatch = new CountDownLatch(1); + private ActiveMQQueue mainReplicationQueue = null; // memoized + private ActiveMQQueue intermediateReplicationQueue = null; // memoized + private ActiveMQQueue sequenceQueue = null; // memoized + private ActiveMQQueue roleQueue = null; // memoized + private ActiveMQTopic roleAdvisoryTopic = null; // memoized + private final Broker broker; + + public ReplicaReplicationQueueSupplier(final Broker broker) { + this.broker = requireNonNull(broker); + } + + public ActiveMQQueue getMainQueue() { + try { + if (initializationLatch.await(1L, TimeUnit.MINUTES)) { + return requireNonNull(mainReplicationQueue); + } + } catch (InterruptedException e) { + throw new ActiveMQReplicaException("Interrupted while waiting for main replication queue initialization", e); + } + throw new ActiveMQReplicaException("Timed out waiting for main replication queue initialization"); + } + + public ActiveMQQueue getIntermediateQueue() { + try { + if (initializationLatch.await(1L, TimeUnit.MINUTES)) { + return requireNonNull(intermediateReplicationQueue); + } + } catch (InterruptedException e) { + throw new ActiveMQReplicaException("Interrupted while waiting for intermediate replication queue initialization", e); + } + throw new ActiveMQReplicaException("Timed out waiting for intermediate replication queue initialization"); + } + + public ActiveMQQueue getSequenceQueue() { + try { + if (sequenceInitializationLatch.await(1L, TimeUnit.MINUTES)) { + return requireNonNull(sequenceQueue); + } + } catch (InterruptedException e) { + throw new ActiveMQReplicaException("Interrupted while waiting for replication sequence queue initialization", e); + } + throw new ActiveMQReplicaException("Timed out waiting for replication sequence queue initialization"); + } + + public ActiveMQQueue getRoleQueue() { + try { + if (roleInitializationLatch.await(1L, TimeUnit.MINUTES)) { + return requireNonNull(roleQueue); + } + } catch (InterruptedException e) { + throw new ActiveMQReplicaException("Interrupted while waiting for role queue initialization", e); + } + throw new ActiveMQReplicaException("Timed out waiting for role queue initialization"); + } + + public ActiveMQTopic getRoleAdvisoryTopic() { + try { + if (roleInitializationLatch.await(1L, TimeUnit.MINUTES)) { + return requireNonNull(roleAdvisoryTopic); + } + } catch (InterruptedException e) { + throw new ActiveMQReplicaException("Interrupted while waiting for role queue initialization", e); + } + throw new ActiveMQReplicaException("Timed out waiting for role queue initialization"); + } + + public void initialize() { + try { + mainReplicationQueue = getOrCreateMainReplicationQueue(); + intermediateReplicationQueue = getOrCreateIntermediateReplicationQueue(); + } catch (Exception e) { + logger.error("Could not obtain replication queues", e); + throw new ActiveMQReplicaException("Failed to get or create replication queues"); + } + initializationLatch.countDown(); + } + + public void initializeSequenceQueue() { + try { + sequenceQueue = getOrCreateSequenceQueue(); + } catch (Exception e) { + logger.error("Could not obtain replication sequence queue", e); + throw new ActiveMQReplicaException("Failed to get or create replication sequence queue"); + } + sequenceInitializationLatch.countDown(); + + } + + public void initializeRoleQueueAndTopic() { + try { + roleQueue = getOrCreateRoleQueue(); + roleAdvisoryTopic = getOrCreateRoleAdvisoryTopic(); + } catch (Exception e) { + logger.error("Could not obtain role queue", e); + throw new ActiveMQReplicaException("Failed to get or create role queue"); + } + roleInitializationLatch.countDown(); + + } + + private ActiveMQQueue getOrCreateMainReplicationQueue() throws Exception { + return getOrCreateQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + } + + private ActiveMQQueue getOrCreateIntermediateReplicationQueue() throws Exception { + return getOrCreateQueue(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + } + + private ActiveMQQueue getOrCreateSequenceQueue() throws Exception { + return getOrCreateQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + } + + private ActiveMQQueue getOrCreateRoleQueue() throws Exception { + return getOrCreateQueue(ReplicaSupport.REPLICATION_ROLE_QUEUE_NAME); + } + + private ActiveMQTopic getOrCreateRoleAdvisoryTopic() throws Exception { + return getOrCreateTopic(ReplicaSupport.REPLICATION_ROLE_ADVISORY_TOPIC_NAME); + } + + private ActiveMQQueue getOrCreateQueue(String replicationQueueName) throws Exception { + Optional existingReplicationQueue = broker.getDurableDestinations() + .stream() + .filter(ActiveMQDestination::isQueue) + .filter(d -> replicationQueueName.equals(d.getPhysicalName())) + .findFirst(); + if (existingReplicationQueue.isPresent()) { + logger.debug("Existing replication queue {}", existingReplicationQueue.get().getPhysicalName()); + return new ActiveMQQueue(existingReplicationQueue.get().getPhysicalName()); + } else { + ActiveMQQueue newReplicationQueue = new ActiveMQQueue(replicationQueueName); + broker.getBrokerService().getBroker().addDestination( + broker.getAdminConnectionContext(), + newReplicationQueue, + false + ); + logger.debug("Created replication queue {}", newReplicationQueue.getPhysicalName()); + return newReplicationQueue; + } + } + + private ActiveMQTopic getOrCreateTopic(String replicationQueueName) throws Exception { + Optional existingReplicationQueue = broker.getDurableDestinations() + .stream() + .filter(ActiveMQDestination::isTopic) + .filter(d -> replicationQueueName.equals(d.getPhysicalName())) + .findFirst(); + if (existingReplicationQueue.isPresent()) { + logger.debug("Existing replication topic {}", existingReplicationQueue.get().getPhysicalName()); + return new ActiveMQTopic(existingReplicationQueue.get().getPhysicalName()); + } else { + ActiveMQTopic newReplicationQueue = new ActiveMQTopic(replicationQueueName); + broker.getBrokerService().getBroker().addDestination( + broker.getAdminConnectionContext(), + newReplicationQueue, + false + ); + logger.debug("Created replication topic {}", newReplicationQueue.getPhysicalName()); + return newReplicationQueue; + } + } + +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRole.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRole.java new file mode 100644 index 00000000000..964e0e39288 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRole.java @@ -0,0 +1,40 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +public enum ReplicaRole { + source, + replica, + await_ack(source), + ack_processed(replica), + ; + + private final ReplicaRole externalRole; + + ReplicaRole() { + externalRole = this; + } + + ReplicaRole(ReplicaRole role) { + externalRole = role; + } + + public ReplicaRole getExternalRole() { + return externalRole; + } +} + diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagement.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagement.java new file mode 100644 index 00000000000..adb2dcea093 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagement.java @@ -0,0 +1,31 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.TransactionId; + +public interface ReplicaRoleManagement { + + void updateBrokerState(ConnectionContext connectionContext, TransactionId tid, ReplicaRole role) throws Exception; + + void stopAllConnections(); + + void startAllConnections() throws Exception; + + void onStopSuccess() throws Exception; +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagementBroker.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagementBroker.java new file mode 100644 index 00000000000..35669513b62 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaRoleManagementBroker.java @@ -0,0 +1,271 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.Service; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.MutableBrokerFilter; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.broker.region.virtual.MirroredQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.ProducerId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.storage.ReplicaRoleStorage; +import org.apache.activemq.util.IdGenerator; +import org.apache.activemq.util.LongSequenceGenerator; +import org.apache.activemq.util.ServiceStopper; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Arrays; +import java.util.concurrent.ConcurrentHashMap; + +public class ReplicaRoleManagementBroker extends MutableBrokerFilter implements ReplicaRoleManagement { + private static final String FAIL_OVER_CONSUMER_CLIENT_ID = "DUMMY_FAIL_OVER_CONSUMER"; + + private final Logger logger = LoggerFactory.getLogger(ReplicaRoleManagementBroker.class); + private final Broker broker; + private final ReplicaPolicy replicaPolicy; + private final ClassLoader contextClassLoader; + private ReplicaRole role; + private final ReplicaStatistics replicaStatistics; + private final ReplicaReplicationQueueSupplier queueProvider; + private final WebConsoleAccessController webConsoleAccessController; + private final ReplicaInternalMessageProducer replicaInternalMessageProducer; + + protected final ProducerId replicationProducerId = new ProducerId(); + private final LongSequenceGenerator eventMessageIdGenerator = new LongSequenceGenerator(); + + ReplicaSourceBroker sourceBroker; + ReplicaBroker replicaBroker; + private ReplicaRoleStorage replicaRoleStorage; + + public ReplicaRoleManagementBroker(Broker broker, ReplicaPolicy replicaPolicy, ReplicaRole role, ReplicaStatistics replicaStatistics) { + super(broker); + this.broker = broker; + this.replicaPolicy = replicaPolicy; + this.role = role; + this.replicaStatistics = replicaStatistics; + + contextClassLoader = Thread.currentThread().getContextClassLoader(); + + replicationProducerId.setConnectionId(new IdGenerator().generateId()); + + queueProvider = new ReplicaReplicationQueueSupplier(broker); + webConsoleAccessController = new WebConsoleAccessController(broker.getBrokerService(), + replicaPolicy.isControlWebConsoleAccess()); + + replicaInternalMessageProducer = new ReplicaInternalMessageProducer(broker); + ReplicationMessageProducer replicationMessageProducer = + new ReplicationMessageProducer(replicaInternalMessageProducer, queueProvider); + ReplicaSequencer replicaSequencer = new ReplicaSequencer(broker, queueProvider, replicaInternalMessageProducer, + replicationMessageProducer, replicaPolicy, replicaStatistics); + + sourceBroker = buildSourceBroker(replicationMessageProducer, replicaSequencer, queueProvider); + replicaBroker = buildReplicaBroker(queueProvider); + + addInterceptor4CompositeQueues(); + addInterceptor4MirroredQueues(); + } + + @Override + public void start() throws Exception { + initializeTransportConnector(); + super.start(); + initializeRoleStorage(); + + MutativeRoleBroker nextByRole = getNextByRole(); + nextByRole.start(role); + setNext(nextByRole); + } + + @Override + public Subscription addConsumer(ConnectionContext context, ConsumerInfo info) throws Exception { + Subscription answer = super.addConsumer(context, info); + + if (ReplicaSupport.isReplicationRoleAdvisoryTopic(info.getDestination())) { + sendAdvisory(role); + } + + return answer; + } + + public ReplicaRole getRole() { + return role; + } + + @Override + public void brokerServiceStarted() { + super.brokerServiceStarted(); + getNextByRole().brokerServiceStarted(role); + } + + public synchronized void switchRole(ReplicaRole role, boolean force) throws Exception { + if (role != ReplicaRole.source && role != ReplicaRole.replica) { + return; + } + if (!force && this.role != ReplicaRole.source && this.role != ReplicaRole.replica) { + return; + } + if (this.role == role) { + return; + } + getNextByRole().stopBeforeRoleChange(force); + } + + public void onStopSuccess() throws Exception { + replicaStatistics.reset(); + MutativeRoleBroker nextByRole = getNextByRole(); + nextByRole.startAfterRoleChange(); + setNext(nextByRole); + } + + public Broker getBroker() { + return broker; + } + + public synchronized void updateBrokerState(ConnectionContext connectionContext, TransactionId tid, ReplicaRole role) throws Exception { + replicaRoleStorage.enqueue(connectionContext, tid, role.name()); + this.role = role; + } + + public void stopAllConnections() { + getBrokerService().stopAllConnectors(new ServiceStopper() { + @Override + public void stop(Service service) { + if (service instanceof TransportConnector && + ((TransportConnector) service).getName().equals(ReplicaSupport.REPLICATION_CONNECTOR_NAME)) { + return; + } + super.stop(service); + } + }); + webConsoleAccessController.stop(); + } + + public void startAllConnections() throws Exception { + ClassLoader originalContextClassLoader = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(contextClassLoader); + getBrokerService().startAllConnectors(); + } finally { + Thread.currentThread().setContextClassLoader(originalContextClassLoader); + } + webConsoleAccessController.start(); + } + + private void initializeRoleStorage() throws Exception { + ConnectionContext connectionContext = createConnectionContext(); + connectionContext.setClientId(FAIL_OVER_CONSUMER_CLIENT_ID); + connectionContext.setConnection(new DummyConnection()); + queueProvider.initializeRoleQueueAndTopic(); + replicaRoleStorage = new ReplicaRoleStorage(broker, queueProvider, replicaInternalMessageProducer); + ReplicaRole savedRole = replicaRoleStorage.initialize(connectionContext); + if (savedRole != null) { + role = savedRole; + } + } + + private ReplicaSourceBroker buildSourceBroker(ReplicationMessageProducer replicationMessageProducer, + ReplicaSequencer replicaSequencer, ReplicaReplicationQueueSupplier queueProvider) { + return new ReplicaSourceBroker(broker, this, replicationMessageProducer, replicaSequencer, + queueProvider, replicaPolicy); + } + + private ReplicaBroker buildReplicaBroker(ReplicaReplicationQueueSupplier queueProvider) { + return new ReplicaBroker(broker, this, queueProvider, replicaPolicy, replicaStatistics); + } + + private void addInterceptor4CompositeQueues() { + final RegionBroker regionBroker = (RegionBroker) broker.getAdaptor(RegionBroker.class); + final CompositeDestinationInterceptor compositeInterceptor = (CompositeDestinationInterceptor) regionBroker.getDestinationInterceptor(); + DestinationInterceptor[] interceptors = compositeInterceptor.getInterceptors(); + interceptors = Arrays.copyOf(interceptors, interceptors.length + 1); + interceptors[interceptors.length - 1] = new ReplicaDestinationInterceptor(sourceBroker, this); + compositeInterceptor.setInterceptors(interceptors); + } + + private void addInterceptor4MirroredQueues() { + final RegionBroker regionBroker = (RegionBroker) broker.getAdaptor(RegionBroker.class); + final CompositeDestinationInterceptor compositeInterceptor = (CompositeDestinationInterceptor) regionBroker.getDestinationInterceptor(); + DestinationInterceptor[] interceptors = compositeInterceptor.getInterceptors(); + int index = -1; + for (int i = 0; i < interceptors.length; i++) { + if (interceptors[i] instanceof MirroredQueue) { + index = i; + break; + } + } + if (index < 0) { + return; + } + DestinationInterceptor[] newInterceptors = new DestinationInterceptor[interceptors.length + 1]; + System.arraycopy(interceptors, 0, newInterceptors, 0, index + 1); + System.arraycopy(interceptors, index + 1, newInterceptors, index + 2, interceptors.length - index - 1); + newInterceptors[index + 1] = new ReplicaMirroredDestinationInterceptor(this); + compositeInterceptor.setInterceptors(newInterceptors); + } + + private MutativeRoleBroker getNextByRole() { + switch (role) { + case source: + case await_ack: + return sourceBroker; + case replica: + case ack_processed: + return replicaBroker; + default: + throw new IllegalStateException("Unknown replication role: " + role); + } + } + + private void initializeTransportConnector() throws Exception { + logger.info("Initializing Replication Transport Connector"); + TransportConnector transportConnector = getBrokerService().addConnector(replicaPolicy.getTransportConnectorUri()); + transportConnector.setUri(replicaPolicy.getTransportConnectorUri()); + transportConnector.setName(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + } + + private void sendAdvisory(ReplicaRole role) throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(role.name()); + message.setTransactionId(null); + message.setDestination(queueProvider.getRoleAdvisoryTopic()); + message.setMessageId(new MessageId(replicationProducerId, eventMessageIdGenerator.getNextSequenceId())); + message.setProducerId(replicationProducerId); + message.setPersistent(false); + message.setResponseRequired(false); + + replicaInternalMessageProducer.sendForcingFlowControl(createConnectionContext(), message); + } + + private ConnectionContext createConnectionContext() { + ConnectionContext connectionContext = getAdminConnectionContext().copy(); + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + + return connectionContext; + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSequencer.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSequencer.java new file mode 100644 index 00000000000..565846099ad --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSequencer.java @@ -0,0 +1,676 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.BrokerStoppedException; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.Command; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.DataStructure; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageDispatch; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.SessionId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.storage.ReplicaRecoverySequenceStorage; +import org.apache.activemq.replica.storage.ReplicaSequenceStorage; +import org.apache.activemq.thread.TaskRunner; +import org.apache.activemq.thread.TaskRunnerFactory; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.util.IdGenerator; +import org.apache.activemq.util.LongSequenceGenerator; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashSet; +import java.util.LinkedList; +import java.util.List; +import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; +import java.util.stream.Collectors; + +public class ReplicaSequencer { + private static final Logger logger = LoggerFactory.getLogger(ReplicaSequencer.class); + + private static final String SOURCE_CONSUMER_CLIENT_ID = "DUMMY_SOURCE_CONSUMER"; + private static final String SEQUENCE_NAME = "primarySeq"; + private static final String RESTORE_SEQUENCE_NAME = "primaryRestoreSeq"; + + private final Broker broker; + private final ReplicaReplicationQueueSupplier queueProvider; + private final ReplicaInternalMessageProducer replicaInternalMessageProducer; + private final ReplicationMessageProducer replicationMessageProducer; + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + + private final Object ackIteratingMutex = new Object(); + private final Object sendIteratingMutex = new Object(); + private final AtomicLong pendingAckWakeups = new AtomicLong(); + private final AtomicLong pendingSendWakeups = new AtomicLong(); + private final AtomicLong pendingSendTriggeredWakeups = new AtomicLong(); + final Set deliveredMessages = new HashSet<>(); + final LinkedList messageToAck = new LinkedList<>(); + final LinkedList sequenceMessageToAck = new LinkedList<>(); + private final ReplicaAckHelper replicaAckHelper; + private final ReplicaPolicy replicaPolicy; + private final ReplicaBatcher replicaBatcher; + private final ReplicaStatistics replicaStatistics; + private final MemoryUsage memoryUsage; + + ReplicaCompactor replicaCompactor; + private final LongSequenceGenerator sessionIdGenerator = new LongSequenceGenerator(); + private final LongSequenceGenerator customerIdGenerator = new LongSequenceGenerator(); + private TaskRunner ackTaskRunner; + private TaskRunner sendTaskRunner; + private Queue mainQueue; + private ConnectionContext subscriptionConnectionContext; + private ScheduledExecutorService scheduler; + + private PrefetchSubscription subscription; + boolean hasConsumer; + ReplicaSequenceStorage sequenceStorage; + ReplicaRecoverySequenceStorage restoreSequenceStorage; + + BigInteger sequence = BigInteger.ZERO; + + private final AtomicLong lastProcessTime = new AtomicLong(); + + private final AtomicBoolean initialized = new AtomicBoolean(); + + public ReplicaSequencer(Broker broker, ReplicaReplicationQueueSupplier queueProvider, + ReplicaInternalMessageProducer replicaInternalMessageProducer, + ReplicationMessageProducer replicationMessageProducer, ReplicaPolicy replicaPolicy, + ReplicaStatistics replicaStatistics) { + this.broker = broker; + this.queueProvider = queueProvider; + this.replicaInternalMessageProducer = replicaInternalMessageProducer; + this.replicationMessageProducer = replicationMessageProducer; + this.replicaAckHelper = new ReplicaAckHelper(broker); + this.replicaPolicy = replicaPolicy; + this.replicaBatcher = new ReplicaBatcher(replicaPolicy); + this.replicaStatistics = replicaStatistics; + memoryUsage = broker.getBrokerService().getSystemUsage().getMemoryUsage(); + } + + void initialize() throws Exception { + if (initialized.get()) { + return; + } + + BrokerService brokerService = broker.getBrokerService(); + TaskRunnerFactory taskRunnerFactory = brokerService.getTaskRunnerFactory(); + ackTaskRunner = taskRunnerFactory.createTaskRunner(this::iterateAck, "ReplicationPlugin.Sequencer.Ack"); + sendTaskRunner = taskRunnerFactory.createTaskRunner(this::iterateSend, "ReplicationPlugin.Sequencer.Send"); + + Queue intermediateQueue = broker.getDestinations(queueProvider.getIntermediateQueue()).stream().findFirst() + .map(DestinationExtractor::extractQueue).orElseThrow(); + mainQueue = broker.getDestinations(queueProvider.getMainQueue()).stream().findFirst() + .map(DestinationExtractor::extractQueue).orElseThrow(); + + if (subscriptionConnectionContext == null) { + subscriptionConnectionContext = createSubscriptionConnectionContext(); + } + if (sequenceStorage == null) { + sequenceStorage = new ReplicaSequenceStorage(broker, queueProvider, replicaInternalMessageProducer, + SEQUENCE_NAME); + } + if (restoreSequenceStorage == null) { + restoreSequenceStorage = new ReplicaRecoverySequenceStorage(broker, queueProvider, + replicaInternalMessageProducer, RESTORE_SEQUENCE_NAME); + } + + ConnectionId connectionId = new ConnectionId(new IdGenerator("ReplicationPlugin.Sequencer").generateId()); + SessionId sessionId = new SessionId(connectionId, sessionIdGenerator.getNextSequenceId()); + ConsumerId consumerId = new ConsumerId(sessionId, customerIdGenerator.getNextSequenceId()); + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setConsumerId(consumerId); + consumerInfo.setPrefetchSize(ReplicaSupport.INTERMEDIATE_QUEUE_PREFETCH_SIZE); + consumerInfo.setDestination(queueProvider.getIntermediateQueue()); + subscription = (PrefetchSubscription) broker.addConsumer(subscriptionConnectionContext, consumerInfo); + + replicaCompactor = new ReplicaCompactor(broker, queueProvider, subscription, + replicaPolicy.getCompactorAdditionalMessagesLimit(), replicaStatistics); + + intermediateQueue.iterate(); + String savedSequences = sequenceStorage.initialize(subscriptionConnectionContext); + List savedSequencesToRestore = restoreSequenceStorage.initialize(subscriptionConnectionContext); + restoreSequence(savedSequences, savedSequencesToRestore); + + scheduler = Executors.newSingleThreadScheduledExecutor(); + scheduler.scheduleAtFixedRate(this::asyncSendWakeup, + replicaPolicy.getSourceSendPeriod(), replicaPolicy.getSourceSendPeriod(), TimeUnit.MILLISECONDS); + + initialized.compareAndSet(false, true); + asyncSendWakeup(); + } + + void deinitialize() throws Exception { + if (!initialized.get()) { + return; + } + + if (ackTaskRunner != null) { + ackTaskRunner.shutdown(); + ackTaskRunner = null; + } + + if (sendTaskRunner != null) { + sendTaskRunner.shutdown(); + sendTaskRunner = null; + } + + mainQueue = null; + + if (subscription != null) { + try { + broker.removeConsumer(subscriptionConnectionContext, subscription.getConsumerInfo()); + } catch (BrokerStoppedException ignored) {} + subscription = null; + } + + replicaCompactor = null; + + if (sequenceStorage != null) { + sequenceStorage.deinitialize(subscriptionConnectionContext); + } + if (restoreSequenceStorage != null) { + restoreSequenceStorage.deinitialize(subscriptionConnectionContext); + } + + if (scheduler != null) { + scheduler.shutdownNow(); + } + + initialized.compareAndSet(true, false); + + } + + void restoreSequence(String savedSequence, List savedSequencesToRestore) throws Exception { + if (savedSequence != null) { + String[] split = savedSequence.split("#"); + if (split.length != 2) { + throw new IllegalStateException("Unknown sequence message format: " + savedSequence); + } + sequence = new BigInteger(split[0]); + } + + if (savedSequencesToRestore.isEmpty()) { + return; + } + + String lastMessage = savedSequencesToRestore.get(savedSequencesToRestore.size() - 1); + String[] splitLast = lastMessage.split("#"); + if (splitLast.length != 3) { + throw new IllegalStateException("Unknown sequence message format: " + lastMessage); + } + + MessageId recoveryMessageId = new MessageId(splitLast[2]); + List matchingMessages = new ArrayList<>(); + boolean found = false; + for (MessageReference mr : subscription.getDispatched()) { + matchingMessages.add(mr); + if (mr.getMessageId().equals(recoveryMessageId)) { + found = true; + break; + } + } + if (!found) { + throw new IllegalStateException("Can't recover sequence. Message with id: " + recoveryMessageId + " not found"); + } + + TransactionId transactionId = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + boolean rollbackOnFail = false; + + ConnectionContext connectionContext = createConnectionContext(); + BigInteger sequence = null; + try { + broker.beginTransaction(connectionContext, transactionId); + rollbackOnFail = true; + for (String seq : savedSequencesToRestore) { + String[] split = seq.split("#"); + if (split.length != 3) { + throw new IllegalStateException("Unknown sequence message format: " + seq); + } + + if (sequence != null && !sequence.equals(new BigInteger(split[0]))) { + throw new IllegalStateException("Sequence recovery error. Incorrect sequence. Expected sequence: " + + sequence + " saved sequence: " + seq); + } + + List batch = getBatch(matchingMessages, new MessageId(split[1]), new MessageId(split[2])); + + sequence = enqueueReplicaEvent(connectionContext, batch, new BigInteger(split[0]), transactionId); + } + + broker.commitTransaction(connectionContext, transactionId, true); + } catch (Exception e) { + logger.error("Failed to persist messages in the main replication queue", e); + if (rollbackOnFail) { + try { + broker.rollbackTransaction(connectionContext, transactionId); + } catch (Exception ex) { + logger.error("Could not rollback transaction", ex); + } + } + throw e; + } + + synchronized (deliveredMessages) { + deliveredMessages.addAll(matchingMessages.stream().map(MessageReference::getMessageId).map(MessageId::toString).collect(Collectors.toList())); + } + } + + private List getBatch(List list, MessageId firstMessageId, MessageId lastMessageId) { + List result = new ArrayList<>(); + boolean inAckRange = false; + for (MessageReference node : list) { + MessageId messageId = node.getMessageId(); + if (firstMessageId.equals(messageId)) { + inAckRange = true; + } + if (inAckRange) { + result.add(node); + if (lastMessageId.equals(messageId)) { + break; + } + } + } + return result; + } + @SuppressWarnings("unchecked") + List acknowledge(ConsumerBrokerExchange consumerExchange, MessageAck ack) throws Exception { + List messagesToAck = replicaAckHelper.getMessagesToAck(ack, mainQueue); + + if (messagesToAck == null || messagesToAck.isEmpty()) { + throw new IllegalStateException("Could not find messages for ack"); + } + List messageIds = new ArrayList<>(); + List sequenceMessageIds = new ArrayList<>(); + long timestamp = messagesToAck.get(0).getMessage().getTimestamp(); + for (MessageReference reference : messagesToAck) { + ActiveMQMessage message = (ActiveMQMessage) reference.getMessage(); + List messageIdsProperty; + if (ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)) == ReplicaEventType.BATCH) { + messageIdsProperty = (List) message.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY); + } else { + messageIdsProperty = List.of(message.getMessageId().toString()); + } + messageIds.addAll(messageIdsProperty); + sequenceMessageIds.add(messageIdsProperty.get(0)); + + timestamp = Math.max(timestamp, message.getTimestamp()); + } + + broker.acknowledge(consumerExchange, ack); + + synchronized (messageToAck) { + messageIds.forEach(messageToAck::addLast); + sequenceMessageIds.forEach(sequenceMessageToAck::addLast); + } + + long currentTime = System.currentTimeMillis(); + replicaStatistics.setTotalReplicationLag(currentTime - timestamp); + replicaStatistics.setSourceLastProcessedTime(currentTime); + + asyncAckWakeup(); + + return messagesToAck; + } + + void updateMainQueueConsumerStatus() { + try { + if (!hasConsumer && !mainQueue.getConsumers().isEmpty()) { + hasConsumer = true; + asyncSendWakeup(); + } else if (hasConsumer && mainQueue.getConsumers().isEmpty()) { + hasConsumer = false; + } + } catch (Exception error) { + logger.error("Failed to update replica consumer count.", error); + } + } + + void asyncAckWakeup() { + try { + pendingAckWakeups.incrementAndGet(); + ackTaskRunner.wakeup(); + } catch (InterruptedException e) { + logger.warn("Async task runner failed to wakeup ", e); + } + } + + void asyncSendWakeup() { + try { + long l = pendingSendWakeups.incrementAndGet(); + if (l % replicaPolicy.getMaxBatchLength() == 0) { + pendingSendTriggeredWakeups.incrementAndGet(); + sendTaskRunner.wakeup(); + pendingSendWakeups.addAndGet(-replicaPolicy.getMaxBatchLength()); + return; + } + + if (System.currentTimeMillis() - lastProcessTime.get() > replicaPolicy.getSourceSendPeriod()) { + pendingSendTriggeredWakeups.incrementAndGet(); + sendTaskRunner.wakeup(); + } + + if (!hasConsumer) { + pendingSendTriggeredWakeups.incrementAndGet(); + sendTaskRunner.wakeup(); + } + } catch (InterruptedException e) { + logger.warn("Async task runner failed to wakeup ", e); + } + } + + boolean iterateAck() { + synchronized (ackIteratingMutex) { + iterateAck0(); + + if (pendingAckWakeups.get() > 0) { + pendingAckWakeups.decrementAndGet(); + } + } + + return pendingAckWakeups.get() > 0; + } + + private void iterateAck0() { + List messages; + List sequenceMessages; + synchronized (messageToAck) { + messages = new ArrayList<>(messageToAck); + sequenceMessages = new ArrayList<>(sequenceMessageToAck); + } + + if (!messages.isEmpty()) { + ConnectionContext connectionContext = createConnectionContext(); + TransactionId transactionId = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + boolean rollbackOnFail = false; + try { + broker.beginTransaction(connectionContext, transactionId); + rollbackOnFail = true; + + ConsumerBrokerExchange consumerExchange = new ConsumerBrokerExchange(); + consumerExchange.setConnectionContext(connectionContext); + consumerExchange.setSubscription(subscription); + + for (String messageId : messages) { + MessageAck ack = new MessageAck(); + ack.setTransactionId(transactionId); + ack.setMessageID(new MessageId(messageId)); + ack.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + ack.setDestination(queueProvider.getIntermediateQueue()); + broker.acknowledge(consumerExchange, ack); + } + + restoreSequenceStorage.acknowledge(connectionContext, transactionId, sequenceMessages); + + broker.commitTransaction(connectionContext, transactionId, true); + + replicaStatistics.increaseTpsCounter(messages.size()); + + synchronized (messageToAck) { + messageToAck.removeAll(messages); + sequenceMessageToAck.removeAll(sequenceMessages); + } + + synchronized (deliveredMessages) { + messages.forEach(deliveredMessages::remove); + } + } catch (Exception e) { + logger.error("Could not acknowledge replication messages", e); + if (rollbackOnFail) { + try { + broker.rollbackTransaction(connectionContext, transactionId); + } catch (Exception ex) { + logger.error("Could not rollback transaction", ex); + } + } + } + } + } + + boolean iterateSend() { + synchronized (sendIteratingMutex) { + lastProcessTime.set(System.currentTimeMillis()); + if (!initialized.get()) { + return false; + } + + if (replicaPolicy.isSourceReplicationFlowControl()) { + long start = System.currentTimeMillis(); + long nextWarn = start; + try { + while (!memoryUsage.waitForSpace(1000, 95)) { + replicaStatistics.setSourceReplicationFlowControl(true); + long now = System.currentTimeMillis(); + if (now >= nextWarn) { + logger.warn("High memory usage. Pausing replication (paused for: {}s)", (now - start) / 1000); + nextWarn = now + 30000; + } + } + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + } + replicaStatistics.setSourceReplicationFlowControl(false); + + iterateSend0(); + + if (pendingSendTriggeredWakeups.get() > 0) { + pendingSendTriggeredWakeups.decrementAndGet(); + } + } + + return pendingSendTriggeredWakeups.get() > 0; + } + + private void iterateSend0() { + List dispatched = subscription.getDispatched(); + List toProcess = new ArrayList<>(); + + synchronized (deliveredMessages) { + Collections.reverse(dispatched); + for (MessageReference reference : dispatched) { + MessageId messageId = reference.getMessageId(); + if (deliveredMessages.contains(messageId.toString())) { + break; + } + toProcess.add(reference); + } + } + + if (toProcess.isEmpty() && hasConsumer) { + return; + } + + ConnectionContext connectionContext = createConnectionContext(); + + Collections.reverse(toProcess); + + try { + toProcess = replicaCompactor.compactAndFilter(connectionContext, toProcess, !hasConsumer && subscription.isFull()); + } catch (Exception e) { + logger.error("Failed to compact messages in the intermediate replication queue", e); + return; + } + if (!hasConsumer) { + asyncSendWakeup(); + return; + } + + if (toProcess.isEmpty()) { + return; + } + + List> batches; + try { + batches = replicaBatcher.batches(toProcess); + } catch (Exception e) { + logger.error("Filed to batch messages in the intermediate replication queue", e); + return; + } + + TransactionId transactionId = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + boolean rollbackOnFail = false; + + try { + broker.beginTransaction(connectionContext, transactionId); + rollbackOnFail = true; + + BigInteger newSequence = sequence; + for (List batch : batches) { + BigInteger newSequence1 = enqueueReplicaEvent(connectionContext, batch, newSequence, transactionId); + + restoreSequenceStorage.send(connectionContext, transactionId, newSequence + "#" + + batch.get(0).getMessageId() + "#" + + batch.get(batch.size() - 1).getMessageId(), batch.get(0).getMessageId()); + + newSequence = newSequence1; + } + sequenceStorage.enqueue(connectionContext, transactionId, newSequence + "#" + toProcess.get(toProcess.size() - 1).getMessageId()); + + broker.commitTransaction(connectionContext, transactionId, true); + + sequence = newSequence; + } catch (Exception e) { + logger.error("Failed to persist messages in the main replication queue", e); + if (rollbackOnFail) { + try { + broker.rollbackTransaction(connectionContext, transactionId); + } catch (Exception ex) { + logger.error("Could not rollback transaction", ex); + } + } + return; + } + + synchronized (deliveredMessages) { + deliveredMessages.addAll(toProcess.stream().map(MessageReference::getMessageId).map(MessageId::toString).collect(Collectors.toList())); + } + } + + private BigInteger enqueueReplicaEvent(ConnectionContext connectionContext, List batch, + BigInteger sequence, TransactionId transactionId) throws Exception { + if (batch.size() == 1) { + MessageReference reference = batch.stream().findFirst() + .orElseThrow(() -> new IllegalStateException("Cannot get message reference from batch")); + + ActiveMQMessage originalMessage = (ActiveMQMessage) reference.getMessage(); + ActiveMQMessage message = (ActiveMQMessage) originalMessage.copy(); + + message.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, sequence.toString()); + message.setDestination(queueProvider.getMainQueue()); + message.setTransactionId(transactionId); + message.setPersistent(false); + replicaInternalMessageProducer.sendForcingFlowControl(connectionContext, message); + sequence = sequence.add(BigInteger.ONE); + return sequence; + } + + List messageIds = new ArrayList<>(); + List messages = new ArrayList<>(); + long timestamp = batch.get(0).getMessage().getTimestamp(); + for (MessageReference reference : batch) { + ActiveMQMessage originalMessage = (ActiveMQMessage) reference.getMessage(); + ActiveMQMessage message = (ActiveMQMessage) originalMessage.copy(); + + message.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, sequence.toString()); + + message.setDestination(null); + message.setTransactionId(null); + message.setPersistent(false); + + messageIds.add(reference.getMessageId().toString()); + messages.add(message); + + sequence = sequence.add(BigInteger.ONE); + + // take timestamp from the newest message for statistics + timestamp = Math.max(timestamp, message.getTimestamp()); + } + + ReplicaEvent replicaEvent = new ReplicaEvent() + .setEventType(ReplicaEventType.BATCH) + .setEventData(eventSerializer.serializeListOfObjects(messages)) + .setTransactionId(transactionId) + .setTimestamp(timestamp) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, messageIds); + + replicationMessageProducer.enqueueMainReplicaEvent(connectionContext, replicaEvent); + + return sequence; + } + + private ConnectionContext createSubscriptionConnectionContext() { + ConnectionContext connectionContext = broker.getAdminConnectionContext().copy(); + connectionContext.setClientId(SOURCE_CONSUMER_CLIENT_ID); + connectionContext.setConnection(new DummyConnection() { + @Override + public void dispatchAsync(Command command) { + dispatchSync(command); + } + + @Override + public void dispatchSync(Command command) { + MessageDispatch messageDispatch = (MessageDispatch) (command.isMessageDispatch() ? command : null); + if (messageDispatch != null && ReplicaSupport.isIntermediateReplicationQueue(messageDispatch.getDestination())) { + asyncSendWakeup(); + } + } + }); + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + + return connectionContext; + } + + private ConnectionContext createConnectionContext() { + ConnectionContext connectionContext = broker.getAdminConnectionContext().copy(); + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + + return connectionContext; + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSourceBroker.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSourceBroker.java new file mode 100644 index 00000000000..a7b3e74a310 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSourceBroker.java @@ -0,0 +1,785 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ScheduledMessage; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.QueueBrowserSubscription; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.Message; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.RemoveSubscriptionInfo; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.filter.DestinationMap; +import org.apache.activemq.filter.DestinationMapEntry; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.Collection; +import java.util.List; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.stream.Collectors; + +import static org.apache.activemq.replica.ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME; + +public class ReplicaSourceBroker extends MutativeRoleBroker { + private static final DestinationMapEntry IS_REPLICATED = new DestinationMapEntry<>() { + }; // used in destination map to indicate mirrored status + + private static final Logger logger = LoggerFactory.getLogger(ReplicaSourceBroker.class); + private final ScheduledExecutorService heartBeatPoller = Executors.newSingleThreadScheduledExecutor(); + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final AtomicBoolean initialized = new AtomicBoolean(); + + private final ReplicationMessageProducer replicationMessageProducer; + private final ReplicaSequencer replicaSequencer; + private final ReplicaReplicationQueueSupplier queueProvider; + private final ReplicaPolicy replicaPolicy; + private final ReplicaAckHelper replicaAckHelper; + private ScheduledFuture heartBeatScheduledFuture; + + final DestinationMap destinationsToReplicate = new DestinationMap(); + + public ReplicaSourceBroker(Broker broker, ReplicaRoleManagement management, ReplicationMessageProducer replicationMessageProducer, + ReplicaSequencer replicaSequencer, ReplicaReplicationQueueSupplier queueProvider, + ReplicaPolicy replicaPolicy) { + super(broker, management); + this.replicationMessageProducer = replicationMessageProducer; + this.replicaSequencer = replicaSequencer; + this.queueProvider = queueProvider; + this.replicaPolicy = replicaPolicy; + this.replicaAckHelper = new ReplicaAckHelper(next); + } + + @Override + public void start(ReplicaRole role) throws Exception { + logger.info("Starting Source broker. " + (role == ReplicaRole.await_ack ? " Awaiting ack." : "")); + + initQueueProvider(); + initialized.compareAndSet(false, true); + replicaSequencer.initialize(); + initializeHeartBeatSender(); + ensureDestinationsAreReplicated(); + } + + @Override + public void brokerServiceStarted(ReplicaRole role) { + if (role == ReplicaRole.await_ack) { + stopAllConnections(); + } + } + + @Override + public void stop() throws Exception { + replicaSequencer.deinitialize(); + super.stop(); + initialized.compareAndSet(true, false); + } + + @Override + public void stopBeforeRoleChange(boolean force) throws Exception { + logger.info("Stopping Source broker. Forced [{}]", force); + stopAllConnections(); + if (force) { + stopBeforeForcedRoleChange(); + } else { + sendFailOverMessage(); + } + } + + @Override + public void startAfterRoleChange() throws Exception { + logger.info("Starting Source broker after role change"); + startAllConnections(); + + initQueueProvider(); + initialized.compareAndSet(false, true); + replicaSequencer.initialize(); + initializeHeartBeatSender(); + replicaSequencer.updateMainQueueConsumerStatus(); + } + + private void initializeHeartBeatSender() { + if (replicaPolicy.getHeartBeatPeriod() > 0) { + heartBeatScheduledFuture = heartBeatPoller.scheduleAtFixedRate(() -> { + try { + enqueueReplicaEvent( + getAdminConnectionContext(), + new ReplicaEvent() + .setEventType(ReplicaEventType.HEART_BEAT) + .setEventData(eventSerializer.serializeReplicationData(null)) + ); + } catch (Exception e) { + logger.error("Failed to send heart beat message", e); + } + }, replicaPolicy.getHeartBeatPeriod(), replicaPolicy.getHeartBeatPeriod(), TimeUnit.MILLISECONDS); + } + } + + + private void stopBeforeForcedRoleChange() throws Exception { + updateBrokerState(ReplicaRole.replica); + completeBeforeRoleChange(); + } + + private void completeBeforeRoleChange() throws Exception { + replicaSequencer.deinitialize(); + if (heartBeatScheduledFuture != null) { + heartBeatScheduledFuture.cancel(true); + } + removeReplicationQueues(); + + onStopSuccess(); + } + + private void initQueueProvider() { + queueProvider.initialize(); + queueProvider.initializeSequenceQueue(); + } + + private void ensureDestinationsAreReplicated() throws Exception { + for (ActiveMQDestination d : getDurableDestinations()) { + if (shouldReplicateDestination(d)) { + replicateDestinationCreation(getAdminConnectionContext(), d); + } + } + } + + private void replicateDestinationCreation(ConnectionContext context, ActiveMQDestination destination) throws Exception { + if (destinationsToReplicate.chooseValue(destination) != null) { + return; + } + + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_UPSERT) + .setEventData(eventSerializer.serializeReplicationData(destination)) + ); + destinationsToReplicate.put(destination, IS_REPLICATED); + } catch (Exception e) { + logger.error("Failed to replicate creation of destination {}", destination.getPhysicalName(), e); + throw e; + } + } + + private boolean shouldReplicateDestination(ActiveMQDestination destination) { + boolean isReplicationQueue = ReplicaSupport.isReplicationDestination(destination); + boolean isAdvisoryDestination = ReplicaSupport.isAdvisoryDestination(destination); + boolean isTemporaryDestination = destination.isTemporary(); + boolean shouldReplicate = !isReplicationQueue && !isAdvisoryDestination && !isTemporaryDestination; + String reason = shouldReplicate ? "" : " because "; + if (isReplicationQueue) reason += "it is a replication queue"; + if (isAdvisoryDestination) reason += "it is an advisory destination"; + if (isTemporaryDestination) reason += "it is a temporary destination"; + logger.debug("Will {}replicate destination {}{}", shouldReplicate ? "" : "not ", destination, reason); + return shouldReplicate; + } + + private boolean isReplicatedDestination(ActiveMQDestination destination) { + if (destinationsToReplicate.chooseValue(destination) == null) { + logger.debug("{} is not a replicated destination", destination.getPhysicalName()); + return false; + } + return true; + } + + public void replicateSend(ConnectionContext context, Message message, TransactionId transactionId) throws Exception { + try { + TransactionId originalTransactionId = message.getTransactionId(); + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)) + .setTransactionId(transactionId) + .setReplicationProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, message.getMessageId().toProducerKey()) + .setReplicationProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, + message.getDestination().isQueue()) + .setReplicationProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, + message.getDestination().toString()) + .setReplicationProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY, + originalTransactionId != null && originalTransactionId.isXATransaction()) + ); + } catch (Exception e) { + logger.error("Failed to replicate message {} for destination {}", message.getMessageId(), message.getDestination().getPhysicalName(), e); + throw e; + } + } + + public boolean needToReplicateSend(ConnectionContext connectionContext, Message message) { + if (isReplicaContext(connectionContext)) { + return false; + } + if (ReplicaSupport.isReplicationDestination(message.getDestination())) { + return false; + } + if (message.getDestination().isTemporary()) { + return false; + } + if (message.isAdvisory()) { + return false; + } + if (!message.isPersistent()) { + return false; + } + + return true; + } + + private void replicateBeginTransaction(ConnectionContext context, TransactionId xid) throws Exception { + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_BEGIN) + .setEventData(eventSerializer.serializeReplicationData(xid)) + ); + } catch (Exception e) { + logger.error("Failed to replicate begin of transaction [{}]", xid); + throw e; + } + } + + private void replicatePrepareTransaction(ConnectionContext context, TransactionId xid) throws Exception { + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_PREPARE) + .setEventData(eventSerializer.serializeReplicationData(xid)) + ); + } catch (Exception e) { + logger.error("Failed to replicate transaction prepare [{}]", xid); + throw e; + } + } + + private void replicateForgetTransaction(ConnectionContext context, TransactionId xid) throws Exception { + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_FORGET) + .setEventData(eventSerializer.serializeReplicationData(xid)) + ); + } catch (Exception e) { + logger.error("Failed to replicate transaction forget [{}]", xid); + throw e; + } + } + + private void replicateRollbackTransaction(ConnectionContext context, TransactionId xid) throws Exception { + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_ROLLBACK) + .setEventData(eventSerializer.serializeReplicationData(xid)) + ); + } catch (Exception e) { + logger.error("Failed to replicate transaction rollback [{}]", xid); + throw e; + } + } + + private void replicateCommitTransaction(ConnectionContext context, TransactionId xid, boolean onePhase) throws Exception { + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_COMMIT) + .setEventData(eventSerializer.serializeReplicationData(xid)) + .setReplicationProperty(ReplicaSupport.TRANSACTION_ONE_PHASE_PROPERTY, onePhase) + ); + } catch (Exception e) { + logger.error("Failed to replicate commit of transaction [{}]", xid); + throw e; + } + } + + + private void replicateDestinationRemoval(ConnectionContext context, ActiveMQDestination destination) throws Exception { + if (!isReplicatedDestination(destination)) { + return; + } + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_DELETE) + .setEventData(eventSerializer.serializeReplicationData(destination)) + ); + destinationsToReplicate.remove(destination, IS_REPLICATED); + } catch (Exception e) { + logger.error("Failed to replicate remove of destination {}", destination.getPhysicalName(), e); + throw e; + } + } + + private void sendFailOverMessage() throws Exception { + ConnectionContext connectionContext = createConnectionContext(); + + LocalTransactionId tid = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + + super.beginTransaction(connectionContext, tid); + try { + enqueueReplicaEvent( + connectionContext, + new ReplicaEvent() + .setEventType(ReplicaEventType.FAIL_OVER) + .setTransactionId(tid) + .setEventData(eventSerializer.serializeReplicationData(null)) + ); + + updateBrokerState(connectionContext, tid, ReplicaRole.await_ack); + super.commitTransaction(connectionContext, tid, true); + } catch (Exception e) { + super.rollbackTransaction(connectionContext, tid); + logger.error("Failed to send fail over message", e); + throw e; + } + } + + @Override + public Subscription addConsumer(ConnectionContext context, ConsumerInfo consumerInfo) throws Exception { + Subscription subscription = super.addConsumer(context, consumerInfo); + replicateAddConsumer(context, consumerInfo); + + if (ReplicaSupport.isMainReplicationQueue(consumerInfo.getDestination())) { + replicaSequencer.updateMainQueueConsumerStatus(); + } + + return subscription; + } + + private void replicateAddConsumer(ConnectionContext context, ConsumerInfo consumerInfo) throws Exception { + if (!needToReplicateConsumer(consumerInfo)) { + return; + } + if (ReplicaSupport.isReplicationTransport(context.getConnector())) { + return; + } + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.ADD_DURABLE_CONSUMER) + .setEventData(eventSerializer.serializeReplicationData(consumerInfo)) + .setReplicationProperty(ReplicaSupport.CLIENT_ID_PROPERTY, context.getClientId()) + ); + } catch (Exception e) { + logger.error("Failed to replicate adding {}", consumerInfo, e); + throw e; + } + } + + private boolean needToReplicateConsumer(ConsumerInfo consumerInfo) { + return consumerInfo.getDestination().isTopic() && + consumerInfo.isDurable() && + !consumerInfo.isNetworkSubscription(); + } + + @Override + public void removeConsumer(ConnectionContext context, ConsumerInfo consumerInfo) throws Exception { + super.removeConsumer(context, consumerInfo); + replicateRemoveConsumer(context, consumerInfo); + if (ReplicaSupport.isMainReplicationQueue(consumerInfo.getDestination())) { + replicaSequencer.updateMainQueueConsumerStatus(); + } + } + + private void replicateRemoveConsumer(ConnectionContext context, ConsumerInfo consumerInfo) throws Exception { + if (!needToReplicateConsumer(consumerInfo)) { + return; + } + if (ReplicaSupport.isReplicationTransport(context.getConnector())) { + return; + } + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.REMOVE_DURABLE_CONSUMER) + .setEventData(eventSerializer.serializeReplicationData(consumerInfo)) + ); + } catch (Exception e) { + logger.error("Failed to replicate adding {}", consumerInfo, e); + throw e; + } + } + + @Override + public void removeSubscription(ConnectionContext context, RemoveSubscriptionInfo subscriptionInfo) throws Exception { + super.removeSubscription(context, subscriptionInfo); + replicateRemoveSubscription(context, subscriptionInfo); + } + + private void replicateRemoveSubscription(ConnectionContext context, RemoveSubscriptionInfo subscriptionInfo) throws Exception { + if (ReplicaSupport.isReplicationTransport(context.getConnector())) { + return; + } + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.REMOVE_DURABLE_CONSUMER_SUBSCRIPTION) + .setEventData(eventSerializer.serializeReplicationData(subscriptionInfo)) + ); + } catch (Exception e) { + logger.error("Failed to replicate removing subscription {}", subscriptionInfo, e); + throw e; + } + } + + @Override + public void commitTransaction(ConnectionContext context, TransactionId xid, boolean onePhase) throws Exception { + super.commitTransaction(context, xid, onePhase); + if (xid.isXATransaction()) { + replicateCommitTransaction(context, xid, onePhase); + } + } + + @Override + public int prepareTransaction(ConnectionContext context, TransactionId xid) throws Exception { + int id = super.prepareTransaction(context, xid); + if (xid.isXATransaction()) { + replicatePrepareTransaction(context, xid); + } + return id; + } + + @Override + public void rollbackTransaction(ConnectionContext context, TransactionId xid) throws Exception { + super.rollbackTransaction(context, xid); + if (xid.isXATransaction()) { + replicateRollbackTransaction(context, xid); + } + } + + @Override + public void send(ProducerBrokerExchange producerExchange, Message messageSend) throws Exception { + final ConnectionContext connectionContext = producerExchange.getConnectionContext(); + if (!needToReplicateSend(connectionContext, messageSend)) { + super.send(producerExchange, messageSend); + return; + } + + boolean isInternalTransaction = false; + TransactionId transactionId = null; + if (messageSend.getTransactionId() != null && !messageSend.getTransactionId().isXATransaction()) { + transactionId = messageSend.getTransactionId(); + } else if (messageSend.getTransactionId() == null) { + transactionId = new LocalTransactionId(new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + if (connectionContext.getTransactions() == null) { + connectionContext.setTransactions(new ConcurrentHashMap<>()); + } + super.beginTransaction(connectionContext, transactionId); + messageSend.setTransactionId(transactionId); + isInternalTransaction = true; + } + try { + super.send(producerExchange, messageSend); + if (isInternalTransaction) { + super.commitTransaction(connectionContext, transactionId, true); + } + } catch (Exception e) { + if (isInternalTransaction) { + super.rollbackTransaction(connectionContext, transactionId); + } + throw e; + } + } + + @Override + public void beginTransaction(ConnectionContext context, TransactionId xid) throws Exception { + super.beginTransaction(context, xid); + if (xid.isXATransaction()) { + replicateBeginTransaction(context, xid); + } + } + + @Override + public void forgetTransaction(ConnectionContext context, TransactionId transactionId) throws Exception { + super.forgetTransaction(context, transactionId); + if (transactionId.isXATransaction()) { + replicateForgetTransaction(context, transactionId); + } + } + + private boolean needToReplicateAck(ConnectionContext connectionContext, MessageAck ack, PrefetchSubscription subscription) { + if (isReplicaContext(connectionContext)) { + return false; + } + if (ReplicaSupport.isReplicationDestination(ack.getDestination())) { + return false; + } + if (ack.getDestination().isTemporary()) { + return false; + } + if (subscription instanceof QueueBrowserSubscription && !connectionContext.isNetworkConnection()) { + return false; + } + + return true; + } + + @Override + public Destination addDestination(ConnectionContext context, ActiveMQDestination destination, boolean createIfTemporary) + throws Exception { + Destination newDestination = super.addDestination(context, destination, createIfTemporary); + if (shouldReplicateDestination(destination)) { + replicateDestinationCreation(context, destination); + } + return newDestination; + } + + @Override + public void removeDestination(ConnectionContext context, ActiveMQDestination destination, long timeout) throws Exception { + super.removeDestination(context, destination, timeout); + replicateDestinationRemoval(context, destination); + } + + @Override + public void acknowledge(ConsumerBrokerExchange consumerExchange, MessageAck ack) throws Exception { + if (ack.isDeliveredAck() || ack.isUnmatchedAck() || ack.isExpiredAck()) { + super.acknowledge(consumerExchange, ack); + return; + } + + ConnectionContext connectionContext = consumerExchange.getConnectionContext(); + + if (MAIN_REPLICATION_QUEUE_NAME.equals(ack.getDestination().getPhysicalName())) { + LocalTransactionId transactionId = new LocalTransactionId( + new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + + super.beginTransaction(connectionContext, transactionId); + ack.setTransactionId(transactionId); + + boolean failover = false; + try { + List ackedMessageList = replicaSequencer.acknowledge(consumerExchange, ack); + + for (MessageReference mr : ackedMessageList) { + ActiveMQMessage message = (ActiveMQMessage) mr.getMessage(); + ReplicaEventType eventType = + ReplicaEventType.valueOf(message.getStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)); + if (eventType == ReplicaEventType.FAIL_OVER) { + failover = true; + break; + } + } + + if (failover) { + updateBrokerState(connectionContext, transactionId, ReplicaRole.replica); + } + + super.commitTransaction(connectionContext, transactionId, true); + } catch (Exception e) { + super.rollbackTransaction(connectionContext, transactionId); + logger.error("Failed to send broker fail over state", e); + throw e; + } + if (failover) { + completeBeforeRoleChange(); + } + + return; + } + + PrefetchSubscription subscription = getDestinations(ack.getDestination()).stream().findFirst() + .map(Destination::getConsumers).stream().flatMap(Collection::stream) + .filter(c -> c.getConsumerInfo().getConsumerId().equals(ack.getConsumerId())) + .findFirst().filter(PrefetchSubscription.class::isInstance).map(PrefetchSubscription.class::cast) + .orElse(null); + if (subscription == null) { + super.acknowledge(consumerExchange, ack); + return; + } + + if (!needToReplicateAck(connectionContext, ack, subscription)) { + super.acknowledge(consumerExchange, ack); + return; + } + + List messageIdsToAck = getMessageIdsToAck(ack, subscription); + if (messageIdsToAck == null || messageIdsToAck.isEmpty()) { + super.acknowledge(consumerExchange, ack); + return; + } + + boolean isInternalTransaction = false; + TransactionId transactionId = null; + + if (!ack.isPoisonAck()) { + if (ack.getTransactionId() != null && !ack.getTransactionId().isXATransaction()) { + transactionId = ack.getTransactionId(); + } else if (ack.getTransactionId() == null) { + transactionId = new LocalTransactionId(new ConnectionId(ReplicaSupport.REPLICATION_PLUGIN_CONNECTION_ID), + ReplicaSupport.LOCAL_TRANSACTION_ID_GENERATOR.getNextSequenceId()); + super.beginTransaction(connectionContext, transactionId); + ack.setTransactionId(transactionId); + isInternalTransaction = true; + } + } + + try { + super.acknowledge(consumerExchange, ack); + replicateAck(connectionContext, ack, transactionId, messageIdsToAck); + if (isInternalTransaction) { + super.commitTransaction(connectionContext, transactionId, true); + } + } catch (Exception e) { + if (isInternalTransaction) { + super.rollbackTransaction(connectionContext, transactionId); + } + throw e; + } + } + + private List getMessageIdsToAck(MessageAck ack, PrefetchSubscription subscription) { + List messagesToAck = replicaAckHelper.getMessagesToAck(ack, subscription); + if (messagesToAck == null) { + return null; + } + return messagesToAck.stream() + .filter(MessageReference::isPersistent) + .map(MessageReference::getMessageId) + .map(MessageId::toProducerKey) + .collect(Collectors.toList()); + } + + private void replicateAck(ConnectionContext connectionContext, MessageAck ack, TransactionId transactionId, + List messageIdsToAck) throws Exception { + try { + TransactionId originalTransactionId = ack.getTransactionId(); + enqueueReplicaEvent( + connectionContext, + new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setTransactionId(transactionId) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, messageIdsToAck) + .setReplicationProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, + ack.getDestination().isQueue()) + .setReplicationProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, + ack.getDestination().toString()) + .setReplicationProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY, + originalTransactionId != null && originalTransactionId.isXATransaction()) + ); + } catch (Exception e) { + logger.error("Failed to replicate ack messages [{} <-> {}] for consumer {}", + ack.getFirstMessageId(), + ack.getLastMessageId(), + ack.getConsumerId(), e); + throw e; + } + } + + @Override + public void queuePurged(ConnectionContext context, ActiveMQDestination destination) { + super.queuePurged(context, destination); + if(!ReplicaSupport.isReplicationDestination(destination)) { + replicateQueuePurged(context, destination); + } else { + logger.error("Replication queue was purged {}", destination.getPhysicalName()); + } + } + + private void replicateQueuePurged(ConnectionContext connectionContext, ActiveMQDestination destination) { + try { + enqueueReplicaEvent( + connectionContext, + new ReplicaEvent() + .setEventType(ReplicaEventType.QUEUE_PURGED) + .setEventData(eventSerializer.serializeReplicationData(destination)) + ); + } catch (Exception e) { + logger.error("Failed to replicate queue purge {}", destination, e); + } + } + + @Override + public void messageExpired(ConnectionContext context, MessageReference message, Subscription subscription) { + super.messageExpired(context, message, subscription); + replicateMessageExpired(context, message); + } + + private void replicateMessageExpired(ConnectionContext context, MessageReference reference) { + Message message = reference.getMessage(); + if (!isReplicatedDestination(message.getDestination())) { + return; + } + try { + enqueueReplicaEvent( + context, + new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_EXPIRED) + .setEventData(eventSerializer.serializeReplicationData(message)) + ); + } catch (Exception e) { + logger.error("Failed to replicate discard of {}", reference.getMessageId(), e); + } + } + + @Override + public boolean sendToDeadLetterQueue(ConnectionContext context, MessageReference messageReference, Subscription subscription, Throwable poisonCause) { + if(ReplicaSupport.isReplicationDestination(messageReference.getMessage().getDestination())) { + logger.error("A replication event is being sent to DLQ. It shouldn't even happen: " + messageReference.getMessage(), poisonCause); + return false; + } + + return super.sendToDeadLetterQueue(context, messageReference, subscription, poisonCause); + } + + private void enqueueReplicaEvent(ConnectionContext connectionContext, ReplicaEvent event) throws Exception { + if (isReplicaContext(connectionContext)) { + return; + } + if (!initialized.get()) { + return; + } + replicationMessageProducer.enqueueIntermediateReplicaEvent(connectionContext, event); + } + + private boolean isReplicaContext(ConnectionContext initialContext) { + return initialContext != null && ReplicaSupport.REPLICATION_PLUGIN_USER_NAME.equals(initialContext.getUserName()); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaStatistics.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaStatistics.java new file mode 100644 index 00000000000..8fc932c2151 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaStatistics.java @@ -0,0 +1,148 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.jmx.MBeanInfo; + +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; + +public class ReplicaStatistics { + + private AtomicLong replicationTps; + private AtomicLong tpsCounter; + private AtomicLong lastTpsCounter; + + private AtomicLong totalReplicationLag; + private AtomicLong sourceLastProcessedTime; + private AtomicLong replicationLag; + private AtomicLong replicaLastProcessedTime; + + private AtomicBoolean sourceReplicationFlowControl; + private AtomicBoolean replicaReplicationFlowControl; + + public ReplicaStatistics() { + Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(() -> { + if (tpsCounter == null) { + return; + } + + long c = tpsCounter.get(); + if (replicationTps == null) { + replicationTps = new AtomicLong(); + } + replicationTps.set((c - lastTpsCounter.get()) / 10); + if (lastTpsCounter == null) { + lastTpsCounter = new AtomicLong(); + } + lastTpsCounter.set(c); + }, 60, 60, TimeUnit.SECONDS); + } + + public void reset() { + replicationTps = null; + tpsCounter = null; + lastTpsCounter = null; + + totalReplicationLag = null; + sourceLastProcessedTime = null; + replicationLag = null; + replicaLastProcessedTime = null; + + sourceReplicationFlowControl = null; + replicaReplicationFlowControl = null; + } + + public void increaseTpsCounter(long size) { + if (tpsCounter == null) { + tpsCounter = new AtomicLong(); + } + tpsCounter.addAndGet(size); + } + + public AtomicLong getReplicationTps() { + return replicationTps; + } + + public AtomicLong getTotalReplicationLag() { + return totalReplicationLag; + } + + public void setTotalReplicationLag(long totalReplicationLag) { + if (this.totalReplicationLag == null) { + this.totalReplicationLag = new AtomicLong(); + } + this.totalReplicationLag.set(totalReplicationLag); + } + + public AtomicLong getSourceLastProcessedTime() { + return sourceLastProcessedTime; + } + + public void setSourceLastProcessedTime(long sourceLastProcessedTime) { + if (this.sourceLastProcessedTime == null) { + this.sourceLastProcessedTime = new AtomicLong(); + } + this.sourceLastProcessedTime.set(sourceLastProcessedTime); + } + + public AtomicLong getReplicationLag() { + return replicationLag; + } + + public void setReplicationLag(long replicationLag) { + if (this.replicationLag == null) { + this.replicationLag = new AtomicLong(); + } + this.replicationLag.set(replicationLag); + } + + public AtomicLong getReplicaLastProcessedTime() { + return replicaLastProcessedTime; + } + + public void setReplicaLastProcessedTime(long replicaLastProcessedTime) { + if (this.replicaLastProcessedTime == null) { + this.replicaLastProcessedTime = new AtomicLong(); + } + this.replicaLastProcessedTime.set(replicaLastProcessedTime); + } + + public AtomicBoolean getSourceReplicationFlowControl() { + return sourceReplicationFlowControl; + } + + public void setSourceReplicationFlowControl(boolean sourceReplicationFlowControl) { + if (this.sourceReplicationFlowControl == null) { + this.sourceReplicationFlowControl = new AtomicBoolean(); + } + this.sourceReplicationFlowControl.set(sourceReplicationFlowControl); + } + + public AtomicBoolean getReplicaReplicationFlowControl() { + return replicaReplicationFlowControl; + } + + public void setReplicaReplicationFlowControl(boolean replicaReplicationFlowControl) { + if (this.replicaReplicationFlowControl == null) { + this.replicaReplicationFlowControl = new AtomicBoolean(); + } + this.replicaReplicationFlowControl.set(replicaReplicationFlowControl); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSupport.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSupport.java new file mode 100644 index 00000000000..a109206e1f4 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicaSupport.java @@ -0,0 +1,99 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.advisory.AdvisorySupport; +import org.apache.activemq.broker.Connector; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.util.LongSequenceGenerator; + +import java.util.Set; +import java.util.UUID; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class ReplicaSupport { + + private ReplicaSupport() { + // Intentionally hidden + } + + public static final int CURRENT_VERSION = 1; + + public static final int INTERMEDIATE_QUEUE_PREFETCH_SIZE = 10000; + + public static final String REPLICATION_CONNECTOR_NAME = "replication"; + + public static final String REPLICATION_PLUGIN_CONNECTION_ID = "replicationID" + UUID.randomUUID(); + + public static final LongSequenceGenerator LOCAL_TRANSACTION_ID_GENERATOR = new LongSequenceGenerator(); + + public static final String REPLICATION_QUEUE_PREFIX = "ActiveMQ.Plugin.Replication."; + public static final String MAIN_REPLICATION_QUEUE_NAME = REPLICATION_QUEUE_PREFIX + "Queue"; + public static final String INTERMEDIATE_REPLICATION_QUEUE_NAME = REPLICATION_QUEUE_PREFIX + "Intermediate.Queue"; + public static final String SEQUENCE_REPLICATION_QUEUE_NAME = REPLICATION_QUEUE_PREFIX + "Sequence.Queue"; + public static final String REPLICATION_ROLE_QUEUE_NAME = REPLICATION_QUEUE_PREFIX + "Role.Queue"; + public static final String REPLICATION_ROLE_ADVISORY_TOPIC_NAME = REPLICATION_QUEUE_PREFIX + "Role.Advisory.Topic"; + public static final String REPLICATION_PLUGIN_USER_NAME = "replication_plugin"; + + public static final String TRANSACTION_ONE_PHASE_PROPERTY = "transactionOnePhaseProperty"; + public static final String CLIENT_ID_PROPERTY = "clientIdProperty"; + public static final String IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY = "isOriginalMessageSentToQueueProperty"; + public static final String ORIGINAL_MESSAGE_DESTINATION_PROPERTY = "originalMessageDestinationProperty"; + public static final String IS_ORIGINAL_MESSAGE_IN_XA_TRANSACTION_PROPERTY = "isOriginalMessageInXaTransactionProperty"; + public static final String MESSAGE_ID_PROPERTY = "MessageIdProperty"; + public static final String MESSAGE_IDS_PROPERTY = "MessageIdsProperty"; + public static final String SEQUENCE_PROPERTY = "sequenceProperty"; + public static final String VERSION_PROPERTY = "versionProperty"; + + public static final Object INTERMEDIATE_QUEUE_MUTEX = new Object(); + + public static final Set DELETABLE_REPLICATION_DESTINATION_NAMES = Set.of(MAIN_REPLICATION_QUEUE_NAME, + INTERMEDIATE_REPLICATION_QUEUE_NAME, SEQUENCE_REPLICATION_QUEUE_NAME); + public static final Set REPLICATION_QUEUE_NAMES = Set.of(MAIN_REPLICATION_QUEUE_NAME, + INTERMEDIATE_REPLICATION_QUEUE_NAME, SEQUENCE_REPLICATION_QUEUE_NAME, REPLICATION_ROLE_QUEUE_NAME); + public static final Set REPLICATION_TOPIC_NAMES = Set.of(REPLICATION_ROLE_ADVISORY_TOPIC_NAME); + + public static final Set REPLICATION_DESTINATION_NAMES = Stream.concat(REPLICATION_QUEUE_NAMES.stream(), + REPLICATION_TOPIC_NAMES.stream()).collect(Collectors.toSet()); + + + public static boolean isReplicationDestination(ActiveMQDestination destination) { + return REPLICATION_DESTINATION_NAMES.contains(destination.getPhysicalName()); + } + + public static boolean isMainReplicationQueue(ActiveMQDestination destination) { + return MAIN_REPLICATION_QUEUE_NAME.equals(destination.getPhysicalName()); + } + + public static boolean isIntermediateReplicationQueue(ActiveMQDestination destination) { + return INTERMEDIATE_REPLICATION_QUEUE_NAME.equals(destination.getPhysicalName()); + } + + public static boolean isReplicationRoleAdvisoryTopic(ActiveMQDestination destination) { + return REPLICATION_ROLE_ADVISORY_TOPIC_NAME.equals(destination.getPhysicalName()); + } + + public static boolean isReplicationTransport(Connector connector) { + return connector instanceof TransportConnector && ((TransportConnector) connector).getName().equals(REPLICATION_CONNECTOR_NAME); + } + + public static boolean isAdvisoryDestination(ActiveMQDestination destination) { + return destination.getPhysicalName().startsWith(AdvisorySupport.ADVISORY_TOPIC_PREFIX); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicationMessageProducer.java b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicationMessageProducer.java new file mode 100644 index 00000000000..c3bc487da8b --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/ReplicationMessageProducer.java @@ -0,0 +1,75 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.ProducerId; +import org.apache.activemq.util.IdGenerator; +import org.apache.activemq.util.LongSequenceGenerator; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +class ReplicationMessageProducer { + + private static final Logger logger = LoggerFactory.getLogger(ReplicationMessageProducer.class); + + private final IdGenerator idGenerator = new IdGenerator(); + private final ProducerId replicationProducerId = new ProducerId(); + private final ReplicaInternalMessageProducer replicaInternalMessageProducer; + private final ReplicaReplicationQueueSupplier queueProvider; + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final LongSequenceGenerator eventMessageIdGenerator = new LongSequenceGenerator(); + + ReplicationMessageProducer(ReplicaInternalMessageProducer replicaInternalMessageProducer, + ReplicaReplicationQueueSupplier queueProvider) { + this.replicaInternalMessageProducer = replicaInternalMessageProducer; + this.queueProvider = queueProvider; + replicationProducerId.setConnectionId(idGenerator.generateId()); + } + + void enqueueIntermediateReplicaEvent(ConnectionContext connectionContext, ReplicaEvent event) throws Exception { + synchronized (ReplicaSupport.INTERMEDIATE_QUEUE_MUTEX) { + enqueueReplicaEvent(connectionContext, event, true, queueProvider.getIntermediateQueue()); + } + } + + void enqueueMainReplicaEvent(ConnectionContext connectionContext, ReplicaEvent event) throws Exception { + enqueueReplicaEvent(connectionContext, event, false, queueProvider.getMainQueue()); + } + + private void enqueueReplicaEvent(ConnectionContext connectionContext, ReplicaEvent event, + boolean persistent, ActiveMQQueue mainQueue) throws Exception { + ActiveMQMessage eventMessage = new ActiveMQMessage(); + eventMessage.setPersistent(persistent); + eventMessage.setType("ReplicaEvent"); + eventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + eventMessage.setMessageId(new MessageId(replicationProducerId, eventMessageIdGenerator.getNextSequenceId())); + eventMessage.setDestination(mainQueue); + eventMessage.setProducerId(replicationProducerId); + eventMessage.setResponseRequired(false); + eventMessage.setContent(event.getEventData()); + eventMessage.setProperties(event.getReplicationProperties()); + eventMessage.setTransactionId(event.getTransactionId()); + eventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, event.getVersion() == null ? ReplicaSupport.CURRENT_VERSION : event.getVersion()); + eventMessage.setTimestamp(event.getTimestamp() == null ? System.currentTimeMillis() : event.getTimestamp()); + replicaInternalMessageProducer.sendForcingFlowControl(connectionContext, eventMessage); + } +} + diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/WebConsoleAccessController.java b/activemq-broker/src/main/java/org/apache/activemq/replica/WebConsoleAccessController.java new file mode 100644 index 00000000000..6342774c180 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/WebConsoleAccessController.java @@ -0,0 +1,109 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.BrokerContext; +import org.apache.activemq.broker.BrokerService; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.lang.reflect.Method; +import java.util.Locale; +import java.util.Map; + +public class WebConsoleAccessController { + + private final Logger logger = LoggerFactory.getLogger(WebConsoleAccessController.class); + + private final BrokerService brokerService; + private Class serverClass; + private Class connectorClass; + private Method getConnectorsMethod; + private Method startMethod; + private Method stopMethod; + private boolean initialized; + + public WebConsoleAccessController(BrokerService brokerService, boolean enabled) { + this.brokerService = brokerService; + if (!enabled) { + return; + } + try { + serverClass = getClass().getClassLoader().loadClass("org.eclipse.jetty.server.Server"); + connectorClass = getClass().getClassLoader().loadClass("org.eclipse.jetty.server.Connector"); + + getConnectorsMethod = serverClass.getMethod("getConnectors"); + startMethod = connectorClass.getMethod("start"); + stopMethod = connectorClass.getMethod("stop"); + initialized = true; + } catch (ClassNotFoundException | NoSuchMethodException e) { + logger.error("Unable to initialize class", e); + } + } + + public void start() { + invoke(startMethod); + } + + public void stop() { + invoke(stopMethod); + } + + private void invoke(Method method) { + if (!initialized) { + return; + } + + if (brokerService.getBrokerContext() != null) { + invoke(method, brokerService.getBrokerContext()); + return; + } + + new Thread(() -> { + BrokerContext brokerContext; + while ((brokerContext = brokerService.getBrokerContext()) == null) { + try { + Thread.sleep(100); + } catch (InterruptedException e) { + logger.error("brokerContext initialization interrupted", e); + return; + } + } + invoke(method, brokerContext); + }).start(); + } + + private void invoke(Method method, BrokerContext brokerContext) { + try { + Map servers = brokerContext.getBeansOfType(serverClass); + if (servers.size() > 0) { + for (Map.Entry server : servers.entrySet()) { + if (server.getKey().toLowerCase(Locale.ROOT).contains("jolokia")) { + continue; + } + + Object[] connectors = (Object[]) getConnectorsMethod.invoke(server.getValue()); + for (Object connector : connectors) { + method.invoke(connector); + } + } + } + } catch (Exception e) { + logger.error("Unable to {} web console connectors", method.getName(), e); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationJmxHelper.java b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationJmxHelper.java new file mode 100644 index 00000000000..90ffec39085 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationJmxHelper.java @@ -0,0 +1,41 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.jmx; + +import org.apache.activemq.broker.BrokerService; + +import javax.management.MalformedObjectNameException; +import javax.management.ObjectName; + +public class ReplicationJmxHelper { + + private ReplicationJmxHelper() { + } + + public static ObjectName createJmxName(BrokerService brokerService) { + try { + String objectNameStr = brokerService.getBrokerObjectName().toString(); + + objectNameStr += "," + "service=Plugins"; + objectNameStr += "," + "instanceName=ReplicationPlugin"; + + return new ObjectName(objectNameStr); + } catch (MalformedObjectNameException e) { + throw new RuntimeException("Failed to create JMX view for ReplicationPlugin", e); + } + } +} \ No newline at end of file diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationView.java b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationView.java new file mode 100644 index 00000000000..9f6d1ecdf8c --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationView.java @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.jmx; + +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaStatistics; + +import java.util.Optional; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicLong; + +public class ReplicationView implements ReplicationViewMBean { + + private final ReplicaPlugin plugin; + private final ReplicaStatistics replicaStatistics; + + public ReplicationView(ReplicaPlugin plugin, ReplicaStatistics replicaStatistics) { + this.plugin = plugin; + this.replicaStatistics = replicaStatistics; + } + + @Override + public Long getReplicationTps() { + return Optional.ofNullable(replicaStatistics.getReplicationTps()).map(AtomicLong::get).orElse(null); + } + + @Override + public void setReplicationRole(String role, boolean force) throws Exception { + plugin.setReplicaRole(ReplicaRole.valueOf(role), force); + } + + @Override + public String getReplicationRole() { + return plugin.getRole().name(); + } + + @Override + public Long getTotalReplicationLag() { + return Optional.ofNullable(replicaStatistics.getTotalReplicationLag()).map(AtomicLong::get).orElse(null); + } + + @Override + public Long getSourceWaitTime() { + return Optional.ofNullable(replicaStatistics.getSourceLastProcessedTime()).map(AtomicLong::get) + .map(v -> System.currentTimeMillis() - v).orElse(null); + } + + @Override + public Long getReplicationLag() { + return Optional.ofNullable(replicaStatistics.getReplicationLag()).map(AtomicLong::get).orElse(null); + } + + @Override + public Long getReplicaWaitTime() { + return Optional.ofNullable(replicaStatistics.getReplicaLastProcessedTime()).map(AtomicLong::get) + .map(v -> System.currentTimeMillis() - v).orElse(null); + } + + @Override + public Boolean getSourceReplicationFlowControl() { + return Optional.ofNullable(replicaStatistics.getSourceReplicationFlowControl()).map(AtomicBoolean::get).orElse(null); + } + + @Override + public Boolean getReplicaReplicationFlowControl() { + return Optional.ofNullable(replicaStatistics.getReplicaReplicationFlowControl()).map(AtomicBoolean::get).orElse(null); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationViewMBean.java b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationViewMBean.java new file mode 100644 index 00000000000..95d6b5141f0 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/jmx/ReplicationViewMBean.java @@ -0,0 +1,49 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.jmx; + +import org.apache.activemq.broker.jmx.MBeanInfo; + +public interface ReplicationViewMBean { + + @MBeanInfo("Replication TPS") + Long getReplicationTps(); + + @MBeanInfo("Set replication role for broker") + void setReplicationRole(String role, boolean force) throws Exception; + + @MBeanInfo("Get current replication role for broker") + String getReplicationRole(); + + @MBeanInfo("Total replication lag") + Long getTotalReplicationLag(); + + @MBeanInfo("Get wait time(if the broker's role is source)") + Long getSourceWaitTime(); + + @MBeanInfo("Get replication lag") + Long getReplicationLag(); + + @MBeanInfo("Get wait time(if the broker's role is replica)") + Long getReplicaWaitTime(); + + @MBeanInfo("Flow control is enabled for replication on the source side") + Boolean getSourceReplicationFlowControl(); + + @MBeanInfo("Flow control is enabled for replication on the replica side") + Boolean getReplicaReplicationFlowControl(); +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseSequenceStorage.java b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseSequenceStorage.java new file mode 100644 index 00000000000..8a415ae4b92 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseSequenceStorage.java @@ -0,0 +1,61 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerStoppedException; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static java.util.Objects.requireNonNull; + +public abstract class ReplicaBaseSequenceStorage extends ReplicaBaseStorage { + + private final Logger logger = LoggerFactory.getLogger(ReplicaBaseSequenceStorage.class); + + static final String SEQUENCE_NAME_PROPERTY = "SequenceName"; + private final String sequenceName; + + public ReplicaBaseSequenceStorage(Broker broker, ReplicaReplicationQueueSupplier queueProvider, + ReplicaInternalMessageProducer replicaInternalMessageProducer, String sequenceName) { + super(broker, replicaInternalMessageProducer, queueProvider.getSequenceQueue(), + "ReplicationPlugin.ReplicaSequenceStorage", + String.format("%s LIKE '%s'", SEQUENCE_NAME_PROPERTY, sequenceName)); + this.sequenceName = requireNonNull(sequenceName); + } + + public void deinitialize(ConnectionContext connectionContext) throws Exception { + queue = null; + + if (subscription != null) { + try { + broker.removeConsumer(connectionContext, subscription.getConsumerInfo()); + } catch (BrokerStoppedException ignored) {} + subscription = null; + } + } + + @Override + public void send(ConnectionContext connectionContext, ActiveMQTextMessage seqMessage) throws Exception { + seqMessage.setStringProperty(SEQUENCE_NAME_PROPERTY, sequenceName); + super.send(connectionContext, seqMessage); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseStorage.java b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseStorage.java new file mode 100644 index 00000000000..82072347dc6 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaBaseStorage.java @@ -0,0 +1,135 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.ProducerId; +import org.apache.activemq.command.SessionId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.DestinationExtractor; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.util.IdGenerator; +import org.apache.activemq.util.LongSequenceGenerator; + +import java.util.List; +import java.util.stream.Collectors; + +import static java.util.Objects.requireNonNull; + +public abstract class ReplicaBaseStorage { + + protected final ProducerId replicationProducerId = new ProducerId(); + private final LongSequenceGenerator eventMessageIdGenerator = new LongSequenceGenerator(); + + protected Broker broker; + protected ConnectionContext connectionContext; + protected ReplicaInternalMessageProducer replicaInternalMessageProducer; + protected ActiveMQQueue destination; + protected Queue queue; + private final String idGeneratorPrefix; + private final String selector; + protected PrefetchSubscription subscription; + + public ReplicaBaseStorage(Broker broker, ReplicaInternalMessageProducer replicaInternalMessageProducer, + ActiveMQQueue destination, String idGeneratorPrefix, String selector) { + this.broker = requireNonNull(broker); + this.replicaInternalMessageProducer = requireNonNull(replicaInternalMessageProducer); + this.destination = destination; + this.idGeneratorPrefix = idGeneratorPrefix; + this.selector = selector; + + replicationProducerId.setConnectionId(new IdGenerator().generateId()); + } + + protected List initializeBase(ConnectionContext connectionContext) throws Exception { + queue = broker.getDestinations(destination).stream().findFirst() + .map(DestinationExtractor::extractQueue).orElseThrow(); + ConnectionId connectionId = new ConnectionId(new IdGenerator(idGeneratorPrefix).generateId()); + SessionId sessionId = new SessionId(connectionId, new LongSequenceGenerator().getNextSequenceId()); + ConsumerId consumerId = new ConsumerId(sessionId, new LongSequenceGenerator().getNextSequenceId()); + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setConsumerId(consumerId); + consumerInfo.setPrefetchSize(ReplicaSupport.INTERMEDIATE_QUEUE_PREFETCH_SIZE); + consumerInfo.setDestination(destination); + if (selector != null) { + consumerInfo.setSelector(selector); + } + subscription = (PrefetchSubscription) broker.addConsumer(connectionContext, consumerInfo); + queue.iterate(); + + return subscription.getDispatched().stream().map(MessageReference::getMessage) + .map(ActiveMQTextMessage.class::cast).collect(Collectors.toList()); + } + + protected void acknowledgeAll(ConnectionContext connectionContext, TransactionId tid) throws Exception { + List dispatched = subscription.getDispatched(); + + if (!dispatched.isEmpty()) { + MessageAck ack = new MessageAck(dispatched.get(dispatched.size() - 1).getMessage(), MessageAck.STANDARD_ACK_TYPE, dispatched.size()); + ack.setFirstMessageId(dispatched.get(0).getMessageId()); + ack.setDestination(destination); + ack.setTransactionId(tid); + acknowledge(connectionContext, ack); + } + } + + protected void acknowledge(ConnectionContext connectionContext, MessageAck ack) throws Exception { + ConsumerBrokerExchange consumerExchange = new ConsumerBrokerExchange(); + consumerExchange.setConnectionContext(connectionContext); + consumerExchange.setSubscription(subscription); + + broker.acknowledge(consumerExchange, ack); + } + + public void enqueue(ConnectionContext connectionContext, TransactionId tid, String message) throws Exception { + // before enqueue message, we acknowledge all messages currently in queue. + acknowledgeAll(connectionContext, tid); + + send(connectionContext, tid, message, + new MessageId(replicationProducerId, eventMessageIdGenerator.getNextSequenceId())); + } + + public void send(ConnectionContext connectionContext, TransactionId tid, String message, MessageId messageId) throws Exception { + ActiveMQTextMessage seqMessage = new ActiveMQTextMessage(); + seqMessage.setText(message); + seqMessage.setTransactionId(tid); + seqMessage.setDestination(destination); + seqMessage.setMessageId(messageId); + seqMessage.setProducerId(replicationProducerId); + seqMessage.setPersistent(true); + seqMessage.setResponseRequired(false); + + send(connectionContext, seqMessage); + } + + public void send(ConnectionContext connectionContext, ActiveMQTextMessage seqMessage) throws Exception { + replicaInternalMessageProducer.sendForcingFlowControl(connectionContext, seqMessage); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorage.java b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorage.java new file mode 100644 index 00000000000..392c39be622 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorage.java @@ -0,0 +1,56 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; + +import java.util.ArrayList; +import java.util.List; + +public class ReplicaRecoverySequenceStorage extends ReplicaBaseSequenceStorage { + + public ReplicaRecoverySequenceStorage(Broker broker, ReplicaReplicationQueueSupplier queueProvider, + ReplicaInternalMessageProducer replicaInternalMessageProducer, String sequenceName) { + super(broker, queueProvider, replicaInternalMessageProducer, sequenceName); + } + + public List initialize(ConnectionContext connectionContext) throws Exception { + List result = new ArrayList<>(); + for (ActiveMQTextMessage message : super.initializeBase(connectionContext)) { + result.add(message.getText()); + } + return result; + } + + public void acknowledge(ConnectionContext connectionContext, TransactionId tid, List messageIds) throws Exception { + for (String messageId : messageIds) { + MessageAck ack = new MessageAck(); + ack.setMessageID(new MessageId(messageId)); + ack.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + ack.setDestination(destination); + ack.setTransactionId(tid); + acknowledge(connectionContext, ack); + } + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRoleStorage.java b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRoleStorage.java new file mode 100644 index 00000000000..39ff84e5a55 --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaRoleStorage.java @@ -0,0 +1,57 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.util.IdGenerator; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.List; + +public class ReplicaRoleStorage extends ReplicaBaseStorage { + + private final Logger logger = LoggerFactory.getLogger(ReplicaRoleStorage.class); + + public ReplicaRoleStorage(Broker broker, ReplicaReplicationQueueSupplier queueProvider, + ReplicaInternalMessageProducer replicaInternalMessageProducer) { + super(broker, replicaInternalMessageProducer, queueProvider.getRoleQueue(), "ReplicationPlugin.ReplicaFailOverStorage", null); + this.replicationProducerId.setConnectionId(new IdGenerator().generateId()); + } + + public ReplicaRole initialize(ConnectionContext connectionContext) throws Exception { + List allMessages = super.initializeBase(connectionContext); + + if (allMessages.size() == 0) { + return null; + } + + if (allMessages.size() > 1) { + logger.error("Found more than one message during role storage initialization"); + for (int i = 0; i < allMessages.size() - 1; i++) { + queue.removeMessage(allMessages.get(i).getMessageId().toString()); + } + } + + return ReplicaRole.valueOf(allMessages.get(0).getText()); + } +} diff --git a/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaSequenceStorage.java b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaSequenceStorage.java new file mode 100644 index 00000000000..e3c485afdcb --- /dev/null +++ b/activemq-broker/src/main/java/org/apache/activemq/replica/storage/ReplicaSequenceStorage.java @@ -0,0 +1,54 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.List; + +public class ReplicaSequenceStorage extends ReplicaBaseSequenceStorage { + + private final Logger logger = LoggerFactory.getLogger(ReplicaSequenceStorage.class); + + public ReplicaSequenceStorage(Broker broker, ReplicaReplicationQueueSupplier queueProvider, + ReplicaInternalMessageProducer replicaInternalMessageProducer, String sequenceName) { + super(broker, queueProvider, replicaInternalMessageProducer, sequenceName); + } + + public String initialize(ConnectionContext connectionContext) throws Exception { + List allMessages = super.initializeBase(connectionContext); + + if (allMessages.size() == 0) { + return null; + } + + if (allMessages.size() > 1) { + logger.error("Found more than one message during sequence storage initialization"); + for (int i = 0; i < allMessages.size() - 1; i++) { + queue.removeMessage(allMessages.get(i).getMessageId().toString()); + } + } + + return allMessages.get(0).getText(); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/DestinationExtractorTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/DestinationExtractorTest.java new file mode 100644 index 00000000000..a4d4b1de6ea --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/DestinationExtractorTest.java @@ -0,0 +1,52 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.region.DestinationFilter; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.Topic; +import org.junit.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.Mockito.mock; + +public class DestinationExtractorTest { + + @Test + public void extractQueueFromQueue() { + Queue queue = mock(Queue.class); + Queue result = DestinationExtractor.extractQueue(queue); + + assertThat(result).isEqualTo(queue); + } + + @Test + public void extractQueueFromDestinationFilter() { + Queue queue = mock(Queue.class); + Queue result = DestinationExtractor.extractQueue(new DestinationFilter(queue)); + + assertThat(result).isEqualTo(queue); + } + + @Test + public void extractNullFromNonQueue() { + Topic topic = mock(Topic.class); + Queue result = DestinationExtractor.extractQueue(topic); + + assertThat(result).isNull(); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBatcherTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBatcherTest.java new file mode 100644 index 00000000000..429f78e9495 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBatcherTest.java @@ -0,0 +1,310 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.Message; +import org.apache.activemq.command.MessageId; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +public class ReplicaBatcherTest { + + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + + @Test + public void batchesSmallMessages() throws Exception { + List list = new ArrayList<>(); + for (int i = 0; i < 1347; i++) { + ActiveMQMessage message = new ActiveMQMessage(); + message.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + list.add(new DummyMessageReference(new MessageId("1:0:0:" + i), message, 1)); + } + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(3); + assertThat(batches.get(0).size()).isEqualTo(replicaPolicy.getMaxBatchLength()); + for (int i = 0; i < replicaPolicy.getMaxBatchLength(); i++) { + assertThat(batches.get(0).get(i).getMessageId().toString()).isEqualTo("1:0:0:" + i); + } + assertThat(batches.get(1).size()).isEqualTo(replicaPolicy.getMaxBatchLength()); + for (int i = 0; i < replicaPolicy.getMaxBatchLength(); i++) { + assertThat(batches.get(1).get(i).getMessageId().toString()).isEqualTo("1:0:0:" + (i + replicaPolicy.getMaxBatchLength())); + } + assertThat(batches.get(2).size()).isEqualTo(347); + for (int i = 0; i < 347; i++) { + assertThat(batches.get(2).get(i).getMessageId().toString()).isEqualTo("1:0:0:" + (i + replicaPolicy.getMaxBatchLength() * 2)); + } + } + + @Test + public void batchesBigMessages() throws Exception { + ActiveMQMessage message = new ActiveMQMessage(); + message.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + List list = new ArrayList<>(); + list.add(new DummyMessageReference(new MessageId("1:0:0:1"), message, replicaPolicy.getMaxBatchSize() + 1)); + list.add(new DummyMessageReference(new MessageId("1:0:0:2"), message, replicaPolicy.getMaxBatchSize() / 2 + 1)); + list.add(new DummyMessageReference(new MessageId("1:0:0:3"), message, replicaPolicy.getMaxBatchSize() / 2)); + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(3); + assertThat(batches.get(0).size()).isEqualTo(1); + assertThat(batches.get(0).get(0).getMessageId().toString()).isEqualTo("1:0:0:1"); + assertThat(batches.get(1).size()).isEqualTo(1); + assertThat(batches.get(1).get(0).getMessageId().toString()).isEqualTo("1:0:0:2"); + assertThat(batches.get(2).size()).isEqualTo(1); + assertThat(batches.get(2).get(0).getMessageId().toString()).isEqualTo("1:0:0:3"); + } + + @Test + public void batchesAcksAfterSendsSameId() throws Exception { + List list = new ArrayList<>(); + ActiveMQMessage activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:1"); + list.add(new DummyMessageReference(new MessageId("1:0:0:1"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:2"); + list.add(new DummyMessageReference(new MessageId("1:0:0:2"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + activeMQMessage.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("1:0:0:1")); + list.add(new DummyMessageReference(new MessageId("1:0:0:3"), activeMQMessage, 1)); + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(2); + assertThat(batches.get(0).size()).isEqualTo(2); + assertThat(batches.get(0).get(0).getMessageId().toString()).isEqualTo("1:0:0:1"); + assertThat(batches.get(0).get(1).getMessageId().toString()).isEqualTo("1:0:0:2"); + assertThat(batches.get(1).size()).isEqualTo(1); + assertThat(batches.get(1).get(0).getMessageId().toString()).isEqualTo("1:0:0:3"); + } + + @Test + public void batchesAcksAfterSendsSameId2() throws Exception { + List list = new ArrayList<>(); + ActiveMQMessage activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:1"); + list.add(new DummyMessageReference(new MessageId("1:0:0:1"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + activeMQMessage.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("1:0:0:1")); + list.add(new DummyMessageReference(new MessageId("1:0:0:2"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:1"); + list.add(new DummyMessageReference(new MessageId("1:0:0:3"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + activeMQMessage.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("1:0:0:1")); + list.add(new DummyMessageReference(new MessageId("1:0:0:4"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:2"); + list.add(new DummyMessageReference(new MessageId("1:0:0:5"), activeMQMessage, 1)); + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(3); + assertThat(batches.get(0).size()).isEqualTo(1); + assertThat(batches.get(0).get(0).getMessageId().toString()).isEqualTo("1:0:0:1"); + assertThat(batches.get(1).size()).isEqualTo(2); + assertThat(batches.get(1).get(0).getMessageId().toString()).isEqualTo("1:0:0:2"); + assertThat(batches.get(1).get(1).getMessageId().toString()).isEqualTo("1:0:0:3"); + assertThat(batches.get(2).size()).isEqualTo(2); + assertThat(batches.get(2).get(0).getMessageId().toString()).isEqualTo("1:0:0:4"); + assertThat(batches.get(2).get(1).getMessageId().toString()).isEqualTo("1:0:0:5"); + } + + @Test + public void batchesAcksAfterSendsDifferentIds() throws Exception { + List list = new ArrayList<>(); + ActiveMQMessage activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:1"); + list.add(new DummyMessageReference(new MessageId("1:0:0:1"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:2"); + list.add(new DummyMessageReference(new MessageId("1:0:0:2"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + activeMQMessage.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("1:0:0:4")); + list.add(new DummyMessageReference(new MessageId("1:0:0:3"), activeMQMessage, 1)); + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(1); + assertThat(batches.get(0).size()).isEqualTo(3); + assertThat(batches.get(0).get(0).getMessageId().toString()).isEqualTo("1:0:0:1"); + assertThat(batches.get(0).get(1).getMessageId().toString()).isEqualTo("1:0:0:2"); + assertThat(batches.get(0).get(2).getMessageId().toString()).isEqualTo("1:0:0:3"); + } + + @Test + public void batchesFailOverMessageSeparately() throws Exception { + List list = new ArrayList<>(); + ActiveMQMessage activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:1"); + list.add(new DummyMessageReference(new MessageId("1:0:0:1"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + activeMQMessage.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("1:0:0:5")); + list.add(new DummyMessageReference(new MessageId("1:0:0:2"), activeMQMessage, 1)); + activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setStringProperty(ReplicaSupport.ORIGINAL_MESSAGE_DESTINATION_PROPERTY, "test"); + activeMQMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.FAIL_OVER.toString()); + activeMQMessage.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "1:0:0:3"); + list.add(new DummyMessageReference(new MessageId("1:0:0:3"), activeMQMessage, 1)); + + List> batches = new ReplicaBatcher(replicaPolicy).batches(list); + assertThat(batches.size()).isEqualTo(2); + assertThat(batches.get(0).size()).isEqualTo(2); + assertThat(batches.get(1).size()).isEqualTo(1); + assertThat(batches.get(0).get(0).getMessageId().toString()).isEqualTo("1:0:0:1"); + assertThat(batches.get(0).get(1).getMessageId().toString()).isEqualTo("1:0:0:2"); + assertThat(batches.get(1).get(0).getMessageId().toString()).isEqualTo("1:0:0:3"); + } + + private static class DummyMessageReference implements MessageReference { + + private final MessageId messageId; + private Message message; + private final int size; + + DummyMessageReference(MessageId messageId, Message message, int size) { + this.messageId = messageId; + this.message = message; + this.size = size; + } + + @Override + public MessageId getMessageId() { + return messageId; + } + + @Override + public Message getMessageHardRef() { + return null; + } + + @Override + public Message getMessage() { + return message; + } + + @Override + public boolean isPersistent() { + return false; + } + + @Override + public Message.MessageDestination getRegionDestination() { + return null; + } + + @Override + public int getRedeliveryCounter() { + return 0; + } + + @Override + public void incrementRedeliveryCounter() { + + } + + @Override + public int getReferenceCount() { + return 0; + } + + @Override + public int incrementReferenceCount() { + return 0; + } + + @Override + public int decrementReferenceCount() { + return 0; + } + + @Override + public ConsumerId getTargetConsumerId() { + return null; + } + + @Override + public int getSize() { + return size; + } + + @Override + public long getExpiration() { + return 0; + } + + @Override + public String getGroupID() { + return null; + } + + @Override + public int getGroupSequence() { + return 0; + } + + @Override + public boolean isExpired() { + return false; + } + + @Override + public boolean isDropped() { + return false; + } + + @Override + public boolean isAdvisory() { + return false; + } + + @Override + public boolean canProcessAsExpired() { + return false; + } + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBrokerEventListenerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBrokerEventListenerTest.java new file mode 100644 index 00000000000..6dbe38a5f02 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaBrokerEventListenerTest.java @@ -0,0 +1,1010 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.TransactionBroker; +import org.apache.activemq.broker.region.Destination; +import org.apache.activemq.broker.region.DurableTopicSubscription; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.Topic; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageDispatchNotification; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.command.XATransactionId; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.usage.SystemUsage; + +import org.apache.activemq.util.IOHelper; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import javax.jms.JMSException; +import javax.transaction.xa.XAException; +import javax.transaction.xa.Xid; +import java.io.File; +import java.math.BigInteger; +import java.nio.charset.StandardCharsets; +import java.util.Collections; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaBrokerEventListenerTest { + + private final ReplicaBroker replicaBroker = mock(ReplicaBroker.class); + private final Broker broker = mock(Broker.class); + private final ActiveMQQueue sequenceQueue = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + private final ActiveMQQueue testQueue = new ActiveMQQueue("TEST.QUEUE"); + private final ActiveMQTopic testTopic = new ActiveMQTopic("TEST.TOPIC"); + private final Destination sequenceDstinationQueue = mock(Queue.class); + private final Destination destinationQueue = mock(Queue.class); + private final Destination destinationTopic = mock(Topic.class); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final ReplicaReplicationQueueSupplier queueProvider = mock(ReplicaReplicationQueueSupplier.class); + private final PrefetchSubscription subscription = mock(PrefetchSubscription.class); + private final TransactionBroker transactionBroker = mock(TransactionBroker.class); + private ReplicaBrokerEventListener listener; + private PeriodAcknowledge acknowledgeCallback; + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final ReplicaStatistics replicaStatistics = new ReplicaStatistics(); + + @Before + public void setUp() throws Exception { + when(replicaBroker.getNext()).thenReturn(broker); + ConnectionContext adminConnectionContext = mock(ConnectionContext.class); + when(adminConnectionContext.copy()).thenReturn(connectionContext); + when(broker.getAdminConnectionContext()).thenReturn(adminConnectionContext); + when(broker.getDestinations(testQueue)).thenReturn(Set.of(destinationQueue)); + when(broker.getDestinations(testTopic)).thenReturn(Set.of(destinationTopic)); + when(connectionContext.isProducerFlowControl()).thenReturn(false); + when(connectionContext.copy()).thenReturn(new ConnectionContext()); + when(connectionContext.getUserName()).thenReturn(ReplicaSupport.REPLICATION_PLUGIN_USER_NAME); + BrokerService brokerService = mock(BrokerService.class); + when(broker.getBrokerService()).thenReturn(brokerService); + File brokerDataDirectory = new File(IOHelper.getDefaultDataDirectory()); + when(brokerService.getBrokerDataDirectory()).thenReturn(brokerDataDirectory); + when(queueProvider.getSequenceQueue()).thenReturn(sequenceQueue); + when(broker.getDestinations(sequenceQueue)).thenReturn(Set.of(sequenceDstinationQueue)); + when(broker.addConsumer(any(), any())).thenReturn(subscription); + when(broker.getAdaptor(TransactionBroker.class)).thenReturn(transactionBroker); + SystemUsage systemUsage = mock(SystemUsage.class); + when(brokerService.getSystemUsage()).thenReturn(systemUsage); + MemoryUsage memoryUsage = mock(MemoryUsage.class); + when(systemUsage.getMemoryUsage()).thenReturn(memoryUsage); + when(memoryUsage.waitForSpace(anyLong(), anyInt())).thenReturn(true); + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + acknowledgeCallback = new PeriodAcknowledge(replicaPolicy); + listener = new ReplicaBrokerEventListener(replicaBroker, queueProvider, acknowledgeCallback, replicaPolicy, replicaStatistics); + listener.initialize(); + } + + @Test + public void canHandleEventOfType_DESTINATION_UPSERT_whenQueueNotExist() throws Exception { + listener.sequence = null; + ActiveMQDestination activeMQDestination = new ActiveMQQueue("NOT.EXIST"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{activeMQDestination}); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_UPSERT) + .setEventData(eventSerializer.serializeReplicationData(testQueue)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker).addDestination(connectionContext, testQueue, true); + } + + @Test + public void canHandleEventOfType_DESTINATION_UPSERT_whenQueueExists() throws Exception { + listener.sequence = null; + ActiveMQDestination activeMQDestination = new ActiveMQQueue("NOT.EXIST"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{activeMQDestination, testQueue}); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_UPSERT) + .setEventData(eventSerializer.serializeReplicationData(testQueue)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker, never()).addDestination(connectionContext, testQueue, true); + } + + @Test + public void canHandleEventOfType_DESTINATION_DELETE_whenDestinationExists() throws Exception { + listener.sequence = null; + ActiveMQDestination activeMQDestination = new ActiveMQQueue("NOT.EXIST"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{activeMQDestination, testQueue}); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_DELETE) + .setEventData(eventSerializer.serializeReplicationData(testQueue)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker).removeDestination(connectionContext, testQueue, 1000); + } + + @Test + public void canHandleEventOfType_DESTINATION_DELETE_whenDestinationNotExists() throws Exception { + listener.sequence = null; + ActiveMQDestination activeMQDestination = new ActiveMQQueue("NOT.EXIST"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{activeMQDestination}); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_DELETE) + .setEventData(eventSerializer.serializeReplicationData(testQueue)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker, never()).removeDestination(connectionContext, testQueue, 1000); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker).getAdminConnectionContext(); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(2)).send(any(), messageArgumentCaptor.capture()); + + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0).getMessageId()).isEqualTo(message.getMessageId()); + assertThat(values.get(1).getDestination()).isEqualTo(sequenceQueue); + + verifyConnectionContext(); + } + + @Test + public void canHandleEventOfType_MESSAGE_ACK_forQueue() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1:1:1"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{testQueue}); + + MessageAck ack = new MessageAck(); + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + ack.setConsumerId(consumerId); + ack.setDestination(testQueue); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, Collections.singletonList(messageId.toString())); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor ciArgumentCaptor = ArgumentCaptor.forClass(ConsumerInfo.class); + verify(broker, times(2)).addConsumer(any(), ciArgumentCaptor.capture()); + List consumerInfos = ciArgumentCaptor.getAllValues(); + assertThat(consumerInfos.get(0).getDestination()).isEqualTo(sequenceQueue); + assertThat(consumerInfos.get(1).getConsumerId()).isEqualTo(consumerId); + assertThat(consumerInfos.get(1).getDestination()).isEqualTo(testQueue); + + + ArgumentCaptor mdnArgumentCaptor = ArgumentCaptor.forClass(MessageDispatchNotification.class); + verify(broker).processDispatchNotification(mdnArgumentCaptor.capture()); + + MessageDispatchNotification mdn = mdnArgumentCaptor.getValue(); + assertThat(mdn.getMessageId()).isEqualTo(messageId); + assertThat(mdn.getDestination()).isEqualTo(testQueue); + assertThat(mdn.getConsumerId()).isEqualTo(consumerId); + + ArgumentCaptor ackArgumentCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker).acknowledge(any(), ackArgumentCaptor.capture()); + + MessageAck value = ackArgumentCaptor.getValue(); + assertThat(value.getDestination()).isEqualTo(testQueue); + assertThat(value.getConsumerId()).isEqualTo(consumerId); + } + + @Test + public void canHandleEventOfType_QUEUE_PURGED() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1:1:1"); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.QUEUE_PURGED) + .setEventData(eventSerializer.serializeReplicationData(testQueue)); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + verify((Queue) destinationQueue).purge(any()); + } + + @Test + public void canHandleEventOfType_TRANSACTION_BEGIN() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_BEGIN) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(TransactionId.class); + verify(broker, times(2)).beginTransaction(any(), messageArgumentCaptor.capture()); + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0)).isNotEqualTo(transactionId); + assertThat(values.get(1)).isEqualTo(transactionId); + } + + @Test + public void canHandleEventOfType_TRANSACTION_PREPARE() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_PREPARE) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(TransactionId.class); + verify(broker).prepareTransaction(any(), messageArgumentCaptor.capture()); + TransactionId value = messageArgumentCaptor.getValue(); + assertThat(value).isEqualTo(transactionId); + } + + @Test + public void canHandleEventOfType_TRANSACTION_FORGET() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_FORGET) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(TransactionId.class); + verify(broker).forgetTransaction(any(), messageArgumentCaptor.capture()); + TransactionId value = messageArgumentCaptor.getValue(); + assertThat(value).isEqualTo(transactionId); + } + + @Test + public void canHandleEventOfType_TRANSACTION_ROLLBACK() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_ROLLBACK) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(TransactionId.class); + verify(broker).rollbackTransaction(any(), messageArgumentCaptor.capture()); + TransactionId value = messageArgumentCaptor.getValue(); + assertThat(value).isEqualTo(transactionId); + } + + @Test + public void canHandleEventOfType_TRANSACTION_COMMIT() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_COMMIT) + .setEventData(eventSerializer.serializeReplicationData(transactionId)) + .setReplicationProperty(ReplicaSupport.TRANSACTION_ONE_PHASE_PROPERTY, true); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(TransactionId.class); + ArgumentCaptor onePhaseArgumentCaptor = ArgumentCaptor.forClass(Boolean.class); + verify(broker, times(2)).commitTransaction(any(), messageArgumentCaptor.capture(), onePhaseArgumentCaptor.capture()); + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0)).isEqualTo(transactionId); + assertThat(values.get(1)).isNotEqualTo(transactionId); + List onePhaseValues = onePhaseArgumentCaptor.getAllValues(); + assertThat(onePhaseValues.get(0)).isTrue(); + assertThat(onePhaseValues.get(1)).isTrue(); + } + + @Test + public void canHandleEventOfType_TRANSACTION_PREPARE_whenXATransactionNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new XATransactionId(getDummyXid()); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_PREPARE) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + when(transactionBroker.getTransaction(any(), any(), anyBoolean())).thenThrow(new XAException("")); + + listener.onMessage(replicaEventMessage); + verify(broker, never()).prepareTransaction(any(), any()); + } + + @Test + public void canHandleEventOfType_TRANSACTION_FORGET_whenXATransactionNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new XATransactionId(getDummyXid()); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_FORGET) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + when(transactionBroker.getTransaction(any(), any(), anyBoolean())).thenThrow(new XAException("")); + + listener.onMessage(replicaEventMessage); + verify(broker, never()).forgetTransaction(any(), any()); + } + + @Test + public void canHandleEventOfType_TRANSACTION_COMMIT_whenXATransactionNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new XATransactionId(getDummyXid()); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_COMMIT) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + when(transactionBroker.getTransaction(any(), any(), anyBoolean())).thenThrow(new XAException("")); + + listener.onMessage(replicaEventMessage); + verify(broker, times(1)).commitTransaction(any(), any(), anyBoolean()); + } + + @Test + public void canHandleEventOfType_TRANSACTION_ROLLBACK_whenXATransactionNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + TransactionId transactionId = new XATransactionId(getDummyXid()); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.TRANSACTION_ROLLBACK) + .setEventData(eventSerializer.serializeReplicationData(transactionId)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + when(transactionBroker.getTransaction(any(), any(), anyBoolean())).thenThrow(new XAException("")); + + listener.onMessage(replicaEventMessage); + verify(broker, never()).rollbackTransaction(any(), any()); + } + + @Test + public void canHandleEventOfType_ADD_DURABLE_CONSUMER() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(testQueue); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + String clientId = "clientId"; + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.ADD_DURABLE_CONSUMER) + .setEventData(eventSerializer.serializeReplicationData(consumerInfo)) + .setReplicationProperty(ReplicaSupport.CLIENT_ID_PROPERTY, clientId); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + DurableTopicSubscription subscription = mock(DurableTopicSubscription.class); + when(broker.addConsumer(any(), any())).thenReturn(subscription); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ConsumerInfo.class); + ArgumentCaptor connectionContextArgumentCaptor = ArgumentCaptor.forClass(ConnectionContext.class); + verify(broker, times(2)).addConsumer(connectionContextArgumentCaptor.capture(), messageArgumentCaptor.capture()); + List consumerInfos = messageArgumentCaptor.getAllValues(); + assertThat(consumerInfos.get(0).getDestination()).isEqualTo(sequenceQueue); + assertThat(consumerInfos.get(1).getDestination()).isEqualTo(testQueue); + ConnectionContext connectionContext = connectionContextArgumentCaptor.getValue(); + assertThat(connectionContext.getClientId()).isEqualTo(clientId); + verify(subscription).deactivate(true, 0); + } + + @Test + public void canHandleEventOfType_REMOVE_DURABLE_CONSUMER() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(testQueue); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + String clientId = "clientId"; + consumerInfo.setClientId(clientId); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.REMOVE_DURABLE_CONSUMER) + .setEventData(eventSerializer.serializeReplicationData(consumerInfo)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + DurableTopicSubscription subscription = mock(DurableTopicSubscription.class); + when(destinationQueue.getConsumers()).thenReturn(Collections.singletonList(subscription)); + when(subscription.getConsumerInfo()).thenReturn(consumerInfo); + when(subscription.getContext()).thenReturn(connectionContext); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ConsumerInfo.class); + verify(broker).removeConsumer(any(), messageArgumentCaptor.capture()); + ConsumerInfo value = messageArgumentCaptor.getValue(); + assertThat(value.getDestination()).isEqualTo(testQueue); + } + + @Test + public void canHandleEventOfType_MESSAGE_EXPIRED() throws Exception { + listener.sequence = null; + + MessageId messageId = new MessageId("1:0:0:1"); + ActiveMQMessage message = new ActiveMQMessage(); + message.setDestination(testQueue); + message.setMessageId(messageId); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_EXPIRED) + .setEventData(eventSerializer.serializeReplicationData(message)); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + + listener.onMessage(replicaEventMessage); + + verify((Queue) destinationQueue).messageExpired(any(), any(), any()); + } + + @Test + public void canHandleEventOfType_MESSAGE_ACK_forTopic() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1:1:1"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{testTopic}); + + MessageAck ack = new MessageAck(); + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + ack.setConsumerId(consumerId); + ack.setDestination(testTopic); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, Collections.singletonList(messageId.toString())); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor ciArgumentCaptor = ArgumentCaptor.forClass(ConsumerInfo.class); + verify(broker).addConsumer(any(), ciArgumentCaptor.capture()); + ConsumerInfo consumerInfo = ciArgumentCaptor.getValue(); + assertThat(consumerInfo.getDestination()).isEqualTo(sequenceQueue); + + ArgumentCaptor mdnArgumentCaptor = ArgumentCaptor.forClass(MessageDispatchNotification.class); + verify(broker).processDispatchNotification(mdnArgumentCaptor.capture()); + + MessageDispatchNotification mdn = mdnArgumentCaptor.getValue(); + assertThat(mdn.getMessageId()).isEqualTo(messageId); + assertThat(mdn.getDestination()).isEqualTo(testTopic); + assertThat(mdn.getConsumerId()).isEqualTo(consumerId); + + ArgumentCaptor ackArgumentCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker).acknowledge(any(), ackArgumentCaptor.capture()); + + MessageAck value = ackArgumentCaptor.getValue(); + assertThat(value.getDestination()).isEqualTo(ack.getDestination()); + assertThat(value.getConsumerId()).isEqualTo(ack.getConsumerId()); + } + + @Test + public void canHandleEventOfType_MESSAGE_ACK_whenMessageNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{testQueue}); + + MessageAck ack = new MessageAck(); + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + ack.setConsumerId(consumerId); + ack.setDestination(testQueue); + + doThrow(new JMSException("Slave broker out of sync with master - Message: " + " does not exist among pending(")).when(broker).processDispatchNotification(any(MessageDispatchNotification.class)); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, Collections.singletonList(messageId.toString()));; + + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + verify(broker, never()).acknowledge(any(), any()); + } + + @Test + public void canHandleEventOfType_MESSAGE_ACK_whenDestinationNotExist() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + + ActiveMQDestination activeMQDestination = new ActiveMQQueue("NOT.EXIST"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{activeMQDestination}); + MessageAck ack = new MessageAck(); + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + ack.setConsumerId(consumerId); + ack.setDestination(testQueue); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, Collections.singletonList(messageId.toString()));; + + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + verify(broker, never()).acknowledge(any(), any()); + } + + @Test + public void canHandleEventOfType_BATCH() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + when(broker.getDestinations()).thenReturn(new ActiveMQDestination[]{testQueue}); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage sendEventMessage = spy(new ActiveMQMessage()); + ReplicaEvent sendEvent = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + sendEventMessage.setContent(sendEvent.getEventData()); + sendEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, sendEvent.getEventType().name()); + sendEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + sendEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + MessageAck ack = new MessageAck(); + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + ack.setConsumerId(consumerId); + ack.setDestination(testQueue); + + ReplicaEvent ackEvent = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_ACK) + .setEventData(eventSerializer.serializeReplicationData(ack)) + .setReplicationProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, Collections.singletonList(messageId.toString())); + ActiveMQMessage ackEventMessage = spy(new ActiveMQMessage()); + ackEventMessage.setType("ReplicaEvent"); + ackEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ackEvent.getEventType().name()); + ackEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "1"); + ackEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + ackEventMessage.setContent(ackEvent.getEventData()); + ackEventMessage.setProperties(ackEvent.getReplicationProperties()); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.BATCH) + .setEventData(eventSerializer.serializeListOfObjects(List.of(sendEventMessage, ackEventMessage))); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + + listener.onMessage(replicaEventMessage); + + verify(broker).getAdminConnectionContext(); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(2)).send(any(), messageArgumentCaptor.capture()); + + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0).getMessageId()).isEqualTo(message.getMessageId()); + assertThat(values.get(1).getDestination()).isEqualTo(sequenceQueue); + + verifyConnectionContext(); + + ArgumentCaptor ciArgumentCaptor = ArgumentCaptor.forClass(ConsumerInfo.class); + verify(broker, times(2)).addConsumer(any(), ciArgumentCaptor.capture()); + List consumerInfos = ciArgumentCaptor.getAllValues(); + assertThat(consumerInfos.get(0).getDestination()).isEqualTo(sequenceQueue); + assertThat(consumerInfos.get(1).getConsumerId()).isEqualTo(consumerId); + assertThat(consumerInfos.get(1).getDestination()).isEqualTo(testQueue); + + + ArgumentCaptor mdnArgumentCaptor = ArgumentCaptor.forClass(MessageDispatchNotification.class); + verify(broker).processDispatchNotification(mdnArgumentCaptor.capture()); + + MessageDispatchNotification mdn = mdnArgumentCaptor.getValue(); + assertThat(mdn.getMessageId()).isEqualTo(messageId); + assertThat(mdn.getDestination()).isEqualTo(testQueue); + assertThat(mdn.getConsumerId()).isEqualTo(consumerId); + + ArgumentCaptor ackArgumentCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker).acknowledge(any(), ackArgumentCaptor.capture()); + + MessageAck ackValue = ackArgumentCaptor.getValue(); + assertThat(ackValue.getDestination()).isEqualTo(testQueue); + assertThat(ackValue.getConsumerId()).isEqualTo(consumerId); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND_correctSequence() throws Exception { + listener.sequence = BigInteger.ZERO; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "1"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + verify(broker).getAdminConnectionContext(); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(2)).send(any(), messageArgumentCaptor.capture()); + + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0).getMessageId()).isEqualTo(message.getMessageId()); + assertThat(values.get(1).getDestination()).isEqualTo(sequenceQueue); + + verifyConnectionContext(); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND_sequenceIsLowerThanCurrent() throws Exception { + listener.sequence = BigInteger.ONE; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + listener.onMessage(replicaEventMessage); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage value = messageArgumentCaptor.getValue(); + assertThat(value.getDestination()).isEqualTo(sequenceQueue); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND_incorrectSequence() throws Exception { + listener.sequence = BigInteger.ZERO; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "2"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + CountDownLatch cdl = new CountDownLatch(1); + Thread thread = new Thread(() -> { + listener.onMessage(replicaEventMessage); + cdl.countDown(); + }); + thread.start(); + + assertThat(cdl.await(2, TimeUnit.SECONDS)).isFalse(); + + thread.interrupt(); + + verify(broker, never()).send(any(), any()); + + verify(replicaEventMessage, never()).acknowledge(); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND_olderVersion() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION - 1); + + listener.onMessage(replicaEventMessage); + + verify(broker).getAdminConnectionContext(); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(2)).send(any(), messageArgumentCaptor.capture()); + + List values = messageArgumentCaptor.getAllValues(); + assertThat(values.get(0).getMessageId()).isEqualTo(message.getMessageId()); + assertThat(values.get(1).getDestination()).isEqualTo(sequenceQueue); + + verifyConnectionContext(); + } + + @Test + public void canHandleEventOfType_MESSAGE_SEND_newerVersion() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION + 1); + + CountDownLatch cdl = new CountDownLatch(1); + Thread thread = new Thread(() -> { + listener.onMessage(replicaEventMessage); + cdl.countDown(); + }); + thread.start(); + + assertThat(cdl.await(2, TimeUnit.SECONDS)).isFalse(); + + thread.interrupt(); + + verify(broker, never()).send(any(), any()); + + verify(replicaEventMessage, never()).acknowledge(); + } + + private void verifyConnectionContext() { + verify(connectionContext, times(2)).isProducerFlowControl(); + verify(connectionContext, times(2)).setProducerFlowControl(true); + verify(connectionContext, times(2)).setProducerFlowControl(false); + } + + + @Test + public void canHandleEventOfType_FAIL_OVER() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1:1:1"); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.FAIL_OVER) + .setEventData(eventSerializer.serializeReplicationData(null)); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + + verify(replicaBroker).updateBrokerState(eq(ReplicaRole.source)); + verify(replicaBroker).completeBeforeRoleChange(); + } + + @Test + public void canHandleEventOfType_HEART_BEAT() throws Exception { + listener.sequence = null; + MessageId messageId = new MessageId("1:1:1:1"); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.HEART_BEAT) + .setEventData(eventSerializer.serializeReplicationData(null)); + ActiveMQMessage replicaEventMessage = spy(new ActiveMQMessage()); + replicaEventMessage.setMessageId(messageId); + replicaEventMessage.setType("ReplicaEvent"); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setProperties(event.getReplicationProperties()); + + listener.onMessage(replicaEventMessage); + } + + private Xid getDummyXid() { + return new Xid() { + @Override + public int getFormatId() { + return 1; + } + + @Override + public byte[] getGlobalTransactionId() { + return UUID.randomUUID().toString().getBytes(); + } + + @Override + public byte[] getBranchQualifier() { + return "branchQualifier".getBytes(StandardCharsets.UTF_8); + } + }; + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaCompactorTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaCompactorTest.java new file mode 100644 index 00000000000..afe59b3504e --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaCompactorTest.java @@ -0,0 +1,185 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.store.MessageStore; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaCompactorTest { + + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final Broker broker = mock(Broker.class); + private final ReplicaReplicationQueueSupplier queueProvider = mock(ReplicaReplicationQueueSupplier.class); + private final MessageStore messageStore = mock(MessageStore.class); + + private final ActiveMQQueue intermediateQueueDestination = new ActiveMQQueue(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + private final Queue intermediateQueue = mock(Queue.class); + + private ReplicaCompactor replicaCompactor; + + @Before + public void setUp() throws Exception { + ConnectionContext adminConnectionContext = mock(ConnectionContext.class); + when(adminConnectionContext.copy()).thenReturn(connectionContext); + when(broker.getAdminConnectionContext()).thenReturn(adminConnectionContext); + + when(queueProvider.getIntermediateQueue()).thenReturn(intermediateQueueDestination); + when(broker.getDestinations(intermediateQueueDestination)).thenReturn(Set.of(intermediateQueue)); + when(intermediateQueue.getMessageStore()).thenReturn(messageStore); + + ConsumerInfo consumerInfo = new ConsumerInfo(); + PrefetchSubscription originalSubscription = mock(PrefetchSubscription.class); + when(originalSubscription.getConsumerInfo()).thenReturn(consumerInfo); + + replicaCompactor = new ReplicaCompactor(broker, queueProvider, originalSubscription, 1000, new ReplicaStatistics()); + } + + @Test + public void compactWhenSendAndAck() throws Exception { + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + + String messageIdToAck = "2:1"; + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message1.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + message3.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageIdToAck)); + + List result = replicaCompactor.compactAndFilter(connectionContext, List.of(message1, message2, message3), false); + + assertThat(result.size()).isEqualTo(1); + assertThat(result.get(0).getMessageId()).isEqualTo(messageId2); + + verify(broker).beginTransaction(any(), any()); + + ArgumentCaptor ackCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker, times(2)).acknowledge(any(), ackCaptor.capture()); + + List values = ackCaptor.getAllValues(); + MessageAck messageAck = values.get(0); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId1); + messageAck = values.get(1); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId3); + + verify(broker).commitTransaction(any(), any(), eq(true)); + } + + @Test + public void compactWhenMultipleSendsAndAcksWithSameId() throws Exception { + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + MessageId messageId4 = new MessageId("1:0:0:4"); + MessageId messageId5 = new MessageId("1:0:0:5"); + MessageId messageId6 = new MessageId("1:0:0:6"); + + String messageIdToAck1 = "2:1"; + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message1.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck1); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + message3.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageIdToAck1)); + ActiveMQMessage message4 = new ActiveMQMessage(); + message4.setMessageId(messageId4); + message4.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message4.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message4.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck1); + ActiveMQMessage message5 = new ActiveMQMessage(); + message5.setMessageId(messageId5); + message5.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message5.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + message5.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageIdToAck1)); + ActiveMQMessage message6 = new ActiveMQMessage(); + message6.setMessageId(messageId6); + message6.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message6.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message6.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck1); + + List result = replicaCompactor.compactAndFilter(connectionContext, + List.of(message1, message2, message3, message4, message5, message6), false); + + assertThat(result.size()).isEqualTo(2); + assertThat(result.get(0).getMessageId()).isEqualTo(messageId2); + assertThat(result.get(1).getMessageId()).isEqualTo(messageId6); + + verify(broker).beginTransaction(any(), any()); + + ArgumentCaptor ackCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker, times(4)).acknowledge(any(), ackCaptor.capture()); + + List values = ackCaptor.getAllValues(); + MessageAck messageAck = values.get(0); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId1); + messageAck = values.get(1); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId3); + + verify(broker).commitTransaction(any(), any(), eq(true)); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaEventSerializerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaEventSerializerTest.java new file mode 100644 index 00000000000..b99c9bd269b --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaEventSerializerTest.java @@ -0,0 +1,283 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.DataStructure; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.util.ByteSequence; +import org.junit.Ignore; +import org.junit.Test; +import org.mockito.Mockito; + +import java.io.IOException; +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assert.fail; + +public class ReplicaEventSerializerTest { + + private final ReplicaEventSerializer serializer = new ReplicaEventSerializer(); + + @Test + public void serializeListOfObjectsTest() throws Exception { + MessageId messageId1 = new MessageId("1:1:1:1"); + ActiveMQTextMessage message1 = new ActiveMQTextMessage(); + message1.setMessageId(messageId1); + String text1 = "testtesttesttesttesttesttest1"; + message1.setText(text1); + + MessageId messageId2 = new MessageId("2:2:2:2"); + ActiveMQTextMessage message2 = new ActiveMQTextMessage(); + message2.setMessageId(messageId2); + String text2 = "testtesttesttesttesttesttesttesttesttesttesttesttesttest2"; + message2.setText(text2); + + byte[] bytes = serializer.serializeListOfObjects(List.of(message1, message2)); + + List objects = serializer.deserializeListOfObjects(bytes); + System.out.println(objects); + assertThat(objects.size()).isEqualTo(2); + Object o1 = objects.get(0); + Object o2 = objects.get(1); + assertThat(o1).isInstanceOf(ActiveMQTextMessage.class); + assertThat(o2).isInstanceOf(ActiveMQTextMessage.class); + ActiveMQTextMessage m1 = (ActiveMQTextMessage) o1; + ActiveMQTextMessage m2 = (ActiveMQTextMessage) o2; + assertThat(m1.getMessageId()).isEqualTo(messageId1); + assertThat(m2.getMessageId()).isEqualTo(messageId2); + assertThat(m1.getText()).isEqualTo(text1); + assertThat(m2.getText()).isEqualTo(text2); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_DESTINATION_UPSERT() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_DESTINATION_DELETE() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_MESSAGE_SEND() throws IOException { + var message = new ActiveMQMessage(); + fail("Need correct data for test"); + + var bytes = serializer.serializeMessageData(message); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(ActiveMQMessage.class) + .isEqualTo(message); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_MESSAGE_ACK() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_MESSAGE_CONSUMED() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_MESSAGE_DISCARDED() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_TRANSACTION_BEGIN() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_TRANSACTION_PREPARE() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_TRANSACTION_ROLLBACK() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_TRANSACTION_COMMIT() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_TRANSACTION_FORGET() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_MESSAGE_EXPIRED() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_SUBSCRIBER_REMOVED() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + @Test + @Ignore + public void canDoRoundTripSerializedForDataOf_SUBSCRIBER_ADDED() throws IOException { + var object = Mockito.mock(DataStructure.class); + var expectedClass = ActiveMQDestination.class; + fail("Need correct object for test"); + + var bytes = serializer.serializeReplicationData(object); + var deserialized = serializer.deserializeMessageData(asSequence(bytes)); + + assertThat(bytes).isNotNull(); + assertThat(deserialized).isInstanceOf(expectedClass) + .isEqualTo(object); + } + + private ByteSequence asSequence(byte[] bytes) { + return new ByteSequence(bytes); + } + +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaInternalMessageProducerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaInternalMessageProducerTest.java new file mode 100644 index 00000000000..28a4541b26f --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaInternalMessageProducerTest.java @@ -0,0 +1,65 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.MessageId; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaInternalMessageProducerTest { + + private final Broker broker = mock(Broker.class); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + + ReplicaInternalMessageProducer producer = new ReplicaInternalMessageProducer(broker); + + @Before + public void setUp() { + when(connectionContext.isProducerFlowControl()).thenReturn(false); + } + + @Test + public void sendsMessageForcingFlowControl() throws Exception { + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + producer.sendForcingFlowControl(connectionContext, message); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + + ActiveMQMessage value = messageArgumentCaptor.getValue(); + assertThat(value).isEqualTo(message); + + verify(connectionContext).isProducerFlowControl(); + verify(connectionContext).setProducerFlowControl(true); + verify(connectionContext).setProducerFlowControl(false); + } + +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginInstallationTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginInstallationTest.java new file mode 100644 index 00000000000..c2ac18ebf0e --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginInstallationTest.java @@ -0,0 +1,85 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerFilter; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.broker.region.policy.PolicyMap; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.usage.SystemUsage; +import org.junit.Before; +import org.junit.Test; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + + +public class ReplicaPluginInstallationTest { + + private final BrokerService brokerService = mock(BrokerService.class); + private final Broker broker = mock(Broker.class); + private final ReplicaPlugin pluginUnderTest = new ReplicaPlugin(); + + @Before + public void setUp() { + pluginUnderTest.setControlWebConsoleAccess(false); + when(broker.getBrokerService()).thenReturn(brokerService); + when(brokerService.isUseJmx()).thenReturn(false); + when(brokerService.getDestinationPolicy()).thenReturn(new PolicyMap()); + + RegionBroker regionBroker = mock(RegionBroker.class); + when(broker.getAdaptor(RegionBroker.class)).thenReturn(regionBroker); + CompositeDestinationInterceptor cdi = mock(CompositeDestinationInterceptor.class); + when(regionBroker.getDestinationInterceptor()).thenReturn(cdi); + when(cdi.getInterceptors()).thenReturn(new DestinationInterceptor[]{}); + + SystemUsage systemUsage = mock(SystemUsage.class); + when(brokerService.getSystemUsage()).thenReturn(systemUsage); + MemoryUsage memoryUsage = mock(MemoryUsage.class); + when(systemUsage.getMemoryUsage()).thenReturn(memoryUsage); + } + + @Test + public void testInstallPluginWithDefaultRole() throws Exception { + pluginUnderTest.setTransportConnectorUri("failover:(tcp://localhost:61616)"); + Broker installedBroker = pluginUnderTest.installPlugin(broker); + assertThat(installedBroker).isInstanceOf(ReplicaAuthorizationBroker.class); + Broker nextBroker = ((BrokerFilter) installedBroker).getNext(); + assertThat(nextBroker).isInstanceOf(ReplicaRoleManagementBroker.class); + assertThat(((BrokerFilter) nextBroker).getNext()).isEqualTo(broker); + assertThat(ReplicaRole.source).isEqualTo(pluginUnderTest.getRole()); + } + + @Test + public void testInstallPluginWithReplicaRole() throws Exception { + pluginUnderTest.setRole(ReplicaRole.replica); + pluginUnderTest.setOtherBrokerUri("failover:(tcp://localhost:61616)"); + Broker installedBroker = pluginUnderTest.installPlugin(broker); + assertThat(installedBroker).isInstanceOf(ReplicaAuthorizationBroker.class); + Broker nextBroker = ((BrokerFilter) installedBroker).getNext(); + assertThat(nextBroker).isInstanceOf(ReplicaRoleManagementBroker.class); + + assertThat(((BrokerFilter) nextBroker).getNext()).isEqualTo(broker); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginTest.java new file mode 100644 index 00000000000..5454413ee9a --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPluginTest.java @@ -0,0 +1,197 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.broker.region.policy.PolicyMap; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.usage.SystemUsage; +import org.assertj.core.api.SoftAssertions; +import org.junit.Before; +import org.junit.Test; + +import java.net.URI; +import java.util.Arrays; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.catchThrowable; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; + +public class ReplicaPluginTest { + + private final ReplicaPlugin plugin = new ReplicaPlugin(); + + @Before + public void setUp() { + plugin.setControlWebConsoleAccess(false); + } + + @Test + public void canSetRole() { + SoftAssertions softly = new SoftAssertions(); + Arrays.stream(ReplicaRole.values()).forEach(role -> { + + softly.assertThat(plugin.setRole(role)).isSameAs(plugin); + softly.assertThat(plugin.role).isEqualTo(role); + + plugin.setRole(role.name()); + softly.assertThat(plugin.role).isEqualTo(role); + }); + softly.assertAll(); + } + + @Test + public void rejectsUnknownRole() { + Throwable exception = catchThrowable(() -> plugin.setRole("unknown")); + + assertThat(exception).isExactlyInstanceOf(IllegalArgumentException.class) + .hasMessage("unknown is not a known " + ReplicaRole.class.getSimpleName()); + } + + @Test + public void canSetOtherBrokerUri() { + plugin.setOtherBrokerUri("failover:(tcp://localhost:61616)"); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory()).isNotNull() + .extracting(ActiveMQConnectionFactory::getBrokerURL) + .isEqualTo("failover:(tcp://localhost:61616)"); + } + + @Test + public void canSetOtherBrokerUriFluently() { + ReplicaPlugin result = plugin.connectedTo(URI.create("failover:(tcp://localhost:61616)")); + + assertThat(result).isSameAs(plugin); + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory()).isNotNull() + .extracting(ActiveMQConnectionFactory::getBrokerURL) + .isEqualTo("failover:(tcp://localhost:61616)"); + } + + @Test + public void rejectsInvalidUnknownOtherBrokerUri() { + Throwable expected = catchThrowable(() -> new ActiveMQConnectionFactory().setBrokerURL("inval:{id}-uri")); + + Throwable exception = catchThrowable(() -> plugin.setOtherBrokerUri("inval:{id}-uri")); + + assertThat(exception).isNotNull().isEqualToComparingFieldByField(expected); + } + + @Test + public void canSetOtherBrokerUriWithAutomaticAdditionOfFailoverTransport() { + plugin.setOtherBrokerUri("tcp://localhost:61616"); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory()).isNotNull() + .extracting(ActiveMQConnectionFactory::getBrokerURL) + .isEqualTo("failover:(tcp://localhost:61616)"); + } + + @Test + public void canSetTransportConnectorUri() { + plugin.setTransportConnectorUri("tcp://0.0.0.0:61618?maximumConnections=1&wireFormat.maxFrameSize=104857600"); + + assertThat(plugin.replicaPolicy.getTransportConnectorUri()).isNotNull() + .isEqualTo(URI.create("tcp://0.0.0.0:61618?maximumConnections=1&wireFormat.maxFrameSize=104857600")); + } + + @Test + public void rejectsInvalidTransportConnectorUri() { + Throwable expected = catchThrowable(() -> URI.create("inval:{id}-uri")); + + Throwable exception = catchThrowable(() -> plugin.setTransportConnectorUri("inval:{id}-uri")); + + assertThat(exception).isNotNull().isEqualToComparingFieldByField(expected); + } + + @Test + public void canSetUserNameAndPassword() { + String userUsername = "testUser"; + String password = "testPassword"; + + plugin.setUserName(userUsername); + plugin.setPassword(password); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getUserName()).isEqualTo(userUsername); + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getPassword()).isEqualTo(password); + } + + @Test(expected = NullPointerException.class) + public void shouldThrowExceptionIfUserIsSetAndPasswordIsNotForReplica() throws Exception { + String userName = "testUser"; + Broker broker = mock(Broker.class); + String replicationTransport = "tcp://localhost:61616"; + + plugin.setRole(ReplicaRole.replica); + plugin.setUserName(userName); + plugin.setTransportConnectorUri(replicationTransport); + plugin.installPlugin(broker); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getUserName()).isEqualTo(userName); + } + + @Test(expected = NullPointerException.class) + public void shouldThrowExceptionIfPasswordIsSetAndUserNameIsNotForReplica() throws Exception { + String password = "testPassword"; + Broker broker = mock(Broker.class); + String replicationTransport = "tcp://localhost:61616"; + + plugin.setRole(ReplicaRole.replica); + plugin.setPassword(password); + plugin.setTransportConnectorUri(replicationTransport); + plugin.installPlugin(broker); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getPassword()).isEqualTo(password); + } + + @Test + public void shouldNotThrowExceptionIfBothUserAndPasswordIsSetForReplica() throws Exception { + String user = "testUser"; + String password = "testPassword"; + Broker broker = mock(Broker.class); + BrokerService brokerService = mock(BrokerService.class); + when(brokerService.getDestinationPolicy()).thenReturn(new PolicyMap()); + when(broker.getBrokerService()).thenReturn(brokerService); + when(brokerService.isUseJmx()).thenReturn(false); + SystemUsage systemUsage = mock(SystemUsage.class); + when(brokerService.getSystemUsage()).thenReturn(systemUsage); + MemoryUsage memoryUsage = mock(MemoryUsage.class); + when(systemUsage.getMemoryUsage()).thenReturn(memoryUsage); + String replicationTransport = "tcp://localhost:61616"; + + RegionBroker regionBroker = mock(RegionBroker.class); + when(broker.getAdaptor(RegionBroker.class)).thenReturn(regionBroker); + CompositeDestinationInterceptor cdi = mock(CompositeDestinationInterceptor.class); + when(regionBroker.getDestinationInterceptor()).thenReturn(cdi); + when(cdi.getInterceptors()).thenReturn(new DestinationInterceptor[]{}); + + plugin.setRole(ReplicaRole.replica); + plugin.setPassword(password); + plugin.setUserName(user); + plugin.setTransportConnectorUri(replicationTransport); + plugin.installPlugin(broker); + + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getUserName()).isEqualTo(user); + assertThat(plugin.replicaPolicy.getOtherBrokerConnectionFactory().getPassword()).isEqualTo(password); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPolicyTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPolicyTest.java new file mode 100644 index 00000000000..5b56550802c --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaPolicyTest.java @@ -0,0 +1,43 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.junit.Test; + +import java.net.URI; + +import static junit.framework.TestCase.assertEquals; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.Assert.assertThrows; + +public class ReplicaPolicyTest { + + @Test + public void testGetTransportConnectorUriNotSet() { + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + Throwable exception = assertThrows(NullPointerException.class, replicaPolicy::getTransportConnectorUri); + assertEquals("Need replication transport connection URI for this broker", exception.getMessage()); + } + + @Test + public void testGetTransportConnectorUriSet() throws Exception { + URI uri = new URI("localhost:8080"); + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + replicaPolicy.setTransportConnectorUri(uri); + assertThat(replicaPolicy.getTransportConnectorUri()).isEqualTo(uri); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplierTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplierTest.java new file mode 100644 index 00000000000..3e9cd3ecba7 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaReplicationQueueSupplierTest.java @@ -0,0 +1,80 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.command.ActiveMQQueue; +import org.junit.Before; +import org.junit.Test; + +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaReplicationQueueSupplierTest { + + private final Broker broker = mock(Broker.class); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final BrokerService brokerService = mock(BrokerService.class); + + private final ReplicaReplicationQueueSupplier supplier = new ReplicaReplicationQueueSupplier(broker); + + @Before + public void setUp() throws Exception { + when(broker.getAdminConnectionContext()).thenReturn(connectionContext); + when(broker.getBrokerService()).thenReturn(brokerService); + when(brokerService.getBroker()).thenReturn(broker); + } + + @Test + public void canCreateQueue() throws Exception { + supplier.initialize(); + + ActiveMQQueue activeMQQueue = supplier.getMainQueue(); + assertThat(activeMQQueue.getPhysicalName()).isEqualTo(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + + verify(broker).addDestination(eq(connectionContext), eq(activeMQQueue), eq(false)); + } + + @Test + public void notCreateQueueIfExists() throws Exception { + ActiveMQQueue mainReplicationQueue = new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + ActiveMQQueue intermediateReplicationQueue = new ActiveMQQueue(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + + when(broker.getDurableDestinations()).thenReturn(Set.of(mainReplicationQueue, intermediateReplicationQueue)); + + supplier.initialize(); + + ActiveMQQueue activeMQQueue = supplier.getMainQueue(); + assertThat(activeMQQueue).isEqualTo(mainReplicationQueue); + + activeMQQueue = supplier.getIntermediateQueue(); + assertThat(activeMQQueue).isEqualTo(intermediateReplicationQueue); + + verify(broker, never()).addDestination(any(), any(), anyBoolean()); + } + +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaRoleManagementBrokerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaRoleManagementBrokerTest.java new file mode 100644 index 00000000000..4bd863e6ac7 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaRoleManagementBrokerTest.java @@ -0,0 +1,228 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.usage.SystemUsage; +import org.junit.Before; +import org.junit.Test; + +import java.net.URI; +import java.util.List; +import java.util.Set; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaRoleManagementBrokerTest { + + private ReplicaRoleManagementBroker replicaRoleManagementBroker; + private final Broker broker = mock(Broker.class); + private final ReplicaBroker replicaBroker = mock(ReplicaBroker.class); + private final ReplicaSourceBroker sourceBroker = mock(ReplicaSourceBroker.class); + private final BrokerService brokerService = mock(BrokerService.class); + private final PrefetchSubscription subscription = mock(PrefetchSubscription.class); + + @Before + public void setUp() throws Exception { + when(broker.getAdminConnectionContext()).thenReturn(new ConnectionContext()); + when(broker.getBrokerService()).thenReturn(brokerService); + when(brokerService.getBroker()).thenReturn(broker); + when(broker.getDurableDestinations()).thenReturn(Set.of(new ActiveMQQueue(ReplicaSupport.REPLICATION_ROLE_QUEUE_NAME))); + when(broker.getDestinations(any())).thenReturn(Set.of(mock(Queue.class))); + when(broker.addConsumer(any(), any())).thenReturn(subscription); + when(brokerService.addConnector(any(URI.class))).thenReturn(mock(TransportConnector.class)); + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + replicaPolicy.setControlWebConsoleAccess(false); + replicaPolicy.setTransportConnectorUri(new URI("tcp://localhost:61617")); + SystemUsage systemUsage = mock(SystemUsage.class); + when(brokerService.getSystemUsage()).thenReturn(systemUsage); + MemoryUsage memoryUsage = mock(MemoryUsage.class); + when(systemUsage.getMemoryUsage()).thenReturn(memoryUsage); + + RegionBroker regionBroker = mock(RegionBroker.class); + when(broker.getAdaptor(RegionBroker.class)).thenReturn(regionBroker); + CompositeDestinationInterceptor cdi = mock(CompositeDestinationInterceptor.class); + when(regionBroker.getDestinationInterceptor()).thenReturn(cdi); + when(cdi.getInterceptors()).thenReturn(new DestinationInterceptor[]{}); + + replicaRoleManagementBroker = new ReplicaRoleManagementBroker(broker, replicaPolicy, ReplicaRole.replica, new ReplicaStatistics()); + replicaRoleManagementBroker.replicaBroker = replicaBroker; + replicaRoleManagementBroker.sourceBroker = sourceBroker; + } + + @Test + public void startAsSourceWhenBrokerFailOverStateIsSource() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.source.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + verify(sourceBroker).start(any()); + verify(replicaBroker, never()).start(); + } + + @Test + public void startAsReplicaWhenBrokerFailOverStateIsReplica() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.replica.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + verify(replicaBroker).start(any()); + verify(sourceBroker, never()).start(); + } + + @Test + public void startAsSourceWhenBrokerFailOverStateIsAwaitAck() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.await_ack.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + verify(sourceBroker).start(any()); + verify(replicaBroker, never()).start(); + } + + @Test + public void startAsReplicaWhenBrokerFailOverStateIsAckProcessed() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.ack_processed.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + verify(replicaBroker).start(any()); + verify(sourceBroker, never()).start(); + } + + @Test + public void switchToSourceWhenHardFailOverInvoked() throws Exception { + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.source, true); + + verify(replicaBroker).stopBeforeRoleChange(true); + } + + @Test + public void switchToReplicaWhenHardFailOverInvoked() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.source.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.replica, true); + + verify(sourceBroker).stopBeforeRoleChange(true); + } + + @Test + public void invokeSwitchToReplicaWhenSoftFailOverInvoked() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.source.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.replica, false); + + verify(sourceBroker).stopBeforeRoleChange(false); + } + + @Test + public void doNotInvokeSwitchToReplicaWhenAwaitAck() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.await_ack.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.replica, false); + + verify(sourceBroker, never()).stopBeforeRoleChange(anyBoolean()); + } + + @Test + public void doNotInvokeSwitchToReplicaWhenAckProcessed() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.ack_processed.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.source, false); + + verify(replicaBroker, never()).stopBeforeRoleChange(anyBoolean()); + } + + @Test + public void invokeSwitchToReplicaWhenAwaitAckAndForce() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.await_ack.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.replica, true); + + verify(sourceBroker).stopBeforeRoleChange(true); + } + + @Test + public void invokeSwitchToReplicaWhenAckProcessedAndForce() throws Exception { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(ReplicaRole.ack_processed.name()); + when(subscription.getDispatched()).thenReturn(List.of(message)); + + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.switchRole(ReplicaRole.source, true); + + verify(replicaBroker).stopBeforeRoleChange(true); + } + + @Test + public void completeSwitchToReplicaWhenSoftFailOverInvoked() throws Exception { + replicaRoleManagementBroker.start(); + + replicaRoleManagementBroker.onStopSuccess(); + + verify(replicaBroker).startAfterRoleChange(); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSequencerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSequencerTest.java new file mode 100644 index 00000000000..2bd4414a015 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSequencerTest.java @@ -0,0 +1,491 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.MessageReferenceFilter; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.QueueMessageReference; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.Message; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.thread.TaskRunner; +import org.apache.activemq.thread.TaskRunnerFactory; +import org.apache.activemq.usage.MemoryUsage; +import org.apache.activemq.usage.SystemUsage; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyInt; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaSequencerTest { + private static final Integer MAXIMUM_MESSAGES = new ReplicaPolicy().getCompactorAdditionalMessagesLimit(); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final Broker broker = mock(Broker.class); + private final ReplicaReplicationQueueSupplier queueProvider = mock(ReplicaReplicationQueueSupplier.class); + private final ReplicaInternalMessageProducer replicaInternalMessageProducer = mock(ReplicaInternalMessageProducer.class); + private final ReplicationMessageProducer replicationMessageProducer = mock(ReplicationMessageProducer.class); + + private ReplicaSequencer sequencer; + + private final ActiveMQQueue intermediateQueueDestination = new ActiveMQQueue(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + private final ActiveMQQueue mainQueueDestination = new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + private final ActiveMQQueue sequenceQueueDestination = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + private final Queue intermediateQueue = mock(Queue.class); + private final Queue mainQueue = mock(Queue.class); + private final Queue sequenceQueue = mock(Queue.class); + + private final ConsumerId consumerId = new ConsumerId("2:2:2:2"); + private final ConsumerInfo consumerInfo = new ConsumerInfo(consumerId); + private final PrefetchSubscription mainSubscription = mock(PrefetchSubscription.class); + private final PrefetchSubscription intermediateSubscription = mock(PrefetchSubscription.class); + + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + + @Before + public void setUp() throws Exception { + BrokerService brokerService = mock(BrokerService.class); + when(broker.getBrokerService()).thenReturn(brokerService); + + TaskRunnerFactory taskRunnerFactory = mock(TaskRunnerFactory.class); + when(brokerService.getTaskRunnerFactory()).thenReturn(taskRunnerFactory); + TaskRunner taskRunner = mock(TaskRunner.class); + when(taskRunnerFactory.createTaskRunner(any(), any())).thenReturn(taskRunner); + + when(queueProvider.getIntermediateQueue()).thenReturn(intermediateQueueDestination); + when(queueProvider.getMainQueue()).thenReturn(mainQueueDestination); + when(queueProvider.getSequenceQueue()).thenReturn(sequenceQueueDestination); + + when(broker.getDestinations(intermediateQueueDestination)).thenReturn(Set.of(intermediateQueue)); + when(broker.getDestinations(mainQueueDestination)).thenReturn(Set.of(mainQueue)); + when(broker.getDestinations(sequenceQueueDestination)).thenReturn(Set.of(sequenceQueue)); + + ConnectionContext adminConnectionContext = mock(ConnectionContext.class); + when(adminConnectionContext.copy()).thenReturn(connectionContext); + when(broker.getAdminConnectionContext()).thenReturn(adminConnectionContext); + + when(mainSubscription.getConsumerInfo()).thenReturn(consumerInfo); + when(mainQueue.getConsumers()).thenReturn(List.of(mainSubscription)); + + SystemUsage systemUsage = mock(SystemUsage.class); + when(brokerService.getSystemUsage()).thenReturn(systemUsage); + MemoryUsage memoryUsage = mock(MemoryUsage.class); + when(systemUsage.getMemoryUsage()).thenReturn(memoryUsage); + when(memoryUsage.waitForSpace(anyLong(), anyInt())).thenReturn(true); + + when(intermediateSubscription.getConsumerInfo()).thenReturn(consumerInfo); + when(broker.addConsumer(any(), any())) + .thenAnswer(a -> a.getArgument(1).getConsumerId().toString().contains("Sequencer") + ? intermediateSubscription : mock(PrefetchSubscription.class)); + + sequencer = new ReplicaSequencer(broker, queueProvider, replicaInternalMessageProducer, replicationMessageProducer, new ReplicaPolicy(), new ReplicaStatistics()); + sequencer.initialize(); + + } + + @Test + public void restoreSequenceWhenNoSequence() throws Exception { + sequencer.sequence = null; + + sequencer.restoreSequence(null, Collections.emptyList()); + + assertThat(sequencer.sequence).isNull(); + } + + @Test + public void restoreSequenceWhenSequenceExistsButNoRecoverySequences() throws Exception { + sequencer.sequence = null; + + MessageId messageId = new MessageId("1:0:0:1"); + sequencer.restoreSequence("1#" + messageId, Collections.emptyList()); + verify(replicationMessageProducer, never()).enqueueMainReplicaEvent(any(), any(ReplicaEvent.class)); + + assertThat(sequencer.sequence).isEqualTo(1); + + verify(replicationMessageProducer, never()).enqueueMainReplicaEvent(any(), any(ReplicaEvent.class)); + } + + @Test + public void restoreSequenceWhenStorageExistAndMessageDoesNotExist() throws Exception { + sequencer.sequence = null; + + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + MessageId messageId4 = new MessageId("1:0:0:4"); + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message4 = new ActiveMQMessage(); + message4.setMessageId(messageId4); + message4.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + + when(intermediateSubscription.getDispatched()).thenReturn(new ArrayList<>(List.of(message1, message2, message3, message4))); + + sequencer.restoreSequence("4#" + messageId4, List.of("1#" + messageId1 + "#" + messageId2, "3#" + messageId3 + "#" + messageId4)); + + assertThat(sequencer.sequence).isEqualTo(4); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(ReplicaEvent.class); + verify(replicationMessageProducer, times(2)).enqueueMainReplicaEvent(any(), argumentCaptor.capture()); + + List values = argumentCaptor.getAllValues(); + assertThat(values.get(0).getEventType()).isEqualTo(ReplicaEventType.BATCH); + assertThat((List) values.get(0).getReplicationProperties().get(ReplicaSupport.MESSAGE_IDS_PROPERTY)).containsOnly(messageId1.toString(), messageId2.toString()); + List objects = eventSerializer.deserializeListOfObjects(values.get(0).getEventData().getData()); + assertThat(objects.size()).isEqualTo(2); + assertThat(((Message) objects.get(0)).getMessageId()).isEqualTo(messageId1); + assertThat(((Message) objects.get(1)).getMessageId()).isEqualTo(messageId2); + + assertThat(values.get(1).getEventType()).isEqualTo(ReplicaEventType.BATCH); + assertThat((List) values.get(1).getReplicationProperties().get(ReplicaSupport.MESSAGE_IDS_PROPERTY)).containsOnly(messageId3.toString(), messageId4.toString()); + objects = eventSerializer.deserializeListOfObjects(values.get(1).getEventData().getData()); + assertThat(objects.size()).isEqualTo(2); + assertThat(((Message) objects.get(0)).getMessageId()).isEqualTo(messageId3); + assertThat(((Message) objects.get(1)).getMessageId()).isEqualTo(messageId4); + } + + @Test + public void acknowledgeTest() throws Exception { + MessageId messageId = new MessageId("1:0:0:1"); + + MessageAck messageAck = new MessageAck(); + messageAck.setMessageID(messageId); + messageAck.setConsumerId(consumerId); + messageAck.setDestination(intermediateQueueDestination); + messageAck.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.BATCH.name()); + message.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageId.toString())); + + when(mainSubscription.getDispatched()).thenReturn(List.of(message)); + + ConsumerBrokerExchange cbe = new ConsumerBrokerExchange(); + cbe.setConnectionContext(connectionContext); + + sequencer.acknowledge(cbe, messageAck); + + verify(broker).acknowledge(cbe, messageAck); + + assertThat(sequencer.messageToAck).containsOnly(messageId.toString()); + } + + @Test + public void iterateAckTest() throws Exception { + sequencer.messageToAck.clear(); + + String messageId1 = "1:0:0:1"; + sequencer.messageToAck.addLast(messageId1); + String messageId2 = "1:0:0:2"; + sequencer.messageToAck.addLast(messageId2); + String messageId3 = "1:0:0:3"; + sequencer.messageToAck.addLast(messageId3); + + sequencer.iterateAck(); + + ArgumentCaptor ackArgumentCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker, times(3)).acknowledge(any(), ackArgumentCaptor.capture()); + + List values = ackArgumentCaptor.getAllValues(); + assertThat(values.get(0).getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(values.get(0).getDestination()).isEqualTo(intermediateQueueDestination); + assertThat(values.get(0).getFirstMessageId().toString()).isEqualTo(messageId1); + assertThat(values.get(0).getLastMessageId().toString()).isEqualTo(messageId1); + assertThat(values.get(1).getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(values.get(1).getDestination()).isEqualTo(intermediateQueueDestination); + assertThat(values.get(1).getFirstMessageId().toString()).isEqualTo(messageId2); + assertThat(values.get(1).getLastMessageId().toString()).isEqualTo(messageId2); + assertThat(values.get(2).getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(values.get(2).getDestination()).isEqualTo(intermediateQueueDestination); + assertThat(values.get(2).getFirstMessageId().toString()).isEqualTo(messageId3); + assertThat(values.get(2).getLastMessageId().toString()).isEqualTo(messageId3); + } + + @Test + public void iterateSendMultipleMessagesTest() throws Exception { + sequencer.hasConsumer = true; + List messages = new ArrayList(); + MessageId messageId = new MessageId("1:0:0:1"); + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + messages.add(message); + + + messageId = new MessageId("1:0:0:2"); + message.setMessageId(messageId); + message.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + messages.add(message); + + when(intermediateSubscription.getDispatched()).thenReturn(messages); + + sequencer.iterateSend(); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(ReplicaEvent.class); + verify(replicationMessageProducer).enqueueMainReplicaEvent(any(), argumentCaptor.capture()); + + ReplicaEvent value = argumentCaptor.getValue(); + assertThat(value.getEventType()).isEqualTo(ReplicaEventType.BATCH); + assertThat((List) value.getReplicationProperties().get(ReplicaSupport.MESSAGE_IDS_PROPERTY)).containsOnly(messageId.toString()); + List objects = eventSerializer.deserializeListOfObjects(value.getEventData().getData()); + assertThat(objects.size()).isEqualTo(2); + assertThat(((Message) objects.get(0)).getMessageId()).isEqualTo(messageId); + } + + @Test + public void iterateSendSingleMessageTest() throws Exception { + sequencer.hasConsumer = true; + + MessageId messageId = new MessageId("1:0:0:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + + when(intermediateSubscription.getDispatched()).thenReturn(List.of(message)); + + sequencer.iterateSend(); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(replicaInternalMessageProducer, times(3)).sendForcingFlowControl(any(), argumentCaptor.capture()); + + ActiveMQMessage activeMQMessage = argumentCaptor.getAllValues().get(0); + assertThat(activeMQMessage.getMessageId()).isEqualTo(messageId); + assertThat(activeMQMessage.getTransactionId()).isNotNull(); + assertThat(activeMQMessage.isPersistent()).isFalse(); + } + + + @Test + public void iterateSendTestWhenSomeMessagesAlreadyDelivered() throws Exception { + sequencer.hasConsumer = true; + + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + + when(intermediateSubscription.getDispatched()).thenReturn(new ArrayList<>(List.of(message1, message2, message3))); + + sequencer.deliveredMessages.add(messageId1.toString()); + sequencer.deliveredMessages.add(messageId2.toString()); + + sequencer.iterateSend(); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(replicaInternalMessageProducer, times(3)).sendForcingFlowControl(any(), argumentCaptor.capture()); + + ActiveMQMessage activeMQMessage = argumentCaptor.getAllValues().get(0); + assertThat(activeMQMessage.getMessageId()).isEqualTo(messageId3); + assertThat(activeMQMessage.getTransactionId()).isNotNull(); + assertThat(activeMQMessage.isPersistent()).isFalse(); + } + + @Test + public void iterateSendTestWhenCompactionPossible() throws Exception { + sequencer.hasConsumer = true; + + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + + String messageIdToAck = "2:1"; + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message1.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + message3.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageIdToAck)); + + when(intermediateSubscription.getDispatched()).thenReturn(new ArrayList<>(List.of(message1, message2, message3))); + + sequencer.iterateSend(); + + ArgumentCaptor argumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(replicaInternalMessageProducer, times(3)).sendForcingFlowControl(any(), argumentCaptor.capture()); + + ActiveMQMessage activeMQMessage = argumentCaptor.getAllValues().get(0); + assertThat(activeMQMessage.getMessageId()).isEqualTo(messageId2); + assertThat(activeMQMessage.getTransactionId()).isNotNull(); + assertThat(activeMQMessage.isPersistent()).isFalse(); + + ArgumentCaptor ackCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker, times(2)).acknowledge(any(), ackCaptor.capture()); + + List values = ackCaptor.getAllValues(); + MessageAck messageAck = values.get(0); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId1); + messageAck = values.get(1); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId3); + } + + @Test + public void iterateSendDoNotSendToMainQueueIfNoConsumer() throws Exception { + sequencer.hasConsumer = false; + when(intermediateSubscription.isFull()).thenReturn(true); + + ActiveMQMessage ackMessage1 = new ActiveMQMessage(); + ackMessage1.setMessageId(new MessageId("2:0:0:1")); + ackMessage1.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("0:0:0:1")); + ackMessage1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + ActiveMQMessage ackMessage2 = new ActiveMQMessage(); + ackMessage2.setMessageId(new MessageId("2:0:0:2")); + ackMessage2.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("0:0:0:2")); + ackMessage2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + ActiveMQMessage ackMessage3 = new ActiveMQMessage(); + ackMessage3.setMessageId(new MessageId("2:0:0:3")); + ackMessage3.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of("0:0:0:3")); + ackMessage3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + + List ackMessageReferences = new ArrayList<>(); + ackMessageReferences.add(new IndirectMessageReference(ackMessage1)); + ackMessageReferences.add(new IndirectMessageReference(ackMessage2)); + ackMessageReferences.add(new IndirectMessageReference(ackMessage3)); + + when(intermediateQueue.getMatchingMessages(eq(connectionContext), any(ReplicaCompactor.AckMessageReferenceFilter.class), eq(MAXIMUM_MESSAGES))) + .thenReturn(ackMessageReferences); + + ActiveMQMessage sendMessage1 = new ActiveMQMessage(); + sendMessage1.setMessageId(new MessageId("2:0:0:1")); + sendMessage1.setProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "0:0:0:1"); + sendMessage1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage sendMessage2 = new ActiveMQMessage(); + sendMessage2.setMessageId(new MessageId("2:0:0:2")); + sendMessage2.setProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "0:0:0:2"); + sendMessage2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage sendMessage3 = new ActiveMQMessage(); + sendMessage3.setMessageId(new MessageId("2:0:0:3")); + sendMessage3.setProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, "0:0:0:3"); + sendMessage3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + + List sendMessageReferences = new ArrayList<>(); + sendMessageReferences.add(new IndirectMessageReference(sendMessage1)); + sendMessageReferences.add(new IndirectMessageReference(sendMessage2)); + sendMessageReferences.add(new IndirectMessageReference(sendMessage3)); + when(intermediateQueue.getMatchingMessages(eq(connectionContext), any(ReplicaCompactor.SendMessageReferenceFilter.class), eq(3))) + .thenReturn(sendMessageReferences); + + String messageIdToAck = "2:1"; + + MessageId messageId1 = new MessageId("1:0:0:1"); + MessageId messageId2 = new MessageId("1:0:0:2"); + MessageId messageId3 = new MessageId("1:0:0:3"); + + ActiveMQMessage message1 = new ActiveMQMessage(); + message1.setMessageId(messageId1); + message1.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message1.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + message1.setStringProperty(ReplicaSupport.MESSAGE_ID_PROPERTY, messageIdToAck); + ActiveMQMessage message2 = new ActiveMQMessage(); + message2.setMessageId(messageId2); + message2.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message2.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_SEND.toString()); + ActiveMQMessage message3 = new ActiveMQMessage(); + message3.setMessageId(messageId3); + message3.setBooleanProperty(ReplicaSupport.IS_ORIGINAL_MESSAGE_SENT_TO_QUEUE_PROPERTY, true); + message3.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, ReplicaEventType.MESSAGE_ACK.toString()); + message3.setProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY, List.of(messageIdToAck)); + + when(intermediateSubscription.getDispatched()).thenReturn(new ArrayList<>(List.of(message1, message2, message3))); + + sequencer.iterateSend(); + + ArgumentCaptor ackCaptor = ArgumentCaptor.forClass(MessageAck.class); + verify(broker, times(2)).acknowledge(any(), ackCaptor.capture()); + + List values = ackCaptor.getAllValues(); + MessageAck messageAck = values.get(0); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId1); + messageAck = values.get(1); + assertThat(messageAck.getAckType()).isEqualTo(MessageAck.INDIVIDUAL_ACK_TYPE); + assertThat(messageAck.getMessageCount()).isEqualTo(1); + assertThat(messageAck.getLastMessageId()).isEqualTo(messageId3); + + verify(broker, times(3)).addConsumer(any(), any()); + verify(replicationMessageProducer, never()).enqueueMainReplicaEvent(any(), any(ReplicaEvent.class)); + + ArgumentCaptor filterArgumentCaptor = ArgumentCaptor.forClass(MessageReferenceFilter.class); + ArgumentCaptor contextArgumentCaptor = ArgumentCaptor.forClass(ConnectionContext.class); + ArgumentCaptor maxMessagesArgumentCaptor = ArgumentCaptor.forClass(Integer.class); + verify(intermediateQueue, times(2)).getMatchingMessages(contextArgumentCaptor.capture(), filterArgumentCaptor.capture(), maxMessagesArgumentCaptor.capture()); + + assertThat(maxMessagesArgumentCaptor.getAllValues().get(0)).isEqualTo(MAXIMUM_MESSAGES); + assertThat(maxMessagesArgumentCaptor.getAllValues().get(1)).isEqualTo(3); + assertThat(filterArgumentCaptor.getAllValues().get(0)).isInstanceOf(ReplicaCompactor.AckMessageReferenceFilter.class); + assertThat(filterArgumentCaptor.getAllValues().get(1)).isInstanceOf(ReplicaCompactor.SendMessageReferenceFilter.class); + contextArgumentCaptor.getAllValues().forEach( + conContext -> assertThat(conContext).isEqualTo(connectionContext) + ); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceAuthorizationBrokerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceAuthorizationBrokerTest.java new file mode 100644 index 00000000000..d6ad38688b3 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceAuthorizationBrokerTest.java @@ -0,0 +1,152 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.advisory.AdvisorySupport; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.region.CompositeDestinationInterceptor; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.RegionBroker; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.ProducerInfo; +import org.junit.Before; +import org.junit.Test; + +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaSourceAuthorizationBrokerTest { + + private final Broker broker = mock(Broker.class); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + + ReplicaAuthorizationBroker source; + private final TransportConnector transportConnector = mock(TransportConnector.class); + + private final ActiveMQQueue testDestination = new ActiveMQQueue("TEST.QUEUE"); + + + @Before + public void setUp() throws Exception { + RegionBroker regionBroker = mock(RegionBroker.class); + when(broker.getAdaptor(RegionBroker.class)).thenReturn(regionBroker); + CompositeDestinationInterceptor cdi = mock(CompositeDestinationInterceptor.class); + when(regionBroker.getDestinationInterceptor()).thenReturn(cdi); + when(cdi.getInterceptors()).thenReturn(new DestinationInterceptor[]{}); + when(connectionContext.getConnector()).thenReturn(transportConnector); + + source = new ReplicaAuthorizationBroker(broker); + } + + @Test + public void letsCreateConsumerForReplicaQueueFromReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME)); + source.addConsumer(connectionContext, consumerInfo); + + verify(broker).addConsumer(eq(connectionContext), eq(consumerInfo)); + } + + @Test(expected = ActiveMQReplicaException.class) + public void doesNotLetCreateConsumerForReplicaQueueFromNonReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn("test"); + + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME)); + source.addConsumer(connectionContext, consumerInfo); + } + + @Test + public void letsCreateConsumerForNonReplicaAdvisoryTopicFromReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + + ActiveMQTopic advisoryTopic = new ActiveMQTopic(AdvisorySupport.ADVISORY_TOPIC_PREFIX + "TEST"); + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(advisoryTopic); + source.addConsumer(connectionContext, consumerInfo); + + verify(broker).addConsumer(eq(connectionContext), eq(consumerInfo)); + } + + @Test + public void letsCreateConsumerForNonReplicaQueueFromNonReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn("test"); + + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(testDestination); + source.addConsumer(connectionContext, consumerInfo); + + verify(broker).addConsumer(eq(connectionContext), eq(consumerInfo)); + } + + @Test(expected = ActiveMQReplicaException.class) + public void doesNoLetCreateConsumerForNonReplicaQueueFromReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + + ConsumerInfo consumerInfo = new ConsumerInfo(); + consumerInfo.setDestination(testDestination); + source.addConsumer(connectionContext, consumerInfo); + } + + @Test(expected = ActiveMQReplicaException.class) + public void doesNotLetCreateProducerForReplicaQueueFromNonReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn("test"); + + ProducerInfo producerInfo = new ProducerInfo(); + producerInfo.setDestination(new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME)); + source.addProducer(connectionContext, producerInfo); + } + + @Test + public void letsCreateProducerForReplicaQueueFromReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + + ProducerInfo producerInfo = new ProducerInfo(); + producerInfo.setDestination(new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME)); + source.addProducer(connectionContext, producerInfo); + + verify(broker).addProducer(eq(connectionContext), eq(producerInfo)); + } + + @Test + public void letsCreateProducerForNonReplicaQueueFromNonReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn("test"); + + ProducerInfo producerInfo = new ProducerInfo(); + producerInfo.setDestination(testDestination); + source.addProducer(connectionContext, producerInfo); + + verify(broker).addProducer(eq(connectionContext), eq(producerInfo)); + } + + @Test(expected = ActiveMQReplicaException.class) + public void doesNotLetCreateProducerForNonReplicaQueueFromReplicaConnection() throws Exception { + when(transportConnector.getName()).thenReturn(ReplicaSupport.REPLICATION_CONNECTOR_NAME); + + ProducerInfo producerInfo = new ProducerInfo(); + producerInfo.setDestination(testDestination); + source.addProducer(connectionContext, producerInfo); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceBrokerTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceBrokerTest.java new file mode 100644 index 00000000000..4f71e861b91 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/ReplicaSourceBrokerTest.java @@ -0,0 +1,565 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica; + +import org.apache.activemq.advisory.AdvisorySupport; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.ConsumerBrokerExchange; +import org.apache.activemq.broker.ProducerBrokerExchange; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.MessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.Subscription; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.command.ConsumerId; +import org.apache.activemq.command.ConsumerInfo; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.RemoveSubscriptionInfo; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.command.XATransactionId; +import org.apache.activemq.filter.DestinationMapEntry; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import java.net.URI; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaSourceBrokerTest { + + private static final DestinationMapEntry IS_REPLICATED = new DestinationMapEntry() {}; + private final Broker broker = mock(Broker.class); + private final BrokerService brokerService = mock(BrokerService.class); + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final MessageReference messageReference = mock(MessageReference.class); + private final Subscription subscription = mock(Subscription.class); + private final URI transportConnectorUri = URI.create("tcp://0.0.0.0:61618?maximumConnections=1&wireFormat.maxFrameSize=104857600"); + private final ReplicaSequencer replicaSequencer = mock(ReplicaSequencer.class); + private final ReplicaReplicationQueueSupplier queueProvider = new ReplicaReplicationQueueSupplier(broker); + private ReplicaSourceBroker source; + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final TransportConnector transportConnector = mock(TransportConnector.class); + + private final ActiveMQQueue testDestination = new ActiveMQQueue("TEST.QUEUE"); + + @Before + public void setUp() throws Exception { + when(broker.getBrokerService()).thenReturn(brokerService); + when(broker.getAdminConnectionContext()).thenReturn(connectionContext); + when(brokerService.addConnector(transportConnectorUri)).thenReturn(transportConnector); + when(connectionContext.isProducerFlowControl()).thenReturn(false); + when(connectionContext.getConnector()).thenReturn(transportConnector); + when(transportConnector.getName()).thenReturn("test"); + when(connectionContext.getClientId()).thenReturn("clientId"); + when(connectionContext.copy()).thenReturn(mock(ConnectionContext.class)); + + ReplicaInternalMessageProducer replicaInternalMessageProducer = new ReplicaInternalMessageProducer(broker); + ReplicationMessageProducer replicationMessageProducer = new ReplicationMessageProducer(replicaInternalMessageProducer, queueProvider); + ReplicaPolicy replicaPolicy = new ReplicaPolicy(); + replicaPolicy.setTransportConnectorUri(transportConnectorUri); + source = new ReplicaSourceBroker(broker, null, replicationMessageProducer, replicaSequencer, queueProvider, replicaPolicy); + when(brokerService.getBroker()).thenReturn(source); + + source.destinationsToReplicate.put(testDestination, IS_REPLICATED); + } + + @Test + public void createsQueueOnInitialization() throws Exception { + source.start(ReplicaRole.source); + + ArgumentCaptor destinationArgumentCaptor = ArgumentCaptor.forClass(ActiveMQDestination.class); + verify(broker, times(3)).addDestination(eq(connectionContext), destinationArgumentCaptor.capture(), anyBoolean()); + + List replicationDestinations = destinationArgumentCaptor.getAllValues(); + assertThat(replicationDestinations.get(0).getPhysicalName()).isEqualTo(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertThat(replicationDestinations.get(1).getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicationDestinations.get(2).getPhysicalName()).isEqualTo(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + } + + @Test + public void doesNotCreateDestinationEventsForNonReplicableDestinations() throws Exception { + source.start(ReplicaRole.source); + + ActiveMQTopic advisoryTopic = new ActiveMQTopic(AdvisorySupport.ADVISORY_TOPIC_PREFIX + "TEST"); + source.addDestination(connectionContext, advisoryTopic, true); + + ArgumentCaptor destinationArgumentCaptor = ArgumentCaptor.forClass(ActiveMQDestination.class); + verify(broker, times(4)).addDestination(eq(connectionContext), destinationArgumentCaptor.capture(), anyBoolean()); + + List destinations = destinationArgumentCaptor.getAllValues(); + + ActiveMQDestination mainReplicationDestination = destinations.get(0); + assertThat(mainReplicationDestination.getPhysicalName()).isEqualTo(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + + ActiveMQDestination intermediateReplicationDestination = destinations.get(1); + assertThat(intermediateReplicationDestination.getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + + ActiveMQDestination sequenceReplicationDestination = destinations.get(2); + assertThat(sequenceReplicationDestination.getPhysicalName()).isEqualTo(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + + ActiveMQDestination advisoryTopicDestination = destinations.get(3); + assertThat(advisoryTopicDestination).isEqualTo(advisoryTopic); + + verify(broker, never()).send(any(), any()); + } + + @Test + public void replicates_MESSAGE_SEND() throws Exception { + source.start(ReplicaRole.source); + + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setDestination(testDestination); + message.setPersistent(true); + + ProducerBrokerExchange producerExchange = new ProducerBrokerExchange(); + producerExchange.setConnectionContext(connectionContext); + + source.send(producerExchange, message); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + + final List values = messageArgumentCaptor.getAllValues(); + + ActiveMQMessage originalMessage = values.get(0); + assertThat(originalMessage).isEqualTo(message); + } + + @Test + public void replicates_QUEUE_PURGED() throws Exception { + source.start(ReplicaRole.source); + + source.queuePurged(connectionContext, testDestination); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicaMessage = messageArgumentCaptor.getValue(); + + assertThat(replicaMessage.getType()).isEqualTo("ReplicaEvent"); + assertThat(replicaMessage.getDestination().getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicaMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.QUEUE_PURGED.name()); + assertThat(replicaMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + + ActiveMQDestination sentMessage = (ActiveMQDestination) eventSerializer.deserializeMessageData(replicaMessage.getContent()); + assertThat(sentMessage).isEqualTo(testDestination); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_MESSAGE_EXPIRED() throws Exception { + ActiveMQMessage message = new ActiveMQMessage(); + MessageId messageId = new MessageId("1:1"); + message.setMessageId(messageId); + message.setDestination(testDestination); + message.setPersistent(true); + when(messageReference.getMessage()).thenReturn(message); + + source.start(ReplicaRole.source); + source.messageExpired(connectionContext, messageReference, subscription); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicaMessage = messageArgumentCaptor.getValue(); + + assertThat(replicaMessage.getType()).isEqualTo("ReplicaEvent"); + assertThat(replicaMessage.getDestination().getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicaMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.MESSAGE_EXPIRED.name()); + assertThat(replicaMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + + ActiveMQMessage sentMessage = (ActiveMQMessage) eventSerializer.deserializeMessageData(replicaMessage.getContent()); + assertThat(sentMessage.getDestination().getPhysicalName()).isEqualTo(testDestination.getPhysicalName()); + verifyConnectionContext(connectionContext); + } + + @Test + public void do_not_replicate_REPLICA_QUEUES_PURGED() throws Exception { + source.start(ReplicaRole.source); + + ActiveMQQueue mainQueue = new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + source.queuePurged(connectionContext, mainQueue); + verify(broker, times(0)).send(any(), any()); + + ActiveMQQueue intermediateQueue = new ActiveMQQueue(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + source.queuePurged(connectionContext, intermediateQueue); + verify(broker, times(0)).send(any(), any()); + + ActiveMQQueue sequenceQueue = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + source.queuePurged(connectionContext, sequenceQueue); + verify(broker, times(0)).send(any(), any()); + } + + @Test + public void replicates_BEGIN_TRANSACTION() throws Exception { + source.start(ReplicaRole.source); + + TransactionId transactionId = new XATransactionId(); + + source.beginTransaction(connectionContext, transactionId); + + verify(broker, times(1)).beginTransaction(any(), eq(transactionId)); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = messageArgumentCaptor.getValue(); + final TransactionId replicatedTransactionId = (TransactionId) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.TRANSACTION_BEGIN.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(replicatedTransactionId).isEqualTo(transactionId); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_PREPARE_TRANSACTION() throws Exception { + source.start(ReplicaRole.source); + + TransactionId transactionId = new XATransactionId(); + + source.prepareTransaction(connectionContext, transactionId); + + verify(broker, times(1)).prepareTransaction(any(), eq(transactionId)); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = messageArgumentCaptor.getValue(); + final TransactionId replicatedTransactionId = (TransactionId) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.TRANSACTION_PREPARE.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(replicatedTransactionId).isEqualTo(transactionId); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_ROLLBACK_TRANSACTION() throws Exception { + source.start(ReplicaRole.source); + + TransactionId transactionId = new XATransactionId(); + + source.rollbackTransaction(connectionContext, transactionId); + + verify(broker, times(1)).rollbackTransaction(any(), eq(transactionId)); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = messageArgumentCaptor.getValue(); + final TransactionId replicatedTransactionId = (TransactionId) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.TRANSACTION_ROLLBACK.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(replicatedTransactionId).isEqualTo(transactionId); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_FORGET_TRANSACTION() throws Exception { + source.start(ReplicaRole.source); + + TransactionId transactionId = new XATransactionId(); + + source.forgetTransaction(connectionContext, transactionId); + + verify(broker, times(1)).forgetTransaction(any(), eq(transactionId)); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = messageArgumentCaptor.getValue(); + final TransactionId replicatedTransactionId = (TransactionId) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.TRANSACTION_FORGET.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(replicatedTransactionId).isEqualTo(transactionId); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_COMMIT_TRANSACTION() throws Exception { + source.start(ReplicaRole.source); + + TransactionId transactionId = new XATransactionId(); + + source.commitTransaction(connectionContext, transactionId, true); + + verify(broker, times(1)).commitTransaction(any(), eq(transactionId), eq(true)); + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = messageArgumentCaptor.getValue(); + final TransactionId replicatedTransactionId = (TransactionId) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.TRANSACTION_COMMIT.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(replicatedTransactionId).isEqualTo(transactionId); + assertThat(replicationMessage.getProperty(ReplicaSupport.TRANSACTION_ONE_PHASE_PROPERTY)).isEqualTo(true); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_ADD_DURABLE_CONSUMER() throws Exception { + source.start(ReplicaRole.source); + + ActiveMQTopic destination = new ActiveMQTopic("TEST.TOPIC"); + + ConsumerInfo message = new ConsumerInfo(); + message.setDestination(destination); + message.setSubscriptionName("SUBSCRIPTION_NAME"); + + source.addConsumer(connectionContext, message); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicaMessage = messageArgumentCaptor.getValue(); + + assertThat(replicaMessage.getType()).isEqualTo("ReplicaEvent"); + assertThat(replicaMessage.getDestination().getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicaMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.ADD_DURABLE_CONSUMER.name()); + assertThat(replicaMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + + final ConsumerInfo ackMessage = (ConsumerInfo) eventSerializer.deserializeMessageData(replicaMessage.getContent()); + assertThat(ackMessage.getDestination()).isEqualTo(destination); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_REMOVE_DURABLE_CONSUMER() throws Exception { + source.start(ReplicaRole.source); + + ActiveMQTopic destination = new ActiveMQTopic("TEST.TOPIC"); + + ConsumerInfo message = new ConsumerInfo(); + message.setDestination(destination); + message.setSubscriptionName("SUBSCRIPTION_NAME"); + + source.removeConsumer(connectionContext, message); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicaMessage = messageArgumentCaptor.getValue(); + + assertThat(replicaMessage.getType()).isEqualTo("ReplicaEvent"); + assertThat(replicaMessage.getDestination().getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicaMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.REMOVE_DURABLE_CONSUMER.name()); + assertThat(replicaMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + + final ConsumerInfo ackMessage = (ConsumerInfo) eventSerializer.deserializeMessageData(replicaMessage.getContent()); + assertThat(ackMessage.getDestination()).isEqualTo(destination); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_REMOVE_DURABLE_CONSUMER_SUBSCRIPTION() throws Exception { + source.start(ReplicaRole.source); + + RemoveSubscriptionInfo removeSubscriptionInfo = new RemoveSubscriptionInfo(); + removeSubscriptionInfo.setClientId("clientId"); + removeSubscriptionInfo.setSubscriptionName("SUBSCRIPTION_NAME"); + + source.removeSubscription(connectionContext, removeSubscriptionInfo); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + ActiveMQMessage replicaMessage = messageArgumentCaptor.getValue(); + + assertThat(replicaMessage.getType()).isEqualTo("ReplicaEvent"); + assertThat(replicaMessage.getDestination().getPhysicalName()).isEqualTo(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertThat(replicaMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.REMOVE_DURABLE_CONSUMER_SUBSCRIPTION.name()); + assertThat(replicaMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + + final RemoveSubscriptionInfo removeSubscriptionInfoMsg = (RemoveSubscriptionInfo) eventSerializer.deserializeMessageData(replicaMessage.getContent()); + assertThat(removeSubscriptionInfoMsg.getClientId()).isEqualTo("clientId"); + assertThat(removeSubscriptionInfoMsg.getSubscriptionName()).isEqualTo("SUBSCRIPTION_NAME"); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_MESSAGE_ACK_individual() throws Exception { + source.start(ReplicaRole.source); + + MessageId messageId = new MessageId("1:1"); + + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setPersistent(true); + + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + MessageAck messageAck = new MessageAck(); + messageAck.setMessageID(messageId); + messageAck.setConsumerId(consumerId); + messageAck.setDestination(testDestination); + messageAck.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + + Queue queue = mock(Queue.class); + when(broker.getDestinations(testDestination)).thenReturn(Set.of(queue)); + PrefetchSubscription subscription = mock(PrefetchSubscription.class); + when(queue.getConsumers()).thenReturn(List.of(subscription)); + ConsumerInfo consumerInfo = new ConsumerInfo(consumerId); + when(subscription.getConsumerInfo()).thenReturn(consumerInfo); + when(subscription.getDispatched()).thenReturn(List.of(new IndirectMessageReference(message))); + + ConsumerBrokerExchange cbe = new ConsumerBrokerExchange(); + cbe.setConnectionContext(connectionContext); + source.acknowledge(cbe, messageAck); + + ArgumentCaptor sendMessageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), sendMessageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = sendMessageArgumentCaptor.getValue(); + final MessageAck originalMessage = (MessageAck) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.MESSAGE_ACK.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(originalMessage.getLastMessageId()).isEqualTo(messageId); + assertThat(originalMessage.getDestination()).isEqualTo(testDestination); + assertThat((List) replicationMessage.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY)).containsOnly(messageId.toString()); + verifyConnectionContext(connectionContext); + } + + @Test + public void replicates_MESSAGE_ACK_individual_nonpersistent() throws Exception { + source.start(ReplicaRole.source); + + MessageId messageId = new MessageId("1:1"); + + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setPersistent(false); + + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + MessageAck messageAck = new MessageAck(); + messageAck.setMessageID(messageId); + messageAck.setConsumerId(consumerId); + messageAck.setDestination(testDestination); + messageAck.setAckType(MessageAck.INDIVIDUAL_ACK_TYPE); + + Queue queue = mock(Queue.class); + when(broker.getDestinations(testDestination)).thenReturn(Set.of(queue)); + PrefetchSubscription subscription = mock(PrefetchSubscription.class); + when(queue.getConsumers()).thenReturn(List.of(subscription)); + ConsumerInfo consumerInfo = new ConsumerInfo(consumerId); + when(subscription.getConsumerInfo()).thenReturn(consumerInfo); + when(subscription.getDispatched()).thenReturn(List.of(new IndirectMessageReference(message))); + + ConsumerBrokerExchange cbe = new ConsumerBrokerExchange(); + cbe.setConnectionContext(connectionContext); + source.acknowledge(cbe, messageAck); + + ArgumentCaptor sendMessageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, never()).send(any(), sendMessageArgumentCaptor.capture()); + } + + @Test + public void replicates_MESSAGE_ACK_standard() throws Exception { + source.start(ReplicaRole.source); + + MessageId firstMessageId = new MessageId("1:1"); + MessageId secondMessageId = new MessageId("1:2"); + MessageId thirdMessageId = new MessageId("1:3"); + + ActiveMQMessage firstMessage = new ActiveMQMessage(); + firstMessage.setMessageId(firstMessageId); + firstMessage.setPersistent(true); + ActiveMQMessage secondMessage = new ActiveMQMessage(); + secondMessage.setMessageId(secondMessageId); + secondMessage.setPersistent(true); + ActiveMQMessage thirdMessage = new ActiveMQMessage(); + thirdMessage.setMessageId(thirdMessageId); + thirdMessage.setPersistent(true); + + ConsumerId consumerId = new ConsumerId("2:2:2:2"); + MessageAck messageAck = new MessageAck(); + messageAck.setConsumerId(consumerId); + messageAck.setFirstMessageId(firstMessageId); + messageAck.setLastMessageId(thirdMessageId); + messageAck.setDestination(testDestination); + messageAck.setAckType(MessageAck.STANDARD_ACK_TYPE); + + Queue queue = mock(Queue.class); + when(broker.getDestinations(testDestination)).thenReturn(Set.of(queue)); + PrefetchSubscription subscription = mock(PrefetchSubscription.class); + when(queue.getConsumers()).thenReturn(List.of(subscription)); + ConsumerInfo consumerInfo = new ConsumerInfo(consumerId); + when(subscription.getConsumerInfo()).thenReturn(consumerInfo); + when(subscription.getDispatched()).thenReturn(List.of( + new IndirectMessageReference(firstMessage), + new IndirectMessageReference(secondMessage), + new IndirectMessageReference(thirdMessage) + )); + + ConsumerBrokerExchange cbe = new ConsumerBrokerExchange(); + cbe.setConnectionContext(connectionContext); + source.acknowledge(cbe, messageAck); + + ArgumentCaptor sendMessageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker, times(1)).send(any(), sendMessageArgumentCaptor.capture()); + ActiveMQMessage replicationMessage = sendMessageArgumentCaptor.getValue(); + final MessageAck originalMessage = (MessageAck) eventSerializer.deserializeMessageData(replicationMessage.getContent()); + assertThat(replicationMessage.getProperty(ReplicaEventType.EVENT_TYPE_PROPERTY)).isEqualTo(ReplicaEventType.MESSAGE_ACK.name()); + assertThat(replicationMessage.getProperty(ReplicaSupport.VERSION_PROPERTY)).isEqualTo(ReplicaSupport.CURRENT_VERSION); + assertThat(originalMessage.getFirstMessageId()).isEqualTo(firstMessageId); + assertThat(originalMessage.getLastMessageId()).isEqualTo(thirdMessageId); + assertThat(originalMessage.getDestination()).isEqualTo(testDestination); + assertThat((List) replicationMessage.getProperty(ReplicaSupport.MESSAGE_IDS_PROPERTY)) + .containsOnly(firstMessageId.toString(), secondMessageId.toString(), thirdMessageId.toString()); + verifyConnectionContext(connectionContext); + } + + @Test + public void doesNotReplicateAdvisoryTopics() throws Exception { + source.start(ReplicaRole.source); + + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setType(AdvisorySupport.ADIVSORY_MESSAGE_TYPE); + message.setDestination(testDestination); + + ProducerBrokerExchange producerExchange = new ProducerBrokerExchange(); + producerExchange.setConnectionContext(connectionContext); + + source.send(producerExchange, message); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(broker).send(any(), messageArgumentCaptor.capture()); + + final List values = messageArgumentCaptor.getAllValues(); + + ActiveMQMessage originalMessage = values.get(0); + assertThat(originalMessage).isEqualTo(message); + + verify(connectionContext, never()).isProducerFlowControl(); + verify(connectionContext, never()).setProducerFlowControl(anyBoolean()); + } + + private void verifyConnectionContext(ConnectionContext context) { + verify(context).isProducerFlowControl(); + verify(context).setProducerFlowControl(true); + verify(context).setProducerFlowControl(false); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorageTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorageTest.java new file mode 100644 index 00000000000..bed40184ec9 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaRecoverySequenceStorageTest.java @@ -0,0 +1,95 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.Before; +import org.junit.Test; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +public class ReplicaRecoverySequenceStorageTest { + + private final static String SEQUENCE_NAME = "testSeq"; + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final Broker broker = mock(Broker.class); + private final ReplicaReplicationQueueSupplier queueProvider = mock(ReplicaReplicationQueueSupplier.class); + private final Queue sequenceQueue = mock(Queue.class); + private final PrefetchSubscription subscription = mock(PrefetchSubscription.class); + private final ActiveMQQueue sequenceQueueDestination = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + private final ReplicaInternalMessageProducer replicaProducer = mock(ReplicaInternalMessageProducer.class); + + private ReplicaRecoverySequenceStorage replicaSequenceStorage; + + @Before + public void setUp() throws Exception { + when(broker.getDestinations(any())).thenReturn(Set.of(sequenceQueue)); + ConnectionContext adminConnectionContext = mock(ConnectionContext.class); + when(adminConnectionContext.copy()).thenReturn(connectionContext); + when(broker.getAdminConnectionContext()).thenReturn(adminConnectionContext); + when(broker.addConsumer(any(), any())).thenReturn(subscription); + when(queueProvider.getSequenceQueue()).thenReturn(sequenceQueueDestination); + + this.replicaSequenceStorage = new ReplicaRecoverySequenceStorage(broker, queueProvider, replicaProducer, SEQUENCE_NAME); + } + + @Test + public void shouldInitializeWhenNoMessagesExist() throws Exception { + when(subscription.getDispatched()).thenReturn(new ArrayList<>()); + + List initialize = replicaSequenceStorage.initialize(connectionContext); + assertThat(initialize).isEmpty(); + verify(sequenceQueue, never()).removeMessage(any()); + } + + @Test + public void shouldInitializeWhenMoreThanOneExist() throws Exception { + ActiveMQTextMessage message1 = new ActiveMQTextMessage(); + message1.setMessageId(new MessageId("1:0:0:1")); + message1.setText("1"); + message1.setStringProperty(ReplicaBaseSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + ActiveMQTextMessage message2 = new ActiveMQTextMessage(); + message2.setMessageId(new MessageId("1:0:0:2")); + message2.setText("2"); + message2.setStringProperty(ReplicaBaseSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + + when(subscription.getDispatched()) + .thenReturn(List.of(new IndirectMessageReference(message1), new IndirectMessageReference(message2))); + + List initialize = replicaSequenceStorage.initialize(connectionContext); + assertThat(initialize).containsExactly(message1.getText(), message2.getText()); + } +} diff --git a/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaSequenceStorageTest.java b/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaSequenceStorageTest.java new file mode 100644 index 00000000000..2832b7f5ef6 --- /dev/null +++ b/activemq-broker/src/test/java/org/apache/activemq/replica/storage/ReplicaSequenceStorageTest.java @@ -0,0 +1,166 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.replica.storage; + +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.ConnectionContext; +import org.apache.activemq.broker.region.IndirectMessageReference; +import org.apache.activemq.broker.region.PrefetchSubscription; +import org.apache.activemq.broker.region.Queue; +import org.apache.activemq.broker.region.QueueMessageReference; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ConnectionId; +import org.apache.activemq.command.LocalTransactionId; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.command.TransactionId; +import org.apache.activemq.replica.ReplicaInternalMessageProducer; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.Before; +import org.junit.FixMethodOrder; +import org.junit.Test; +import org.junit.runners.MethodSorters; +import org.mockito.ArgumentCaptor; + +import java.util.ArrayList; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@FixMethodOrder(MethodSorters.NAME_ASCENDING) +public class ReplicaSequenceStorageTest { + + private final static String SEQUENCE_NAME = "testSeq"; + private final ConnectionContext connectionContext = mock(ConnectionContext.class); + private final Broker broker = mock(Broker.class); + private final ReplicaReplicationQueueSupplier queueProvider = mock(ReplicaReplicationQueueSupplier.class); + private final Queue sequenceQueue = mock(Queue.class); + private final ActiveMQQueue sequenceQueueDestination = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + private final PrefetchSubscription subscription = mock(PrefetchSubscription.class); + private final ReplicaInternalMessageProducer replicaProducer = mock(ReplicaInternalMessageProducer.class); + + + private ReplicaSequenceStorage replicaSequenceStorage; + + @Before + public void setUp() throws Exception { + when(broker.getDestinations(any())).thenReturn(Set.of(sequenceQueue)); + ConnectionContext adminConnectionContext = mock(ConnectionContext.class); + when(adminConnectionContext.copy()).thenReturn(connectionContext); + when(broker.getAdminConnectionContext()).thenReturn(adminConnectionContext); + when(broker.addConsumer(any(), any())).thenReturn(subscription); + when(queueProvider.getSequenceQueue()).thenReturn(sequenceQueueDestination); + + this.replicaSequenceStorage = new ReplicaSequenceStorage(broker, queueProvider, replicaProducer, SEQUENCE_NAME); + } + + @Test + public void shouldInitializeWhenNoMessagesExist() throws Exception { + when(subscription.getDispatched()).thenReturn(new ArrayList<>()).thenReturn(new ArrayList<>()); + + String initialize = replicaSequenceStorage.initialize(connectionContext); + assertThat(initialize).isNull(); + verify(sequenceQueue, never()).removeMessage(any()); + } + + @Test + public void shouldInitializeWhenMoreThanOneExist() throws Exception { + ActiveMQTextMessage message1 = new ActiveMQTextMessage(); + message1.setMessageId(new MessageId("1:0:0:1")); + message1.setText("1"); + message1.setStringProperty(ReplicaBaseSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + ActiveMQTextMessage message2 = new ActiveMQTextMessage(); + message2.setMessageId(new MessageId("1:0:0:2")); + message2.setText("2"); + message2.setStringProperty(ReplicaBaseSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + + when(subscription.getDispatched()) + .thenReturn(List.of(new IndirectMessageReference(message1), new IndirectMessageReference(message2))); + + String initialize = replicaSequenceStorage.initialize(connectionContext); + assertThat(initialize).isEqualTo(message1.getText()); + verify(sequenceQueue, times(1)).removeMessage(eq(message1.getMessageId().toString())); + } + + @Test + public void shouldEnqueueMessage() throws Exception { + String messageToEnqueue = "THIS IS A MESSAGE"; + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + ArgumentCaptor activeMQTextMessageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQTextMessage.class); + when(subscription.getDispatched()).thenReturn(new ArrayList<>()); + replicaSequenceStorage.initialize(connectionContext); + + replicaSequenceStorage.enqueue(connectionContext, transactionId, messageToEnqueue); + + verify(replicaProducer, times(1)).sendForcingFlowControl(any(), activeMQTextMessageArgumentCaptor.capture()); + assertThat(activeMQTextMessageArgumentCaptor.getValue().getText()).isEqualTo(messageToEnqueue); + assertThat(activeMQTextMessageArgumentCaptor.getValue().getTransactionId()).isEqualTo(transactionId); + assertThat(activeMQTextMessageArgumentCaptor.getValue().getDestination()).isEqualTo(sequenceQueueDestination); + assertThat(activeMQTextMessageArgumentCaptor.getValue().isPersistent()).isTrue(); + assertThat(activeMQTextMessageArgumentCaptor.getValue().isResponseRequired()).isFalse(); + reset(broker); + reset(subscription); + } + + @Test + public void shouldAcknowledgeAllMessagesWhenEnqueue() throws Exception { + ActiveMQTextMessage message1 = new ActiveMQTextMessage(); + message1.setMessageId(new MessageId("1:0:0:1")); + message1.setText("1"); + message1.setStringProperty(ReplicaSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + ActiveMQTextMessage message2 = new ActiveMQTextMessage(); + message2.setMessageId(new MessageId("1:0:0:3")); + message2.setText("3"); + message2.setStringProperty(ReplicaSequenceStorage.SEQUENCE_NAME_PROPERTY, SEQUENCE_NAME); + + QueueMessageReference messageReference1 = mock(QueueMessageReference.class); + when(messageReference1.getMessage()).thenReturn(message1); + when(messageReference1.getMessageId()).thenReturn(message1.getMessageId()); + QueueMessageReference messageReference2 = mock(QueueMessageReference.class); + when(messageReference2.getMessage()).thenReturn(message2); + when(messageReference2.getMessageId()).thenReturn(message2.getMessageId()); + + when(subscription.getDispatched()).thenReturn(List.of(messageReference1, messageReference2)); + replicaSequenceStorage.initialize(connectionContext); + + ArgumentCaptor ackArgumentCaptor = ArgumentCaptor.forClass(MessageAck.class); + + String messageToEnqueue = "THIS IS A MESSAGE"; + TransactionId transactionId = new LocalTransactionId(new ConnectionId("10101010"), 101010); + + replicaSequenceStorage.enqueue(connectionContext, transactionId, messageToEnqueue); + verify(broker).acknowledge(any(), ackArgumentCaptor.capture()); + MessageAck value = ackArgumentCaptor.getValue(); + assertThat(value.getFirstMessageId()).isEqualTo(message1.getMessageId()); + assertThat(value.getLastMessageId()).isEqualTo(message2.getMessageId()); + assertThat(value.getDestination()).isEqualTo(sequenceQueueDestination); + assertThat(value.getMessageCount()).isEqualTo(2); + assertThat(value.getAckType()).isEqualTo(MessageAck.STANDARD_ACK_TYPE); + + } +} diff --git a/activemq-broker/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker b/activemq-broker/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker new file mode 100644 index 00000000000..58004542b23 --- /dev/null +++ b/activemq-broker/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker @@ -0,0 +1,18 @@ +## --------------------------------------------------------------------------- +## Licensed to the Apache Software Foundation (ASF) under one or more +## contributor license agreements. See the NOTICE file distributed with +## this work for additional information regarding copyright ownership. +## The ASF licenses this file to You under the Apache License, Version 2.0 +## (the "License"); you may not use this file except in compliance with +## the License. You may obtain a copy of the License at +## +## http://www.apache.org/licenses/LICENSE-2.0 +## +## Unless required by applicable law or agreed to in writing, software +## distributed under the License is distributed on an "AS IS" BASIS, +## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +## See the License for the specific language governing permissions and +## limitations under the License. +## --------------------------------------------------------------------------- + +mock-maker-inline \ No newline at end of file diff --git a/activemq-unit-tests/pom.xml b/activemq-unit-tests/pom.xml index b1f8ee4678c..fbee7b4531b 100644 --- a/activemq-unit-tests/pom.xml +++ b/activemq-unit-tests/pom.xml @@ -293,6 +293,12 @@ mockito-inline test + + org.eclipse.paho + org.eclipse.paho.client.mqttv3 + 1.2.1 + test + @@ -1160,5 +1166,26 @@ + + activemq.tests-replica-plugin + + + activemq.tests + replica-plugin + + + + + + maven-surefire-plugin + + + **/replica/*Test.* + + + + + + diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaAcknowledgeReplicationEventTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaAcknowledgeReplicationEventTest.java new file mode 100644 index 00000000000..e7252ce2fda --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaAcknowledgeReplicationEventTest.java @@ -0,0 +1,287 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQMessageConsumer; +import org.apache.activemq.ActiveMQSession; +import org.apache.activemq.ActiveMQXAConnectionFactory; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.MessageAck; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaPolicy; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaRoleManagementBroker; +import org.apache.activemq.replica.ReplicaStatistics; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import javax.jms.*; +import java.lang.IllegalStateException; +import java.net.URI; + +import java.text.MessageFormat; +import java.util.LinkedList; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.doNothing; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.spy; + +public class ReplicaAcknowledgeReplicationEventTest extends ReplicaPluginTestSupport { + static final int MAX_BATCH_LENGTH = 500; + + private static final Logger LOG = LoggerFactory.getLogger(ReplicaAcknowledgeReplicationEventTest.class); + + protected Connection firstBrokerConnection; + + ActiveMQConnectionFactory mockConnectionFactorySpy; + + ActiveMQConnection mockConnectionSpy; + + ReplicaPolicy mockReplicaPolicy; + + ActiveMQSession mockReplicaSession; + + @Before + public void setUp() throws Exception { + firstBroker = createFirstBroker(); + firstBroker.start(); + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + firstBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(firstBindAddress); + + mockReplicaPolicy = spy(ReplicaPolicy.class); + mockConnectionFactorySpy = spy(new ActiveMQConnectionFactory(firstReplicaBindAddress)); + mockConnectionSpy = spy((ActiveMQConnection) mockConnectionFactorySpy.createConnection()); + doReturn(mockConnectionFactorySpy).when(mockReplicaPolicy).getOtherBrokerConnectionFactory(); + doReturn(mockConnectionSpy).when(mockConnectionFactorySpy).createConnection(); + + if (secondBroker == null) { + secondBroker = createSecondBroker(); + } + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + } + + @After + protected void tearDown() throws Exception { + firstBrokerConnection.close(); + mockConnectionSpy.close(); + mockReplicaSession.close(); + super.tearDown(); + } + + @Test + public void testReplicaBrokerDoNotAckOnReplicaEvent() throws Exception { + mockReplicaSession = spy((ActiveMQSession) mockConnectionSpy.createSession(false, ActiveMQSession.CLIENT_ACKNOWLEDGE)); + doReturn(mockReplicaSession).when(mockConnectionSpy).createSession(eq(false), eq(ActiveMQSession.CLIENT_ACKNOWLEDGE)); + doNothing().when(mockReplicaSession).acknowledge(); + + startSecondBroker(); + destination = createDestination(); + Thread.sleep(SHORT_TIMEOUT); + + waitUntilReplicationQueueHasConsumer(firstBroker); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. 0"); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 0); + assertTrue(firstBrokerMainQueueView.getEnqueueCount() >= 1); + + secondBroker.stop(); + secondBroker.waitUntilStopped(); + + message = new ActiveMQTextMessage(); + message.setText(getName() + " No. 1"); + firstBrokerProducer.send(message); + + secondBroker = super.createSecondBroker(); + secondBroker.start(); + Thread.sleep(LONG_TIMEOUT * 2); + + waitUntilReplicationQueueHasConsumer(firstBroker); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerQueueView.getDequeueCount(), 3); + assertTrue(firstBrokerQueueView.getEnqueueCount() >= 2); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + } catch (Exception|Error urlException) { + LOG.error("Caught error during wait: " + urlException.getMessage()); + throw new RuntimeException(urlException); + } + }); + + } + + @Test + public void testReplicaSendCorrectAck() throws Exception { + mockConnectionSpy.start(); + LinkedList messagesToAck = new LinkedList<>(); + ActiveMQQueue replicationSourceQueue = mockConnectionSpy.getDestinationSource().getQueues().stream() + .peek(q -> System.out.println("Queue: " + q.getPhysicalName())) + .filter(d -> ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME.equals(d.getPhysicalName())) + .findFirst() + .orElseThrow(() -> new IllegalStateException( + MessageFormat.format("There is no replication queue on the source broker {0}", mockConnectionSpy.getBrokerName()) + )); + + mockReplicaSession = (ActiveMQSession) mockConnectionSpy.createSession(false, ActiveMQSession.CLIENT_ACKNOWLEDGE); + ActiveMQMessageConsumer mainQueueConsumer = (ActiveMQMessageConsumer) mockReplicaSession.createConsumer(replicationSourceQueue); + + mainQueueConsumer.setMessageListener(message -> { + ActiveMQMessage msg = (ActiveMQMessage) message; + messagesToAck.add(msg); + }); + + destination = createDestination(); + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + for (int i = 0; i < MAX_BATCH_LENGTH * 4 + 10; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT * 2); + + MessageAck ack = new MessageAck(messagesToAck.getLast(), MessageAck.STANDARD_ACK_TYPE, messagesToAck.size()); + ack.setFirstMessageId(messagesToAck.getFirst().getMessageId()); + ack.setConsumerId(mainQueueConsumer.getConsumerId()); + mockReplicaSession.syncSendPacket(ack); + Thread.sleep(LONG_TIMEOUT); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), messagesToAck.size()); + assertEquals(firstBrokerMainQueueView.getEnqueueCount(), messagesToAck.size()); + } catch (Exception|Error urlException) { + LOG.error("Caught error during wait: " + urlException.getMessage()); + throw new RuntimeException(urlException); + } + }); + } + + @Test + public void testReplicaSendOutOfOrderAck() throws Exception { + mockConnectionSpy.start(); + ActiveMQQueue replicationSourceQueue = mockConnectionSpy.getDestinationSource().getQueues().stream() + .peek(q -> System.out.println("Queue: " + q.getPhysicalName())) + .filter(d -> ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME.equals(d.getPhysicalName())) + .findFirst() + .orElseThrow(() -> new IllegalStateException( + MessageFormat.format("There is no replication queue on the source broker {0}", mockConnectionSpy.getBrokerName()) + )); + + mockReplicaSession = (ActiveMQSession) mockConnectionSpy.createSession(false, ActiveMQSession.CLIENT_ACKNOWLEDGE); + ActiveMQMessageConsumer mainQueueConsumer = (ActiveMQMessageConsumer) mockReplicaSession.createConsumer(replicationSourceQueue); + + mainQueueConsumer.setMessageListener(new MessageListener() { + @Override + public void onMessage(Message message) { + try { + ActiveMQMessage msg = (ActiveMQMessage) message; + MessageAck ack = new MessageAck(msg, MessageAck.STANDARD_ACK_TYPE, 1); + MessageId outOfOrderMessageId = msg.getMessageId().copy(); + outOfOrderMessageId.setProducerSequenceId(msg.getMessageId().getProducerSequenceId() + 100); + ack.setFirstMessageId(outOfOrderMessageId); + ack.setLastMessageId(outOfOrderMessageId); + ack.setConsumerId(mainQueueConsumer.getConsumerId()); + mockReplicaSession.syncSendPacket(ack); + fail("should have thrown IllegalStateException!"); + } catch (JMSException e) { + assertTrue(e.getMessage().contains("Could not find messages for ack")); + assertTrue(e.getCause() instanceof IllegalStateException); + } + } + }); + + destination = createDestination(); + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. 0"); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT * 2); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 0); + assertTrue(firstBrokerMainQueueView.getEnqueueCount() >= 1); + } catch (Exception|Error urlException) { + LOG.error("Caught error during wait: " + urlException.getMessage()); + throw new RuntimeException(urlException); + } + }); + } + + @Override + protected BrokerService createSecondBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(secondBindAddress); + answer.setDataDirectory(SECOND_KAHADB_DIRECTORY); + answer.setBrokerName("secondBroker"); + mockReplicaPolicy.setTransportConnectorUri(URI.create(secondReplicaBindAddress)); + + ReplicaPlugin replicaPlugin = new ReplicaPlugin() { + @Override + public Broker installPlugin(final Broker broker) { + return new ReplicaRoleManagementBroker(broker, mockReplicaPolicy, ReplicaRole.replica, new ReplicaStatistics()); + } + }; + replicaPlugin.setRole(ReplicaRole.replica); + replicaPlugin.setTransportConnectorUri(secondReplicaBindAddress); + replicaPlugin.setOtherBrokerUri(firstReplicaBindAddress); + replicaPlugin.setControlWebConsoleAccess(false); + replicaPlugin.setHeartBeatPeriod(0); + + answer.setPlugins(new BrokerPlugin[]{replicaPlugin}); + answer.setSchedulerSupport(true); + return answer; + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionLevelMQTTConnectionTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionLevelMQTTConnectionTest.java new file mode 100644 index 00000000000..00381cf41db --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionLevelMQTTConnectionTest.java @@ -0,0 +1,268 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.TestSupport; +import org.apache.activemq.broker.BrokerFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.jmx.TopicViewMBean; +import org.apache.activemq.command.ActiveMQBytesMessage; +import org.apache.activemq.command.ActiveMQTopic; +import org.eclipse.paho.client.mqttv3.IMqttDeliveryToken; +import org.eclipse.paho.client.mqttv3.MqttCallback; +import org.eclipse.paho.client.mqttv3.MqttClient; +import org.eclipse.paho.client.mqttv3.MqttConnectOptions; +import org.eclipse.paho.client.mqttv3.MqttMessage; +import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence; +import org.fusesource.mqtt.client.QoS; +import org.fusesource.mqtt.client.Topic; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.mockito.ArgumentCaptor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.Connection; +import javax.jms.ConnectionFactory; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageListener; +import javax.jms.Session; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.MalformedObjectNameException; +import javax.management.ObjectName; +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.nio.charset.StandardCharsets; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Arrays; +import java.util.Collection; +import java.util.Date; +import java.util.UUID; + +import static org.apache.activemq.broker.replica.ReplicaPluginTestSupport.SHORT_TIMEOUT; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.Mockito.atLeastOnce; +import static org.mockito.Mockito.atMostOnce; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.verify; + +@RunWith(Parameterized.class) +public class ReplicaConnectionLevelMQTTConnectionTest extends TestSupport { + private static final Logger LOG = LoggerFactory.getLogger(ReplicaConnectionLevelMQTTConnectionTest.class); + public static final String KEYSTORE_TYPE = "jks"; + public static final String PASSWORD = "password"; + public static final String SERVER_KEYSTORE = "src/test/resources/server.keystore"; + public static final String TRUST_KEYSTORE = "src/test/resources/client.keystore"; + public static final String PRIMARY_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-primary.xml"; + public static final String REPLICA_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-replica.xml"; + private static final DateFormat df = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss.S"); + protected static final String SECOND_BROKER_BINDING_ADDRESS = "vm://secondBrokerLocalhost"; + protected static final int LONG_TIMEOUT = 15000; + private static final String CLIENT_ID_ONE = "one"; + private ConnectionFactory secondBrokerConnectionFactory; + private Connection secondBrokerConnection; + private final String protocol; + protected BrokerService firstBroker; + protected BrokerService secondBroker; + protected Topic destination; + + @Before + public void setUp() throws Exception { + firstBroker = setUpBrokerService(PRIMARY_BROKER_CONFIG); + secondBroker = setUpBrokerService(REPLICA_BROKER_CONFIG); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(SECOND_BROKER_BINDING_ADDRESS); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.setClientID(CLIENT_ID_ONE); + secondBrokerConnection.start(); + destination = new Topic(getDestinationString(), QoS.AT_LEAST_ONCE); + } + + @After + public void tearDown() throws Exception { + secondBrokerConnection.stop(); + if (firstBroker != null) { + try { + firstBroker.stop(); + firstBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + if (secondBroker != null) { + try { + secondBroker.stop(); + secondBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + } + + @Parameterized.Parameters(name="protocol={0}") + public static Collection getTestParameters() { + return Arrays.asList(new String[][] { + {"mqtt"}, {"mqtt+ssl"}, {"mqtt+nio+ssl"}, {"mqtt+nio"} + }); + } + + static { + System.setProperty("javax.net.ssl.trustStore", TRUST_KEYSTORE); + System.setProperty("javax.net.ssl.trustStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.trustStoreType", KEYSTORE_TYPE); + System.setProperty("javax.net.ssl.keyStore", SERVER_KEYSTORE); + System.setProperty("javax.net.ssl.keyStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.keyStoreType", KEYSTORE_TYPE); + } + + public ReplicaConnectionLevelMQTTConnectionTest(String protocol) { + this.protocol = protocol; + } + + @Test + public void testConnectWithMqttProtocol() throws Exception { + MqttConnectOptions firstBrokerOptions = new MqttConnectOptions(); + firstBrokerOptions.setCleanSession(false); + firstBrokerOptions.setAutomaticReconnect(true); + String firstBrokerConnectionUri = getMQTTClientUri(firstBroker.getTransportConnectorByScheme(protocol)); + MqttClient firstBrokerClient = new MqttClient(firstBrokerConnectionUri, UUID.randomUUID().toString(), new MemoryPersistence()); + firstBrokerClient.connect(firstBrokerOptions); + String payloadMessage = "testConnectWithMqttProtocol payload"; + + MqttCallback mqttCallback = new MqttCallback() { + public void connectionLost(Throwable cause) { + } + + public void messageArrived(String topic, MqttMessage message) throws Exception { + System.out.println(String.format("%s - Receiver: received '%s'", df.format(new Date()), new String(message.getPayload()))); + assertEquals(payloadMessage, new String(message.getPayload())); + } + + public void deliveryComplete(IMqttDeliveryToken token) { + } + }; + + + MqttCallback callbackSpy = spy(mqttCallback); + firstBrokerClient.setCallback(callbackSpy); + + LOG.info(String.format("mqtt client successfully connected to %s", firstBrokerClient.getServerURI())); + firstBrokerClient.subscribe(destination.toString()); + firstBrokerClient.publish(destination.toString(), payloadMessage.getBytes(StandardCharsets.UTF_8), 1, false); + Thread.sleep(SHORT_TIMEOUT); + + ArgumentCaptor mqttMessageArgumentCaptor = ArgumentCaptor.forClass(MqttMessage.class); + verify(callbackSpy).messageArrived(anyString(), mqttMessageArgumentCaptor.capture()); + MqttMessage messageReceived = mqttMessageArgumentCaptor.getValue(); + assertEquals(payloadMessage, new String(messageReceived.getPayload())); + verify(callbackSpy, never()).connectionLost(any()); + verify(callbackSpy, atMostOnce()).deliveryComplete(any()); + + firstBrokerClient.disconnect(); + } + + @Test + public void testReplicaReceiveMessage() throws Exception { + MqttConnectOptions firstBrokerOptions = new MqttConnectOptions(); + firstBrokerOptions.setCleanSession(false); + firstBrokerOptions.setAutomaticReconnect(true); + String firstBrokerConnectionUri = getMQTTClientUri(firstBroker.getTransportConnectorByScheme(protocol)); + MqttClient firstBrokerClient = new MqttClient(firstBrokerConnectionUri, UUID.randomUUID().toString(), new MemoryPersistence()); + firstBrokerClient.connect(firstBrokerOptions); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + String destinationName = "testReplicaReceiveMessage"; + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber(new ActiveMQTopic(destinationName), CLIENT_ID_ONE); + String payloadMessage = "testConnectWithMqttProtocol payload"; + LOG.info(String.format("mqtt client successfully connected to %s", firstBrokerClient.getServerURI())); + + var listener = spy(new MessageListener() { + @Override + public void onMessage(Message message) { + assertTrue(message instanceof ActiveMQBytesMessage); + assertEquals(payloadMessage, new String(((ActiveMQBytesMessage) message).getContent().getData())); + } + }); + secondBrokerConsumer.setMessageListener(listener); + + firstBrokerClient.publish(destinationName, payloadMessage.getBytes(StandardCharsets.UTF_8), 1, false); + Thread.sleep(LONG_TIMEOUT); + TopicViewMBean secondBrokerDestinationTopicView = getTopicView(secondBroker, destinationName); + assertEquals(secondBrokerDestinationTopicView.getDequeueCount(), 0); + assertEquals(secondBrokerDestinationTopicView.getEnqueueCount(), 1); + verify(listener, atLeastOnce()).onMessage(any()); + + firstBrokerClient.disconnect(); + secondBrokerSession.close(); + } + + protected TopicViewMBean getTopicView(BrokerService broker, String topicName) throws MalformedObjectNameException { + MBeanServer mbeanServer = broker.getManagementContext().getMBeanServer(); + String objectNameStr = broker.getBrokerObjectName().toString(); + objectNameStr += ",destinationType=Topic,destinationName=" + topicName; + ObjectName topicViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + return MBeanServerInvocationHandler.newProxyInstance(mbeanServer, topicViewMBeanName, TopicViewMBean.class, true); + } + + protected BrokerService setUpBrokerService(String configurationUri) throws Exception { + BrokerService broker = createBroker(configurationUri); + broker.setPersistent(false); + return broker; + } + + protected BrokerService createBroker(String uri) throws Exception { + LOG.info("Loading broker configuration from the classpath with URI: " + uri); + return BrokerFactory.createBroker(new URI("xbean:" + uri)); + } + + private String getMQTTClientUri(TransportConnector mqttConnector) throws IOException, URISyntaxException { + if (protocol.contains("ssl")) { + return "ssl://localhost:" + mqttConnector.getConnectUri().getPort(); + } else { + return "tcp://localhost:" + mqttConnector.getConnectUri().getPort(); + } + } + + protected String getDestinationString() { + return getClass().getName() + "." + getName(); + } + + protected ObjectName assertRegisteredObjectName(MBeanServer mbeanServer, String name) throws MalformedObjectNameException, NullPointerException { + ObjectName objectName = new ObjectName(name); + if (mbeanServer.isRegistered(objectName)) { + System.out.println("Bean Registered: " + objectName); + } else { + fail("Could not find MBean!: " + objectName); + } + return objectName; + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionModeTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionModeTest.java new file mode 100644 index 00000000000..d9ee576ce33 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaConnectionModeTest.java @@ -0,0 +1,168 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; + +public class ReplicaConnectionModeTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + @Override + protected void setUp() throws Exception { + cleanKahaDB(FIRST_KAHADB_DIRECTORY); + cleanKahaDB(SECOND_KAHADB_DIRECTORY); + super.setUp(); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + super.tearDown(); + } + + @Test (timeout = 60000) + public void testAsyncConnection() throws Exception { + ((ActiveMQConnection) firstBrokerConnection).setUseAsyncSend(true); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + receivedMessage.acknowledge(); + Thread.sleep(LONG_TIMEOUT); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testConsumerAutoAcknowledgeMode() throws Exception { + firstBrokerConnection.start(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + + Session firstBrokerConsumerSession = firstBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageConsumer firstBrokerConsumer = firstBrokerConsumerSession.createConsumer(destination); + + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + Thread.sleep(LONG_TIMEOUT); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testConsumerDupsOkAcknowledgeMode() throws Exception { + firstBrokerConnection.start(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + + Session firstBrokerConsumerSession = firstBrokerConnection.createSession(false, Session.DUPS_OK_ACKNOWLEDGE); + MessageConsumer firstBrokerConsumer = firstBrokerConsumerSession.createConsumer(destination); + + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertTrue(((TextMessage) receivedMessage).getText().contains(getName())); + Thread.sleep(LONG_TIMEOUT); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaHardFailoverTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaHardFailoverTest.java new file mode 100644 index 00000000000..3a7207e5604 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaHardFailoverTest.java @@ -0,0 +1,240 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.jmx.ReplicationViewMBean; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; + +public class ReplicaHardFailoverTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + private ReplicationViewMBean firstBrokerReplicationView; + private ReplicationViewMBean secondBrokerReplicationView; + + @Override + protected void setUp() throws Exception { + firstBroker = setUpFirstBroker(); + secondBroker = setUpSecondBroker(); + + ReplicaPlugin firstBrokerPlugin = new ReplicaPlugin(); + firstBrokerPlugin.setRole(ReplicaRole.source); + firstBrokerPlugin.setTransportConnectorUri(firstReplicaBindAddress); + firstBrokerPlugin.setOtherBrokerUri(secondReplicaBindAddress); + firstBrokerPlugin.setControlWebConsoleAccess(false); + firstBrokerPlugin.setHeartBeatPeriod(0); + firstBroker.setPlugins(new BrokerPlugin[]{firstBrokerPlugin}); + + ReplicaPlugin secondBrokerPlugin = new ReplicaPlugin(); + secondBrokerPlugin.setRole(ReplicaRole.replica); + secondBrokerPlugin.setTransportConnectorUri(secondReplicaBindAddress); + secondBrokerPlugin.setOtherBrokerUri(firstReplicaBindAddress); + secondBrokerPlugin.setControlWebConsoleAccess(false); + secondBrokerPlugin.setHeartBeatPeriod(0); + secondBroker.setPlugins(new BrokerPlugin[]{secondBrokerPlugin}); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + firstBrokerReplicationView = getReplicationView(firstBroker); + secondBrokerReplicationView = getReplicationView(secondBroker); + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + destination = createDestination(); + + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + @Test + public void testGetReplicationRoleViaJMX() throws Exception { + firstBrokerReplicationView = getReplicationView(firstBroker); + secondBrokerReplicationView = getReplicationView(secondBroker); + + assertEquals(ReplicaRole.source, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + } + + @Test + public void testHardFailover() throws Exception { + firstBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), true); + secondBrokerReplicationView.setReplicationRole(ReplicaRole.source.name(), true); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + assertEquals(ReplicaRole.source, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerProducer.send(message); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testBothBrokerFailoverToPrimary() throws Exception { + secondBrokerReplicationView.setReplicationRole(ReplicaRole.source.name(), true); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(ReplicaRole.source, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + assertEquals(ReplicaRole.source, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + ActiveMQDestination destination2 = new ActiveMQQueue(getDestinationString() + "No2"); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination2); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination2); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerProducer.send(message); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testBothBrokerFailoverToReplica() throws Exception { + firstBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), true); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + ActiveMQDestination destination2 = new ActiveMQQueue(getDestinationString() + "No2"); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination2); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination2); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerProducer.send(message); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + private BrokerService setUpSecondBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(secondBindAddress); + answer.setDataDirectory(SECOND_KAHADB_DIRECTORY); + answer.setBrokerName("secondBroker"); + return answer; + } + + private BrokerService setUpFirstBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(firstBindAddress); + answer.setDataDirectory(FIRST_KAHADB_DIRECTORY); + answer.setBrokerName("firstBroker"); + return answer; + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaMessagePropertyTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaMessagePropertyTest.java new file mode 100644 index 00000000000..12d8e5ac5bd --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaMessagePropertyTest.java @@ -0,0 +1,273 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQMessageConsumer; +import org.apache.activemq.ActiveMQSession; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.MessageAck; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.DeliveryMode; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; + +public class ReplicaMessagePropertyTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + @Override + protected void setUp() throws Exception { + super.setUp(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + @Test + public void testNonPersistentMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + firstBrokerProducer.setDeliveryMode(DeliveryMode.NON_PERSISTENT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message secondBrokerReceivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(secondBrokerReceivedMessage); + + Message firstBrokerReceivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(firstBrokerReceivedMessage); + assertTrue(firstBrokerReceivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) firstBrokerReceivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testMessagePriority() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress + "?jms.messagePrioritySupported=true"); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress + "?jms.messagePrioritySupported=true"); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + for (int i = 1; i <= 3; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + "No." + i); + firstBrokerProducer.send(message, DeliveryMode.PERSISTENT, i, 0); + } + + Thread.sleep(LONG_TIMEOUT); + + for (int i = 3; i >=1; i--) { + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName() + "No." + i, ((TextMessage) receivedMessage).getText()); + } + + assertNull(secondBrokerConsumer.receive(SHORT_TIMEOUT)); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testMessageWithJMSXGroupID() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumerA = secondBrokerSession.createConsumer(destination); + MessageConsumer secondBrokerConsumerB = secondBrokerSession.createConsumer(destination); + + int messagesToSendNum = 20; + for (int i = 0; i < messagesToSendNum; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + "No." + i); + if (i % 2 == 0) { + message.setStringProperty("JMSXGroupID", "Group-A"); + } else { + message.setStringProperty("JMSXGroupID", "Group-B"); + } + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT); + + for (int i =0; i < messagesToSendNum/2; i++) { + Message consumerAReceivedMessage = secondBrokerConsumerA.receive(LONG_TIMEOUT); + assertNotNull(consumerAReceivedMessage); + assertTrue(consumerAReceivedMessage instanceof TextMessage); + assertTrue(((TextMessage) consumerAReceivedMessage).getText().contains(getName())); + assertEquals(consumerAReceivedMessage.getStringProperty("JMSXGroupID"), "Group-A"); + + Message consumerBReceivedMessage = secondBrokerConsumerB.receive(LONG_TIMEOUT); + assertNotNull(consumerBReceivedMessage); + assertTrue(consumerBReceivedMessage instanceof TextMessage); + assertTrue(((TextMessage) consumerBReceivedMessage).getText().contains(getName())); + assertEquals(consumerBReceivedMessage.getStringProperty("JMSXGroupID"), "Group-B"); + } + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testExpiredAcknowledgeReplication() throws Exception { + ActiveMQSession firstBrokerSession = (ActiveMQSession) firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + ActiveMQMessageConsumer firstBrokerConsumer = (ActiveMQMessageConsumer) firstBrokerSession.createConsumer(destination); + ActiveMQSession secondBrokerSession = (ActiveMQSession) secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerDestinationQueueView = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(secondBrokerDestinationQueueView.browseMessages().size(), 1); + + ActiveMQMessage receivedMessage = (ActiveMQMessage) firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + MessageAck ack = new MessageAck(receivedMessage, MessageAck.EXPIRED_ACK_TYPE, 1); + ack.setFirstMessageId(receivedMessage.getMessageId()); + ack.setConsumerId(firstBrokerConsumer.getConsumerId()); + firstBrokerSession.syncSendPacket(ack); + + assertEquals(secondBrokerDestinationQueueView.getDequeueCount(), 0); + assertEquals(secondBrokerDestinationQueueView.getEnqueueCount(), 1); + + receivedMessage = (ActiveMQMessage) secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + + firstBrokerSession.close(); + } + + @Test + public void testPoisonAcknowledgeReplication() throws Exception { + ActiveMQSession firstBrokerSession = (ActiveMQSession) firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + ActiveMQMessageConsumer firstBrokerConsumer = (ActiveMQMessageConsumer) firstBrokerSession.createConsumer(destination); + ActiveMQSession secondBrokerSession = (ActiveMQSession) secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerDestinationQueueView = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(secondBrokerDestinationQueueView.browseMessages().size(), 1); + + ActiveMQMessage receivedMessage = (ActiveMQMessage) firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + MessageAck ack = new MessageAck(receivedMessage, MessageAck.EXPIRED_ACK_TYPE, 1); + ack.setFirstMessageId(receivedMessage.getMessageId()); + ack.setConsumerId(firstBrokerConsumer.getConsumerId()); + firstBrokerSession.syncSendPacket(ack); + + assertEquals(secondBrokerDestinationQueueView.getDequeueCount(), 0); + assertEquals(secondBrokerDestinationQueueView.getEnqueueCount(), 1); + + receivedMessage = (ActiveMQMessage) secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + + firstBrokerSession.close(); + } + + @Test + public void testReDeliveredAcknowledgeReplication() throws Exception { + ActiveMQSession firstBrokerSession = (ActiveMQSession) firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + ActiveMQMessageConsumer firstBrokerConsumer = (ActiveMQMessageConsumer) firstBrokerSession.createConsumer(destination); + ActiveMQSession secondBrokerSession = (ActiveMQSession) secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerDestinationQueueView = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(secondBrokerDestinationQueueView.browseMessages().size(), 1); + + ActiveMQMessage receivedMessage = (ActiveMQMessage) firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + MessageAck ack = new MessageAck(receivedMessage, MessageAck.REDELIVERED_ACK_TYPE, 1); + ack.setFirstMessageId(receivedMessage.getMessageId()); + ack.setConsumerId(firstBrokerConsumer.getConsumerId()); + firstBrokerSession.syncSendPacket(ack); + + assertEquals(secondBrokerDestinationQueueView.getDequeueCount(), 0); + assertEquals(secondBrokerDestinationQueueView.getEnqueueCount(), 1); + + receivedMessage = (ActiveMQMessage) secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + + firstBrokerSession.close(); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorTest.java new file mode 100644 index 00000000000..6131845d1ae --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorTest.java @@ -0,0 +1,353 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.BrokerFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.jmx.BrokerViewMBean; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.network.DiscoveryNetworkConnector; +import org.apache.activemq.network.NetworkBridge; +import org.apache.activemq.network.NetworkConnector; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.util.Wait; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.ObjectName; +import java.net.URI; +import java.util.Arrays; +import java.util.List; + +public class ReplicaNetworkConnectorTest extends ReplicaPluginTestSupport { + + protected Connection firstBroker2Connection; + protected Connection secondBroker2Connection; + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + protected BrokerService firstBroker2; + protected BrokerService secondBroker2; + protected NetworkConnector primarySideNetworkConnector; + protected NetworkConnector replicaSideNetworkConnector; + protected BrokerViewMBean firstBrokerMBean; + protected BrokerViewMBean secondBrokerMBean; + protected BrokerViewMBean firstBroker2MBean; + protected BrokerViewMBean secondBroker2MBean; + protected static final String FIRSTBROKER2_KAHADB_DIRECTORY = "target/activemq-data/firstBroker2/"; + protected static final String SECONDBROKER2_KAHADB_DIRECTORY = "target/activemq-data/secondBroker2/"; + protected String firstBroker2URI = "vm://firstBroker2"; + protected String secondBroker2URI = "vm://secondBroker2"; + + + @Override + protected void setUp() throws Exception { + cleanKahaDB(FIRST_KAHADB_DIRECTORY); + cleanKahaDB(SECOND_KAHADB_DIRECTORY); + super.setUp(); + firstBroker2 = createBrokerFromBrokerFactory(new URI("broker:(" + firstBroker2URI + ")/firstBroker2?persistent=false"), FIRSTBROKER2_KAHADB_DIRECTORY); + secondBroker2 = createBrokerFromBrokerFactory(new URI("broker:(" + secondBroker2URI + ")/secondBroker2?persistent=false"), SECONDBROKER2_KAHADB_DIRECTORY); + + firstBroker2.start(); + secondBroker2.start(); + firstBroker2.waitUntilStarted(); + secondBroker2.waitUntilStarted(); + + primarySideNetworkConnector = startNetworkConnector(firstBroker, firstBroker2); + replicaSideNetworkConnector = startNetworkConnector(secondBroker, secondBroker2); + + firstBroker2Connection = new ActiveMQConnectionFactory(firstBroker2URI).createConnection(); + secondBroker2Connection = new ActiveMQConnectionFactory(secondBroker2URI).createConnection(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + + firstBroker2Connection.start(); + secondBroker2Connection.start(); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + + firstBrokerMBean = setBrokerMBean(firstBroker); + firstBroker2MBean = setBrokerMBean(firstBroker2); + secondBrokerMBean = setBrokerMBean(secondBroker); + secondBroker2MBean = setBrokerMBean(secondBroker2); + + waitUntilReplicationQueueHasConsumer(firstBroker); + } + + @Override + protected void tearDown() throws Exception { + if (firstBroker2Connection != null) { + firstBroker2Connection.close(); + firstBroker2Connection = null; + } + if (secondBroker2Connection != null) { + secondBroker2Connection.close(); + secondBroker2Connection = null; + } + + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + primarySideNetworkConnector.stop(); + replicaSideNetworkConnector.stop(); + + if (firstBroker2 != null) { + try { + firstBroker2.stop(); + } catch (Exception e) { + } + } + if (secondBroker2 != null) { + try { + secondBroker2.stop(); + } catch (Exception e) { + } + } + + super.tearDown(); + } + + @Test + public void testNetworkConnectorConsumeMessageInPrimarySide() throws Exception { + Session firstBroker2ProducerSession = firstBroker2Connection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageProducer producer = firstBroker2ProducerSession.createProducer(destination); + + TextMessage message = firstBroker2ProducerSession.createTextMessage(getName()); + producer.send(message); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageConsumer consumer = firstBrokerSession.createConsumer(destination); + + TextMessage receivedMessage = (TextMessage) consumer.receive(LONG_TIMEOUT); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(getName(), receivedMessage.getText()); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerDestinationQueue = getQueueView(firstBroker, destination.getPhysicalName()); + assertEquals(1, firstBrokerDestinationQueue.getDequeueCount()); + QueueViewMBean firstBroker2DestinationQueue = getQueueView(firstBroker2, destination.getPhysicalName()); + assertEquals(1, firstBroker2DestinationQueue.getDequeueCount()); + } catch (Exception urlException) { + urlException.printStackTrace(); + throw new RuntimeException(urlException); + } + }); + + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + assertNull(secondBrokerConsumer.receive(LONG_TIMEOUT)); + + firstBroker2ProducerSession.close(); + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testNetworkConnectorConsumeMessageInFirstBroker2() throws Exception { + Session firstBrokerProducerSession = firstBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageProducer producer = firstBrokerProducerSession.createProducer(destination); + + TextMessage message = firstBrokerProducerSession.createTextMessage(getName()); + producer.send(message); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + Session firstBroker2Session = firstBroker2Connection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer consumer = firstBroker2Session.createConsumer(destination); + TextMessage receivedMessage = (TextMessage) consumer.receive(LONG_TIMEOUT); + assertEquals(getName(), receivedMessage.getText()); + Thread.sleep(LONG_TIMEOUT); + receivedMessage.acknowledge(); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerDestinationQueue = getQueueView(firstBroker, destination.getPhysicalName()); + assertEquals(1, firstBrokerDestinationQueue.getDequeueCount()); + QueueViewMBean firstBroker2DestinationQueue = getQueueView(firstBroker2, destination.getPhysicalName()); + assertEquals(1, firstBroker2DestinationQueue.getDequeueCount()); + QueueViewMBean secondBrokerDestinationQueue = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(1, secondBrokerDestinationQueue.getDequeueCount()); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + firstBroker2Session.close(); + firstBrokerProducerSession.close(); + } + + protected BrokerViewMBean setBrokerMBean(BrokerService broker) throws Exception { + MBeanServer mBeanServer = broker.getManagementContext().getMBeanServer(); + ObjectName brokerViewMBeanName = assertRegisteredObjectName(mBeanServer, broker.getBrokerObjectName().toString()); + return MBeanServerInvocationHandler.newProxyInstance(mBeanServer, brokerViewMBeanName, BrokerViewMBean.class, true); + } + + protected NetworkConnector startNetworkConnector(BrokerService broker1, BrokerService broker2) throws Exception { + NetworkConnector nc = bridgeBrokers(broker1, broker2); + nc.start(); + waitForNetworkBridgesFormation(List.of(broker1, broker2)); + return nc; + } + + protected BrokerService createBrokerFromBrokerFactory(URI brokerUri, String KahaDBDir) throws Exception { + cleanKahaDB(KahaDBDir); + BrokerService broker = BrokerFactory.createBroker(brokerUri); + broker.setDataDirectory(KahaDBDir); + broker.getManagementContext().setCreateConnector(false); + return broker; + } + + private NetworkConnector bridgeBrokers(BrokerService localBroker, BrokerService remoteBroker) throws Exception { + List transportConnectors = remoteBroker.getTransportConnectors(); + if (!transportConnectors.isEmpty()) { + URI remoteURI = transportConnectors.get(0).getConnectUri(); + String uri = "static:(" + remoteURI + ")"; + NetworkConnector connector = new DiscoveryNetworkConnector(new URI(uri)); + connector.setName("to-" + remoteBroker.getBrokerName()); + connector.setDynamicOnly(false); + connector.setConduitSubscriptions(true); + localBroker.addNetworkConnector(connector); + + connector.setDuplex(true); + return connector; + } else { + throw new Exception("Remote broker has no registered connectors."); + } + } + + private void waitForNetworkBridgesFormation(List brokerServices) throws Exception { + for (BrokerService broker: brokerServices) { + waitForNetworkConnectorStarts(broker); + } + } + + private void waitForNetworkConnectorStarts(BrokerService broker) throws Exception { + if (!broker.getNetworkConnectors().isEmpty()) { + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + int activeCount = 0; + for (NetworkBridge bridge : broker.getNetworkConnectors().get(0).activeBridges()) { + if (bridge.getRemoteBrokerName() != null) { + System.out.println("found bridge[" + bridge + "] to " + bridge.getRemoteBrokerName() + " on broker :" + broker.getBrokerName()); + activeCount++; + } + } + return activeCount >= 1; + } + }, Wait.MAX_WAIT_MILLIS*2); + } else { + System.out.println("broker: " + broker.getBrokerName() + " doesn't have nc"); + } + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorsOnTwoPairsOfReplicationBrokersTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorsOnTwoPairsOfReplicationBrokersTest.java new file mode 100644 index 00000000000..ca08aaf4b4b --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaNetworkConnectorsOnTwoPairsOfReplicationBrokersTest.java @@ -0,0 +1,170 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.Test; + +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.management.ObjectName; +import java.net.URI; +import java.util.Arrays; + +public class ReplicaNetworkConnectorsOnTwoPairsOfReplicationBrokersTest extends ReplicaNetworkConnectorTest { + protected String pair2FirstReplicaBindAddress = "tcp://localhost:61620"; + protected String pair2SecondReplicaBindAddress = "tcp://localhost:61621"; + @Override + protected void setUp() throws Exception { + if (firstBroker == null) { + firstBroker = createFirstBroker(); + } + if (secondBroker == null) { + secondBroker = createSecondBroker(); + } + + startFirstBroker(); + startSecondBroker(); + + firstBroker2 = createBrokerFromBrokerFactory(new URI("broker:(" + firstBroker2URI + ")/firstBroker2?persistent=false"), FIRSTBROKER2_KAHADB_DIRECTORY); + secondBroker2 = createBrokerFromBrokerFactory(new URI("broker:(" + secondBroker2URI + ")/secondBroker2?persistent=false"), SECONDBROKER2_KAHADB_DIRECTORY); + ReplicaPlugin firstBroker2ReplicaPlugin = new ReplicaPlugin(); + firstBroker2ReplicaPlugin.setRole(ReplicaRole.source); + firstBroker2ReplicaPlugin.setTransportConnectorUri(pair2FirstReplicaBindAddress); + firstBroker2ReplicaPlugin.setOtherBrokerUri(pair2SecondReplicaBindAddress); + firstBroker2ReplicaPlugin.setControlWebConsoleAccess(false); + firstBroker2ReplicaPlugin.setHeartBeatPeriod(0); + firstBroker2.setPlugins(new BrokerPlugin[]{firstBroker2ReplicaPlugin}); + + ReplicaPlugin secondBroker2ReplicaPlugin = new ReplicaPlugin(); + secondBroker2ReplicaPlugin.setRole(ReplicaRole.replica); + secondBroker2ReplicaPlugin.setTransportConnectorUri(pair2SecondReplicaBindAddress); + secondBroker2ReplicaPlugin.setOtherBrokerUri(pair2FirstReplicaBindAddress); + secondBroker2ReplicaPlugin.setControlWebConsoleAccess(false); + secondBroker2ReplicaPlugin.setHeartBeatPeriod(0); + secondBroker2.setPlugins(new BrokerPlugin[]{secondBroker2ReplicaPlugin}); + + firstBroker2.start(); + secondBroker2.start(); + firstBroker2.waitUntilStarted(); + secondBroker2.waitUntilStarted(); + + primarySideNetworkConnector = startNetworkConnector(firstBroker, firstBroker2); + replicaSideNetworkConnector = startNetworkConnector(secondBroker, secondBroker2); + + firstBroker2Connection = new ActiveMQConnectionFactory(firstBroker2URI).createConnection(); + secondBroker2Connection = new ActiveMQConnectionFactory(secondBroker2URI).createConnection(); + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + firstBroker2Connection.start(); + secondBroker2Connection.start(); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + firstBrokerMBean = setBrokerMBean(firstBroker); + firstBroker2MBean = setBrokerMBean(firstBroker2); + secondBrokerMBean = setBrokerMBean(secondBroker); + secondBroker2MBean = setBrokerMBean(secondBroker2); + + destination = createDestination(); + } + + @Test + public void testMessageConsumedByReplicaSideNetworkConnectorBroker() throws Exception { + Session firstBrokerProducerSession = firstBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageProducer producer = firstBrokerProducerSession.createProducer(destination); + TextMessage message = firstBrokerProducerSession.createTextMessage(getName()); + producer.send(message); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + Session firstBroker2Session = firstBroker2Connection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer consumer = firstBroker2Session.createConsumer(destination); + + TextMessage receivedMessage = (TextMessage) consumer.receive(LONG_TIMEOUT); + assertEquals(getName(), receivedMessage.getText()); + Thread.sleep(LONG_TIMEOUT); + receivedMessage.acknowledge(); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(Arrays.stream(firstBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(firstBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + assertEquals(Arrays.stream(secondBroker2MBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerDestinationQueue = getQueueView(firstBroker, destination.getPhysicalName()); + assertEquals(1, firstBrokerDestinationQueue.getDequeueCount()); + QueueViewMBean first2BrokerDestinationQueue = getQueueView(firstBroker2, destination.getPhysicalName()); + assertEquals(1, first2BrokerDestinationQueue.getDequeueCount()); + QueueViewMBean secondBrokerDestinationQueue = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(1, secondBrokerDestinationQueue.getDequeueCount()); + QueueViewMBean secondBroker2DestinationQueue = getQueueView(secondBroker2, destination.getPhysicalName()); + assertEquals(1, secondBroker2DestinationQueue.getDequeueCount()); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + firstBrokerProducerSession.close(); + firstBroker2Session.close(); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginFunctionsTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginFunctionsTest.java new file mode 100644 index 00000000000..f995c9f81df --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginFunctionsTest.java @@ -0,0 +1,210 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaReplicationQueueSupplier; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.util.Wait; +import org.apache.commons.lang.RandomStringUtils; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.jms.XAConnection; +import javax.management.MalformedObjectNameException; +import java.util.function.Function; + + +public class ReplicaPluginFunctionsTest extends ReplicaPluginTestSupport { + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + protected XAConnection firstBrokerXAConnection; + protected XAConnection secondBrokerXAConnection; + + protected ReplicaReplicationQueueSupplier replicationQueueSupplier; + static final int MAX_BATCH_LENGTH = 500; + static final int MAX_BATCH_SIZE = 5_000_000; // 5 Mb + static final int CONSUMER_PREFETCH_LIMIT = 10_000; + + @Override + protected void setUp() throws Exception { + super.setUp(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + firstBrokerXAConnection = firstBrokerXAConnectionFactory.createXAConnection(); + firstBrokerXAConnection.start(); + + secondBrokerXAConnection = secondBrokerXAConnectionFactory.createXAConnection(); + secondBrokerXAConnection.start(); + + replicationQueueSupplier = new ReplicaReplicationQueueSupplier(secondBroker.getBroker()); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + if (firstBrokerXAConnection != null) { + firstBrokerXAConnection.close(); + firstBrokerXAConnection = null; + } + if (secondBrokerXAConnection != null) { + secondBrokerXAConnection.close(); + secondBrokerXAConnection = null; + } + + super.tearDown(); + } + + @Test + public void testSendMessageOverMAX_BATCH_LENGTH() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + for (int i = 0; i < (int) (MAX_BATCH_LENGTH * 1.5); i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 2); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), (int) (MAX_BATCH_LENGTH * 1.5) + 1); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testSendMessageOverMAX_BATCH_SIZE() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + String bigTextMessage = RandomStringUtils.randomAlphanumeric(MAX_BATCH_SIZE + 100); + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(bigTextMessage); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + waitForCondition(() -> { + try { + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 2); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), 2); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testSendMessageOverPrefetchLimit() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. 0"); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 1); + + secondBrokerConnection.close(); + secondBrokerXAConnection.close(); + secondBroker.stop(); + secondBroker.waitUntilStopped(); + + for (int i = 1; i < CONSUMER_PREFETCH_LIMIT + 50; i++) { + message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT); + + secondBroker = createSecondBroker(); + secondBroker.setPersistent(false); + startSecondBroker(); + secondBroker.waitUntilStarted(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + secondBrokerXAConnection = secondBrokerXAConnectionFactory.createXAConnection(); + secondBrokerXAConnection.start(); + waitUntilReplicationQueueHasConsumer(firstBroker); + + Thread.sleep(LONG_TIMEOUT); + + waitForCondition(() -> { + try { + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), CONSUMER_PREFETCH_LIMIT + 51); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + + firstBrokerSession.close(); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginMirrorQueueTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginMirrorQueueTest.java new file mode 100644 index 00000000000..721504addb7 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginMirrorQueueTest.java @@ -0,0 +1,207 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.command.ActiveMQTextMessage; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; + +public class ReplicaPluginMirrorQueueTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + @Override + protected void setUp() throws Exception { + + } + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + public void testSendMessageWhenPrimaryIsMirrored() throws Exception { + firstBroker = createFirstBroker(); + firstBroker.setUseMirroredQueues(true); + secondBroker = createSecondBroker(); + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + destination = createDestination(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage.acknowledge(); + + Thread.sleep(LONG_TIMEOUT); + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageWhenReplicaIsMirrored() throws Exception { + firstBroker = createFirstBroker(); + secondBroker = createSecondBroker(); + secondBroker.setUseMirroredQueues(true); + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + destination = createDestination(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage.acknowledge(); + + Thread.sleep(LONG_TIMEOUT); + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageWhenBothSidesMirrored() throws Exception { + firstBroker = createFirstBroker(); + firstBroker.setUseMirroredQueues(true); + secondBroker = createSecondBroker(); + secondBroker.setUseMirroredQueues(true); + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + destination = createDestination(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage.acknowledge(); + + Thread.sleep(LONG_TIMEOUT); + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + +} \ No newline at end of file diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginPersistentBrokerFunctionTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginPersistentBrokerFunctionTest.java new file mode 100644 index 00000000000..b4943a52a89 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginPersistentBrokerFunctionTest.java @@ -0,0 +1,197 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQXAConnectionFactory; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.jms.XAConnection; + +public class ReplicaPluginPersistentBrokerFunctionTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + protected XAConnection firstBrokerXAConnection; + protected XAConnection secondBrokerXAConnection; + + @Override + protected void setUp() throws Exception { + + if (firstBroker == null) { + firstBroker = createFirstBroker(); + firstBroker.setPersistent(true); + } + if (secondBroker == null) { + secondBroker = createSecondBroker(); + secondBroker.setPersistent(true); + } + + cleanKahaDB(FIRST_KAHADB_DIRECTORY); + cleanKahaDB(SECOND_KAHADB_DIRECTORY); + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + firstBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(firstBindAddress); + secondBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(secondBindAddress); + + destination = createDestination(); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + firstBrokerXAConnection = firstBrokerXAConnectionFactory.createXAConnection(); + firstBrokerXAConnection.start(); + + secondBrokerXAConnection = secondBrokerXAConnectionFactory.createXAConnection(); + secondBrokerXAConnection.start(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + if (firstBrokerXAConnection != null) { + firstBrokerXAConnection.close(); + firstBrokerXAConnection = null; + } + if (secondBrokerXAConnection != null) { + secondBrokerXAConnection.close(); + secondBrokerXAConnection = null; + } + + cleanKahaDB(FIRST_KAHADB_DIRECTORY); + cleanKahaDB(SECOND_KAHADB_DIRECTORY); + super.tearDown(); + } + + @Test + public void testReplicaBrokerShouldAbleToRestoreSequence() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + int messagesToSend = 10; + for (int i = 0; i < messagesToSend; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean firstBrokerMainQueueView = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertEquals(firstBrokerMainQueueView.getDequeueCount(), 1); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), messagesToSend + 1); + secondBrokerSession.close(); + + restartSecondBroker(true); + Thread.sleep(LONG_TIMEOUT); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), messagesToSend + 1); + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testReplicaBrokerHasMessageToCatchUp() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + int messagesToSend = 10; + for (int i = 0; i < messagesToSend; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + Thread.sleep(LONG_TIMEOUT); + + secondBroker.stop(); + secondBroker.waitUntilStopped(); + + for (int i = messagesToSend; i < messagesToSend * 2; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + firstBrokerProducer.send(message); + } + + restartSecondBroker(true); + + Thread.sleep(LONG_TIMEOUT); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), messagesToSend * 2 + 1); + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + private void restartSecondBroker(boolean persistent) throws Exception { + secondBrokerConnection.close(); + secondBrokerXAConnection.close(); + secondBroker.stop(); + secondBroker.waitUntilStopped(); + + secondBroker = createSecondBroker(); + secondBroker.setPersistent(persistent); + startSecondBroker(); + secondBroker.waitUntilStarted(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + secondBrokerXAConnection = secondBrokerXAConnectionFactory.createXAConnection(); + secondBrokerXAConnection.start(); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginQueueTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginQueueTest.java new file mode 100644 index 00000000000..33741fed231 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginQueueTest.java @@ -0,0 +1,539 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.broker.jmx.BrokerViewMBean; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ActiveMQTopic; +import org.junit.Ignore; + +import javax.jms.Connection; +import javax.jms.Destination; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Queue; +import javax.jms.Session; +import javax.jms.TemporaryQueue; +import javax.jms.TextMessage; +import javax.jms.Topic; +import javax.jms.XAConnection; +import javax.jms.XASession; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.ObjectName; +import javax.transaction.xa.XAResource; +import javax.transaction.xa.Xid; +import java.util.Arrays; +import java.util.UUID; + +public class ReplicaPluginQueueTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + protected XAConnection firstBrokerXAConnection; + protected XAConnection secondBrokerXAConnection; + + @Override + protected void setUp() throws Exception { + super.setUp(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + firstBrokerXAConnection = firstBrokerXAConnectionFactory.createXAConnection(); + firstBrokerXAConnection.start(); + + secondBrokerXAConnection = secondBrokerXAConnectionFactory.createXAConnection(); + secondBrokerXAConnection.start(); + waitUntilReplicationQueueHasConsumer(firstBroker); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + if (firstBrokerXAConnection != null) { + firstBrokerXAConnection.close(); + firstBrokerXAConnection = null; + } + if (secondBrokerXAConnection != null) { + secondBrokerXAConnection.close(); + secondBrokerXAConnection = null; + } + + super.tearDown(); + } + + public void testSendMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testAcknowledgeMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage.acknowledge(); + + Thread.sleep(LONG_TIMEOUT); + + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageTransactionCommit() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(true, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.commit(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageTransactionRollback() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(true, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.rollback(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageXATransactionCommit() throws Exception { + XASession firstBrokerSession = firstBrokerXAConnection.createXASession(); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + XAResource xaRes = firstBrokerSession.getXAResource(); + Xid xid = createXid(); + xaRes.start(xid, XAResource.TMNOFLAGS); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + firstBrokerProducer.send(message); + + xaRes.end(xid, XAResource.TMSUCCESS); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.prepare(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.commit(xid, false); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageXATransactionCommitOnReplica() throws Exception { + XASession firstBrokerSession = firstBrokerXAConnection.createXASession(); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + XASession secondBrokerXaSession = secondBrokerXAConnection.createXASession(); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + XAResource xaRes = firstBrokerSession.getXAResource(); + Xid xid = createXid(); + xaRes.start(xid, XAResource.TMNOFLAGS); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + firstBrokerProducer.send(message); + + xaRes.end(xid, XAResource.TMSUCCESS); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.prepare(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes = secondBrokerXaSession.getXAResource(); + xaRes.commit(xid, false); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerXaSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageXATransactionRollback() throws Exception { + XASession firstBrokerSession = firstBrokerXAConnection.createXASession(); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + XAResource xaRes = firstBrokerSession.getXAResource(); + Xid xid = createXid(); + xaRes.start(xid, XAResource.TMNOFLAGS); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + firstBrokerProducer.send(message); + + xaRes.end(xid, XAResource.TMSUCCESS); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.prepare(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.rollback(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testPurge() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT * 2); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + MBeanServer mbeanServer = firstBroker.getManagementContext().getMBeanServer(); + String objectNameStr = firstBroker.getBrokerObjectName().toString(); + objectNameStr += ",destinationType=Queue,destinationName="+getDestinationString(); + ObjectName queueViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + QueueViewMBean proxy = MBeanServerInvocationHandler.newProxyInstance(mbeanServer, queueViewMBeanName, QueueViewMBean.class, true); + proxy.purge(); + + Thread.sleep(LONG_TIMEOUT); + + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testExpireMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + firstBrokerProducer.setTimeToLive(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + Thread.sleep(LONG_TIMEOUT + SHORT_TIMEOUT); + + secondBrokerSession.close(); + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageVirtualTopic() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + Topic virtualTopic = new ActiveMQTopic("VirtualTopic." + getDestinationString()); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(virtualTopic); + + Queue queueOne = new ActiveMQQueue("Consumer.One." + virtualTopic.getTopicName()); + Queue queueTwo = new ActiveMQQueue("Consumer.Two." + virtualTopic.getTopicName()); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumerOne = secondBrokerSession.createConsumer(queueOne); + MessageConsumer secondBrokerConsumerTwo = secondBrokerSession.createConsumer(queueTwo); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumerOne.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage = secondBrokerConsumerTwo.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Ignore ("Skipped because Pause Queue event is not replicated") + public void pauseQueueAndResume() throws Exception { + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean firstBrokerDestinationQueueView = getQueueView(firstBroker, destination.getPhysicalName()); + firstBrokerDestinationQueueView.pause(); + assertTrue(firstBrokerDestinationQueueView.isPaused()); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerDestinationQueueView = getQueueView(secondBroker, destination.getPhysicalName()); + assertTrue(secondBrokerDestinationQueueView.isPaused()); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerDestinationQueueView.resume(); + Thread.sleep(LONG_TIMEOUT); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testBrowseMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerDestinationQueueView = getQueueView(secondBroker, destination.getPhysicalName()); + assertEquals(secondBrokerDestinationQueueView.browseMessages().size(), 1); + TextMessage destinationMessage = (TextMessage) secondBrokerDestinationQueueView.browseMessages().get(0); + assertEquals(destinationMessage.getText(), getName()); + + assertEquals(secondBrokerDestinationQueueView.getProducerCount(), 0); + assertEquals(secondBrokerDestinationQueueView.getConsumerCount(), 0); + + Message receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + receivedMessage.acknowledge(); + Thread.sleep(LONG_TIMEOUT); + assertEquals(secondBrokerDestinationQueueView.getDequeueCount(), 1); + firstBrokerSession.close(); + } + + public void testDeleteMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + MBeanServer secondBrokerMbeanServer = secondBroker.getManagementContext().getMBeanServer(); + ObjectName secondBrokerViewMBeanName = assertRegisteredObjectName(secondBrokerMbeanServer, secondBroker.getBrokerObjectName().toString()); + BrokerViewMBean secondBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(secondBrokerMbeanServer, secondBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(secondBrokerMBean.getQueues().length, 3); + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains(destination.getPhysicalName())) + .count(), 1); + + MBeanServer firstBrokerMbeanServer = firstBroker.getManagementContext().getMBeanServer(); + ObjectName firstBrokerViewMBeanName = assertRegisteredObjectName(firstBrokerMbeanServer, firstBroker.getBrokerObjectName().toString()); + BrokerViewMBean firstBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(firstBrokerMbeanServer, firstBrokerViewMBeanName, BrokerViewMBean.class, true); + firstBrokerMBean.removeQueue(destination.getPhysicalName()); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(secondBrokerMBean.getQueues().length, 2); + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains(destination.getPhysicalName())) + .count(), 0); + + firstBrokerSession.close(); + } + + public void testTemporaryQueueIsNotReplicated() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + TemporaryQueue tempQueue = firstBrokerSession.createTemporaryQueue(); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + String id = UUID.randomUUID().toString(); + message.setJMSReplyTo(tempQueue); + message.setJMSCorrelationID(id); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + MessageConsumer firstBrokerDestinationConsumer = firstBrokerSession.createConsumer(destination); + Message firstBrokerMessageDestinationReceived = firstBrokerDestinationConsumer.receive(LONG_TIMEOUT); + if (firstBrokerMessageDestinationReceived instanceof TextMessage) { + TextMessage textMessage = (TextMessage) firstBrokerMessageDestinationReceived; + Destination replyBackQueue = textMessage.getJMSReplyTo(); + MessageProducer producer = firstBrokerSession.createProducer(replyBackQueue); + + TextMessage msg = firstBrokerSession.createTextMessage("Message Received : " + textMessage.getText()); + producer.send(msg); + } + + MessageConsumer firstBrokerTempQueueConsumer = firstBrokerSession.createConsumer(tempQueue); + Message firstBrokerMessageReceived = firstBrokerTempQueueConsumer.receive(LONG_TIMEOUT); + assertNotNull(firstBrokerMessageReceived); + assertTrue(((TextMessage) firstBrokerMessageReceived).getText().contains(getName())); + + String tempQueueJMXName = tempQueue.getQueueName().replaceAll(":", "_"); + MBeanServer firstBrokerMbeanServer = firstBroker.getManagementContext().getMBeanServer(); + ObjectName firstBrokerViewMBeanName = assertRegisteredObjectName(firstBrokerMbeanServer, firstBroker.getBrokerObjectName().toString()); + BrokerViewMBean firstBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(firstBrokerMbeanServer, firstBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(firstBrokerMBean.getTemporaryQueues().length, 1); + assertTrue(firstBrokerMBean.getTemporaryQueues()[0].toString().contains(tempQueueJMXName)); + + MBeanServer secondBrokerMbeanServer = secondBroker.getManagementContext().getMBeanServer(); + ObjectName secondBrokerViewMBeanName = assertRegisteredObjectName(secondBrokerMbeanServer, secondBroker.getBrokerObjectName().toString()); + BrokerViewMBean secondBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(secondBrokerMbeanServer, secondBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(secondBrokerMBean.getTemporaryQueues().length, 0); + + firstBrokerSession.close(); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTestSupport.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTestSupport.java new file mode 100644 index 00000000000..f2013c6f3da --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTestSupport.java @@ -0,0 +1,270 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQXAConnectionFactory; +import org.apache.activemq.AutoFailTestSupport; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.broker.jmx.TopicViewMBean; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTopic; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.replica.jmx.ReplicationViewMBean; +import org.apache.activemq.util.Wait; +import org.apache.commons.io.FileUtils; + +import javax.jms.ConnectionFactory; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.MalformedObjectNameException; +import javax.management.ObjectName; +import javax.transaction.xa.Xid; +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.File; +import java.io.IOException; + +public abstract class ReplicaPluginTestSupport extends AutoFailTestSupport { + + protected static final int LONG_TIMEOUT = 15000; + protected static final int SHORT_TIMEOUT = 6000; + + protected static final String FIRST_KAHADB_DIRECTORY = "target/activemq-data/first/"; + protected static final String SECOND_KAHADB_DIRECTORY = "target/activemq-data/second/"; + + protected String firstBindAddress = "vm://firstBroker"; + protected String firstReplicaBindAddress = "tcp://localhost:61610"; + protected String secondReplicaBindAddress = "tcp://localhost:61611"; + protected String secondBindAddress = "vm://secondBroker"; + + protected BrokerService firstBroker; + protected BrokerService secondBroker; + + protected boolean useTopic; + + protected ConnectionFactory firstBrokerConnectionFactory; + protected ConnectionFactory secondBrokerConnectionFactory; + + protected ActiveMQXAConnectionFactory firstBrokerXAConnectionFactory; + protected ActiveMQXAConnectionFactory secondBrokerXAConnectionFactory; + + protected ActiveMQDestination destination; + + private static long txGenerator = 67; + + @Override + protected void setUp() throws Exception { + if (firstBroker == null) { + firstBroker = createFirstBroker(); + } + if (secondBroker == null) { + secondBroker = createSecondBroker(); + } + + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + firstBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(firstBindAddress); + secondBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(secondBindAddress); + + destination = createDestination(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBroker != null) { + try { + firstBroker.stop(); + } catch (Exception e) { + } + } + if (secondBroker != null) { + try { + secondBroker.stop(); + } catch (Exception e) { + } + } + } + + protected BrokerService createFirstBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(firstBindAddress); + answer.setDataDirectory(FIRST_KAHADB_DIRECTORY); + answer.setBrokerName("firstBroker"); + + ReplicaPlugin replicaPlugin = new ReplicaPlugin(); + replicaPlugin.setRole(ReplicaRole.source); + replicaPlugin.setTransportConnectorUri(firstReplicaBindAddress); + replicaPlugin.setOtherBrokerUri(secondReplicaBindAddress); + replicaPlugin.setControlWebConsoleAccess(false); + replicaPlugin.setHeartBeatPeriod(0); + + answer.setPlugins(new BrokerPlugin[]{replicaPlugin}); + answer.setSchedulerSupport(true); + return answer; + } + + protected BrokerService createSecondBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(secondBindAddress); + answer.setDataDirectory(SECOND_KAHADB_DIRECTORY); + answer.setBrokerName("secondBroker"); + + ReplicaPlugin replicaPlugin = new ReplicaPlugin(); + replicaPlugin.setRole(ReplicaRole.replica); + replicaPlugin.setTransportConnectorUri(secondReplicaBindAddress); + replicaPlugin.setOtherBrokerUri(firstReplicaBindAddress); + replicaPlugin.setControlWebConsoleAccess(false); + replicaPlugin.setHeartBeatPeriod(0); + + answer.setPlugins(new BrokerPlugin[]{replicaPlugin}); + answer.setSchedulerSupport(true); + return answer; + } + + protected void startFirstBroker() throws Exception { + firstBroker.start(); + } + + protected void startSecondBroker() throws Exception { + secondBroker.start(); + } + + protected ActiveMQDestination createDestination() { + return createDestination(getDestinationString()); + } + + protected ActiveMQDestination createDestination(String subject) { + if (useTopic) { + return new ActiveMQTopic(subject); + } else { + return new ActiveMQQueue(subject); + } + } + + protected String getDestinationString() { + return getClass().getName() + "." + getName(); + } + + protected Xid createXid() throws IOException { + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + DataOutputStream os = new DataOutputStream(baos); + os.writeLong(++txGenerator); + os.close(); + final byte[] bs = baos.toByteArray(); + + return new Xid() { + + public int getFormatId() { + return 86; + } + + + public byte[] getGlobalTransactionId() { + return bs; + } + + + public byte[] getBranchQualifier() { + return bs; + } + }; + } + + protected QueueViewMBean getQueueView(BrokerService broker, String queueName) throws MalformedObjectNameException { + MBeanServer mbeanServer = broker.getManagementContext().getMBeanServer(); + String objectNameStr = broker.getBrokerObjectName().toString(); + objectNameStr += ",destinationType=Queue,destinationName="+queueName; + ObjectName queueViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + return MBeanServerInvocationHandler.newProxyInstance(mbeanServer, queueViewMBeanName, QueueViewMBean.class, true); + } + + protected TopicViewMBean getTopicView(BrokerService broker, String topicName) throws MalformedObjectNameException { + MBeanServer mbeanServer = broker.getManagementContext().getMBeanServer(); + String objectNameStr = broker.getBrokerObjectName().toString(); + objectNameStr += ",destinationType=Topic,destinationName=" + topicName; + ObjectName topicViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + return MBeanServerInvocationHandler.newProxyInstance(mbeanServer, topicViewMBeanName, TopicViewMBean.class, true); + } + + protected ReplicationViewMBean getReplicationView(BrokerService broker) throws Exception { + MBeanServer mbeanServer = broker.getManagementContext().getMBeanServer(); + String objectNameStr = broker.getBrokerObjectName().toString(); + objectNameStr += ",service=Plugins,instanceName=ReplicationPlugin"; + ObjectName replicaViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + return MBeanServerInvocationHandler.newProxyInstance(mbeanServer, replicaViewMBeanName, ReplicationViewMBean.class, true); + } + + protected ObjectName assertRegisteredObjectName(MBeanServer mbeanServer, String name) throws MalformedObjectNameException, NullPointerException { + ObjectName objectName = new ObjectName(name); + if (mbeanServer.isRegistered(objectName)) { + System.out.println("Bean Registered: " + objectName); + } else { + fail("Could not find MBean!: " + objectName); + } + return objectName; + } + + protected void cleanKahaDB(String filePath) throws IOException { + File kahaDBFile = new File(filePath); + if (kahaDBFile.exists()) { + FileUtils.cleanDirectory(kahaDBFile); + } + } + + protected void waitForCondition(Runnable condition) throws Exception { + assertTrue(Wait.waitFor(() -> { + try { + condition.run(); + return true; + } catch (Exception|Error e) { + e.printStackTrace(); + return false; + } + }, Wait.MAX_WAIT_MILLIS * 5)); + } + + protected void waitUntilReplicationQueueHasConsumer(BrokerService broker) throws Exception { + assertTrue("Replication Main Queue has Consumer", + Wait.waitFor(() -> { + try { + QueueViewMBean brokerMainQueueView = getQueueView(broker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + return brokerMainQueueView.getConsumerCount() > 0; + } catch (Exception e) { + e.printStackTrace(); + return false; + } + }, Wait.MAX_WAIT_MILLIS*2)); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTopicTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTopicTest.java new file mode 100644 index 00000000000..94aa10b8755 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginTopicTest.java @@ -0,0 +1,479 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.broker.jmx.BrokerViewMBean; +import org.apache.activemq.broker.jmx.TopicViewMBean; +import org.apache.activemq.command.ActiveMQTextMessage; + +import javax.jms.Connection; +import javax.jms.Destination; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TemporaryTopic; +import javax.jms.TextMessage; +import javax.jms.Topic; +import javax.jms.XAConnection; +import javax.jms.XASession; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.ObjectName; +import javax.transaction.xa.XAResource; +import javax.transaction.xa.Xid; +import java.util.Arrays; +import java.util.UUID; + +public class ReplicaPluginTopicTest extends ReplicaPluginTestSupport { + + private static final String CLIENT_ID_ONE = "one"; + private static final String CLIENT_ID_TWO = "two"; + private static final String CLIENT_ID_XA = "xa"; + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + protected Connection firstBrokerConnection2; + protected Connection secondBrokerConnection2; + + protected XAConnection firstBrokerXAConnection; + protected Connection secondBrokerXAConnection; + + @Override + protected void setUp() throws Exception { + useTopic = true; + + super.setUp(); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.setClientID(CLIENT_ID_ONE); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.setClientID(CLIENT_ID_ONE); + secondBrokerConnection.start(); + + firstBrokerConnection2 = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection2.setClientID(CLIENT_ID_TWO); + firstBrokerConnection2.start(); + + secondBrokerConnection2 = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection2.setClientID(CLIENT_ID_TWO); + secondBrokerConnection2.start(); + + firstBrokerXAConnection = firstBrokerXAConnectionFactory.createXAConnection(); + firstBrokerXAConnection.setClientID(CLIENT_ID_XA); + firstBrokerXAConnection.start(); + + secondBrokerXAConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerXAConnection.setClientID(CLIENT_ID_XA); + secondBrokerXAConnection.start(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + if (firstBrokerXAConnection != null) { + firstBrokerXAConnection.close(); + firstBrokerXAConnection = null; + } + if (secondBrokerXAConnection != null) { + secondBrokerXAConnection.close(); + secondBrokerXAConnection = null; + } + + super.tearDown(); + } + + public void testSendMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testAcknowledgeMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + firstBrokerConnection2.createSession(false, Session.CLIENT_ACKNOWLEDGE).createDurableSubscriber((Topic) destination, CLIENT_ID_TWO); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + firstBrokerSession.close(); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + secondBrokerSession.close(); + + firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + receivedMessage.acknowledge(); + + Thread.sleep(LONG_TIMEOUT); + + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + secondBrokerSession.close(); + + + secondBrokerSession = secondBrokerConnection2.createSession(false, Session.CLIENT_ACKNOWLEDGE); + secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_TWO); + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + secondBrokerSession.close(); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + + public void testSendMessageTransactionCommit() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(true, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.commit(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageTransactionRollback() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(true, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.rollback(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageXATransactionCommit() throws Exception { + XASession firstBrokerSession = firstBrokerXAConnection.createXASession(); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + XAResource xaRes = firstBrokerSession.getXAResource(); + Xid xid = createXid(); + xaRes.start(xid, XAResource.TMNOFLAGS); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerXAConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + xaRes.end(xid, XAResource.TMSUCCESS); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.prepare(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.commit(xid, false); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testSendMessageXATransactionRollback() throws Exception { + XASession firstBrokerSession = firstBrokerXAConnection.createXASession(); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + XAResource xaRes = firstBrokerSession.getXAResource(); + Xid xid = createXid(); + xaRes.start(xid, XAResource.TMNOFLAGS); + + TextMessage message = firstBrokerSession.createTextMessage(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT); + + Session secondBrokerSession = secondBrokerXAConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + xaRes.end(xid, XAResource.TMSUCCESS); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.prepare(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + xaRes.rollback(xid); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testExpireMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + firstBrokerProducer.setTimeToLive(LONG_TIMEOUT); + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Thread.sleep(LONG_TIMEOUT + SHORT_TIMEOUT); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + Message receivedMessage = secondBrokerConsumer.receive(SHORT_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testBrowseMessage() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + TopicViewMBean secondBrokerDestinationTopicView = getTopicView(secondBroker, destination.getPhysicalName()); + assertEquals(secondBrokerDestinationTopicView.browseMessages().size(), 1); + TextMessage destinationMessage = (TextMessage) secondBrokerDestinationTopicView.browseMessages().get(0); + assertEquals(destinationMessage.getText(), getName()); + + assertEquals(secondBrokerDestinationTopicView.getProducerCount(), 0); + assertEquals(secondBrokerDestinationTopicView.getConsumerCount(), 1); + + Message receivedMessage = firstBrokerConsumer.receive(SHORT_TIMEOUT); + assertNotNull(receivedMessage); + receivedMessage.acknowledge(); + Thread.sleep(LONG_TIMEOUT); + assertEquals(secondBrokerDestinationTopicView.getDequeueCount(), 1); + firstBrokerSession.close(); + } + + public void testDeleteTopic() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_XA); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + MBeanServer secondBrokerMbeanServer = secondBroker.getManagementContext().getMBeanServer(); + ObjectName secondBrokerViewMBeanName = assertRegisteredObjectName(secondBrokerMbeanServer, secondBroker.getBrokerObjectName().toString()); + BrokerViewMBean secondBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(secondBrokerMbeanServer, secondBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(Arrays.stream(secondBrokerMBean.getTopics()) + .map(ObjectName::toString) + .peek(name -> System.out.println("topic name: " + name)) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 1); + + MBeanServer firstBrokerMbeanServer = firstBroker.getManagementContext().getMBeanServer(); + ObjectName firstBrokerViewMBeanName = assertRegisteredObjectName(firstBrokerMbeanServer, firstBroker.getBrokerObjectName().toString()); + BrokerViewMBean firstBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(firstBrokerMbeanServer, firstBrokerViewMBeanName, BrokerViewMBean.class, true); + firstBrokerMBean.removeTopic(destination.getPhysicalName()); + Thread.sleep(LONG_TIMEOUT); + + assertEquals(Arrays.stream(secondBrokerMBean.getQueues()) + .map(ObjectName::toString) + .filter(name -> name.contains("destinationName=" + destination.getPhysicalName())) + .count(), 0); + + firstBrokerSession.close(); + } + + public void testDurableSubscribers() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumerOne = firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + "No. 1"); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + TopicViewMBean secondBrokerDestinationTopicView = getTopicView(secondBroker, destination.getPhysicalName()); + TopicViewMBean firstBrokerDestinationTopicView = getTopicView(firstBroker, destination.getPhysicalName()); + assertEquals(firstBrokerDestinationTopicView.getConsumerCount(), 1); + assertEquals(secondBrokerDestinationTopicView.getConsumerCount(), 1); + + + firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_TWO); + message = new ActiveMQTextMessage(); + message.setText(getName() + "No. 2"); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + assertEquals(secondBrokerDestinationTopicView.getConsumerCount(), 2); + assertEquals(firstBrokerDestinationTopicView.getConsumerCount(), 2); + + firstBrokerConsumerOne.close(); + firstBrokerSession.unsubscribe(CLIENT_ID_ONE); + message = new ActiveMQTextMessage(); + message.setText(getName() + "No. 3"); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + assertEquals(firstBrokerDestinationTopicView.getConsumerCount(), 1); + assertEquals(secondBrokerDestinationTopicView.getConsumerCount(), 1); + firstBrokerSession.close(); + } + + public void testTemporaryTopicIsNotReplicated() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createDurableSubscriber((Topic) destination, CLIENT_ID_ONE); + + TemporaryTopic temporaryTopic = firstBrokerSession.createTemporaryTopic(); + MessageConsumer firstBrokerTempTopicConsumer = firstBrokerSession.createConsumer(temporaryTopic); + TextMessage message = firstBrokerSession.createTextMessage(getName()); + String id = UUID.randomUUID().toString(); + message.setJMSReplyTo(temporaryTopic); + message.setJMSCorrelationID(id); + firstBrokerProducer.send(message); + Thread.sleep(LONG_TIMEOUT); + + Message firstBrokerMessageDestinationReceived = firstBrokerConsumer.receive(LONG_TIMEOUT); + if (firstBrokerMessageDestinationReceived instanceof TextMessage) { + TextMessage textMessage = (TextMessage) firstBrokerMessageDestinationReceived; + Destination replyBackTopic = textMessage.getJMSReplyTo(); + MessageProducer producer = firstBrokerSession.createProducer(replyBackTopic); + + TextMessage msg = firstBrokerSession.createTextMessage("Message Received : " + textMessage.getText()); + producer.send(msg); + } + + Message firstBrokerMessageReceived = firstBrokerTempTopicConsumer.receive(LONG_TIMEOUT); + assertNotNull(firstBrokerMessageReceived); + assertTrue(((TextMessage) firstBrokerMessageReceived).getText().contains(getName())); + + String tempTopicJMXName = temporaryTopic.getTopicName().replaceAll(":", "_"); + MBeanServer firstBrokerMbeanServer = firstBroker.getManagementContext().getMBeanServer(); + ObjectName firstBrokerViewMBeanName = assertRegisteredObjectName(firstBrokerMbeanServer, firstBroker.getBrokerObjectName().toString()); + BrokerViewMBean firstBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(firstBrokerMbeanServer, firstBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(firstBrokerMBean.getTemporaryTopics().length, 1); + assertTrue(firstBrokerMBean.getTemporaryTopics()[0].toString().contains(tempTopicJMXName)); + + MBeanServer secondBrokerMbeanServer = secondBroker.getManagementContext().getMBeanServer(); + ObjectName secondBrokerViewMBeanName = assertRegisteredObjectName(secondBrokerMbeanServer, secondBroker.getBrokerObjectName().toString()); + BrokerViewMBean secondBrokerMBean = MBeanServerInvocationHandler.newProxyInstance(secondBrokerMbeanServer, secondBrokerViewMBeanName, BrokerViewMBean.class, true); + assertEquals(secondBrokerMBean.getTemporaryTopics().length, 0); + + firstBrokerSession.close(); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginVirtualDestinationTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginVirtualDestinationTest.java new file mode 100644 index 00000000000..8c4566aa5aa --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaPluginVirtualDestinationTest.java @@ -0,0 +1,237 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.broker.jmx.TopicViewMBean; +import org.apache.activemq.broker.region.DestinationInterceptor; +import org.apache.activemq.broker.region.virtual.CompositeQueue; +import org.apache.activemq.broker.region.virtual.VirtualDestination; +import org.apache.activemq.broker.region.virtual.VirtualDestinationInterceptor; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.command.ActiveMQTopic; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.management.MBeanServer; +import javax.management.ObjectName; +import java.util.Arrays; +import java.util.Collections; + +public class ReplicaPluginVirtualDestinationTest extends ReplicaPluginTestSupport { + + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + private static final String CLIENT_ID_ONE = "one"; + + private static final String VIRTUAL_QUEUE = "VIRT.QUEUE"; + private static final String PHYSICAL_QUEUE = "VQueue"; + private static final String PHYSICAL_TOPIC = "VTopic"; + + private static final String VIRTUAL_QUEUE_FIRST_BROKER = "VIRT.QUEUE1"; + private static final String PHYSICAL_QUEUE_FIRST_BROKER = "VQueueFirst"; + + private static final String VIRTUAL_QUEUE_SECOND_BROKER = "VIRT.QUEUE2"; + private static final String PHYSICAL_QUEUE_SECOND_BROKER = "VQueueSecond"; + + private void setupCompositeDestinationsBothBrokers(BrokerService firstBroker, BrokerService secondBroker) { + CompositeQueue virtualDestination = new CompositeQueue(); + virtualDestination.setName(VIRTUAL_QUEUE); + virtualDestination.setForwardOnly(true); + virtualDestination.setForwardTo(Arrays.asList(new ActiveMQQueue(PHYSICAL_QUEUE), new ActiveMQTopic(PHYSICAL_TOPIC))); + + VirtualDestinationInterceptor virtualDestinationInterceptor = new VirtualDestinationInterceptor(); + virtualDestinationInterceptor.setVirtualDestinations(Collections.singletonList(virtualDestination).toArray(VirtualDestination[]::new)); + DestinationInterceptor[] interceptors = Collections.singletonList(virtualDestinationInterceptor).toArray(DestinationInterceptor[]::new); + + firstBroker.setDestinationInterceptors(interceptors); + secondBroker.setDestinationInterceptors(interceptors); + } + + private void setupCompositeDestinationsOneBrokerOnly(BrokerService broker, String virtualQueue, String physicalQueue) { + CompositeQueue virtualDestination = new CompositeQueue(); + virtualDestination.setName(virtualQueue); + virtualDestination.setForwardOnly(true); + virtualDestination.setForwardTo(Collections.singletonList(new ActiveMQQueue(physicalQueue))); + + VirtualDestinationInterceptor virtualDestinationInterceptor = new VirtualDestinationInterceptor(); + virtualDestinationInterceptor.setVirtualDestinations(Collections.singletonList(virtualDestination).toArray(VirtualDestination[]::new)); + DestinationInterceptor[] interceptors = broker.getDestinationInterceptors(); + interceptors = Arrays.copyOf(interceptors, interceptors.length + 1); + interceptors[interceptors.length - 1] = virtualDestinationInterceptor; + + broker.setDestinationInterceptors(interceptors); + } + + @Override + protected void setUp() throws Exception { + + if (firstBroker == null) { + firstBroker = createFirstBroker(); + } + if (secondBroker == null) { + secondBroker = createSecondBroker(); + } + + setupCompositeDestinationsBothBrokers(firstBroker, secondBroker); + setupCompositeDestinationsOneBrokerOnly(firstBroker, VIRTUAL_QUEUE_FIRST_BROKER, PHYSICAL_QUEUE_FIRST_BROKER); + setupCompositeDestinationsOneBrokerOnly(secondBroker, VIRTUAL_QUEUE_SECOND_BROKER, PHYSICAL_QUEUE_SECOND_BROKER); + + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.setClientID(CLIENT_ID_ONE); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.setClientID(CLIENT_ID_ONE); + secondBrokerConnection.start(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + public void testVirtualDestinationConfigurationBothBrokers() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + ActiveMQQueue virtualQueue = new ActiveMQQueue(VIRTUAL_QUEUE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(virtualQueue); + + ActiveMQTopic physicalTopic = new ActiveMQTopic(PHYSICAL_TOPIC); + firstBrokerSession.createDurableSubscriber(physicalTopic, CLIENT_ID_ONE); + + ActiveMQQueue physicalQueue = new ActiveMQQueue(PHYSICAL_QUEUE); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumerQueue = secondBrokerSession.createConsumer(physicalQueue); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessageFromQueue = secondBrokerConsumerQueue.receive(LONG_TIMEOUT); + assertNotNull(receivedMessageFromQueue); + assertTrue(receivedMessageFromQueue instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessageFromQueue).getText()); + + MessageConsumer secondBrokerConsumerTopic = secondBrokerSession.createDurableSubscriber(physicalTopic, CLIENT_ID_ONE); + Message receivedMessageFromTopic = secondBrokerConsumerTopic.receive(); + assertNotNull(receivedMessageFromTopic); + assertTrue(receivedMessageFromTopic instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessageFromTopic).getText()); + + QueueViewMBean secondBrokerVirtualQueueViewMBean = getQueueView(secondBroker, virtualQueue.getPhysicalName()); + assertEquals(secondBrokerVirtualQueueViewMBean.getEnqueueCount(), 0); + + QueueViewMBean secondBrokerPhysicalQueueViewMBean = getQueueView(secondBroker, physicalQueue.getPhysicalName()); + assertEquals(secondBrokerPhysicalQueueViewMBean.getEnqueueCount(), 1); + + TopicViewMBean secondBrokerPhysicalTopicViewMBean = getTopicView(secondBroker, physicalTopic.getPhysicalName()); + assertEquals(secondBrokerPhysicalTopicViewMBean.getEnqueueCount(), 1); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testVirtualDestinationConfigurationFirstBrokerOnly() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + ActiveMQQueue virtualQueue = new ActiveMQQueue(VIRTUAL_QUEUE_FIRST_BROKER); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(virtualQueue); + + ActiveMQQueue physicalQueue = new ActiveMQQueue(PHYSICAL_QUEUE_FIRST_BROKER); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(physicalQueue); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT * 2); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + // despite the fact that we don't have a virtual configuration on the replica broker + // the state should be the same as on the source broker + QueueViewMBean secondBrokerVirtualQueueViewMBean = getQueueView(secondBroker, virtualQueue.getPhysicalName()); + assertEquals(secondBrokerVirtualQueueViewMBean.getEnqueueCount(), 0); + + QueueViewMBean secondBrokerPhysicalQueueViewMBean = getQueueView(secondBroker, physicalQueue.getPhysicalName()); + assertEquals(secondBrokerPhysicalQueueViewMBean.getEnqueueCount(), 1); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + public void testVirtualDestinationConfigurationSecondBrokerOnly() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + // on the first broker the destination is physical! but on the second it is virtual as per config + ActiveMQQueue virtualQueue = new ActiveMQQueue(VIRTUAL_QUEUE_SECOND_BROKER); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(virtualQueue); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(virtualQueue); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + // on the replica side it should be treated like physical despite the virtual configuration + QueueViewMBean secondBrokerVirtualQueueViewMBean = getQueueView(secondBroker, virtualQueue.getPhysicalName()); + assertEquals(secondBrokerVirtualQueueViewMBean.getEnqueueCount(), 1); + + // that is why virtual queue on the replica shouldn't forward to the physical destination as per configuration + ActiveMQQueue physicalQueue = new ActiveMQQueue(PHYSICAL_QUEUE_SECOND_BROKER); + String objectNameStr = secondBroker.getBrokerObjectName().toString(); + objectNameStr += ",destinationType=Queue,destinationName=" + physicalQueue.getPhysicalName(); + ObjectName objectName = new ObjectName(objectNameStr); + MBeanServer mbeanServer = secondBroker.getManagementContext().getMBeanServer(); + assertFalse(mbeanServer.isRegistered(objectName)); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolConnectionTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolConnectionTest.java new file mode 100644 index 00000000000..0947bfc26b3 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolConnectionTest.java @@ -0,0 +1,180 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQSslConnectionFactory; +import org.apache.activemq.TestSupport; +import org.apache.activemq.broker.BrokerFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import java.io.IOException; +import java.net.URI; +import java.net.URISyntaxException; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Arrays; +import java.util.Collection; + +import static org.apache.activemq.broker.replica.ReplicaPluginTestSupport.LONG_TIMEOUT; + +@RunWith(Parameterized.class) +public class ReplicaProtocolConnectionTest extends TestSupport { + + private static final Logger LOG = LoggerFactory.getLogger(ReplicaProtocolConnectionTest.class); + public static final String KEYSTORE_TYPE = "jks"; + public static final String PASSWORD = "password"; + public static final String SERVER_KEYSTORE = "src/test/resources/server.keystore"; + public static final String TRUST_KEYSTORE = "src/test/resources/client.keystore"; + public static final String PRIMARY_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-primary.xml"; + public static final String REPLICA_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-replica.xml"; + protected static final String SECOND_BROKER_BINDING_ADDRESS = "vm://secondBrokerLocalhost"; + private static final DateFormat df = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss.S"); + private final String protocol; + protected BrokerService firstBroker; + protected BrokerService secondBroker; + protected ActiveMQDestination destination; + + @Before + public void setUp() throws Exception { + firstBroker = setUpBrokerService(PRIMARY_BROKER_CONFIG); + secondBroker = setUpBrokerService(REPLICA_BROKER_CONFIG); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + destination = new ActiveMQQueue(getClass().getName() + "." + getName()); + } + + @After + public void tearDown() throws Exception { + if (firstBroker != null) { + try { + firstBroker.stop(); + firstBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + if (secondBroker != null) { + try { + secondBroker.stop(); + secondBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + } + + @Parameterized.Parameters(name="protocol={0}") + public static Collection getTestParameters() { + return Arrays.asList(new String[][] { + {"auto"}, {"auto+ssl"}, {"auto+nio+ssl"}, {"auto+nio"}, + {"tcp"}, {"ssl"}, {"nio+ssl"}, {"nio"} + }); + } + + static { + System.setProperty("javax.net.ssl.trustStore", TRUST_KEYSTORE); + System.setProperty("javax.net.ssl.trustStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.trustStoreType", KEYSTORE_TYPE); + System.setProperty("javax.net.ssl.keyStore", SERVER_KEYSTORE); + System.setProperty("javax.net.ssl.keyStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.keyStoreType", KEYSTORE_TYPE); + } + + @Test + public void testBrokerConnection() throws Exception { + Connection firstBrokerConnection = getClientConnectionFactory(firstBroker.getTransportConnectorByScheme(protocol)).createConnection(); + Connection secondBrokerConnection = new ActiveMQConnectionFactory(SECOND_BROKER_BINDING_ADDRESS).createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + receivedMessage.acknowledge(); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerSession.close(); + secondBrokerSession.close(); + firstBrokerConnection.close(); + secondBrokerConnection.close(); + } + + private ActiveMQConnectionFactory getClientConnectionFactory(TransportConnector connector) throws IOException, URISyntaxException { + String connectionUri = protocol + "://localhost:" + connector.getConnectUri().getPort(); + if (protocol.contains("ssl")) { + return new ActiveMQSslConnectionFactory(connectionUri); + } else { + return new ActiveMQConnectionFactory(connectionUri); + } + } + + + + public ReplicaProtocolConnectionTest(String protocol) { + this.protocol = protocol; + } + + protected BrokerService setUpBrokerService(String configurationUri) throws Exception { + BrokerService broker = createBroker(configurationUri); + broker.setPersistent(false); + return broker; + } + + protected BrokerService createBroker(String uri) throws Exception { + LOG.info("Loading broker configuration from the classpath with URI: " + uri); + return BrokerFactory.createBroker(new URI("xbean:" + uri)); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolStompConnectionTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolStompConnectionTest.java new file mode 100644 index 00000000000..319fb1ab9b7 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaProtocolStompConnectionTest.java @@ -0,0 +1,188 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.TestSupport; +import org.apache.activemq.broker.BrokerFactory; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.transport.stomp.Stomp; +import org.apache.activemq.transport.stomp.StompConnection; +import org.apache.activemq.transport.stomp.StompFrame; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import javax.jms.Connection; +import javax.jms.ConnectionFactory; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.net.SocketFactory; +import javax.net.ssl.SSLSocketFactory; +import java.net.Socket; +import java.net.URI; +import java.text.DateFormat; +import java.text.SimpleDateFormat; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.UUID; + +import static org.apache.activemq.broker.replica.ReplicaPluginTestSupport.LONG_TIMEOUT; + +@RunWith(Parameterized.class) +public class ReplicaProtocolStompConnectionTest extends TestSupport { + + private static final Logger LOG = LoggerFactory.getLogger(ReplicaProtocolStompConnectionTest.class); + public static final String KEYSTORE_TYPE = "jks"; + public static final String PASSWORD = "password"; + public static final String SERVER_KEYSTORE = "src/test/resources/server.keystore"; + public static final String TRUST_KEYSTORE = "src/test/resources/client.keystore"; + public static final String PRIMARY_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-primary.xml"; + public static final String REPLICA_BROKER_CONFIG = "org/apache/activemq/broker/replica/transport-protocol-test-replica.xml"; + protected static final String SECOND_BROKER_BINDING_ADDRESS = "vm://secondBrokerLocalhost"; + private static final DateFormat df = new SimpleDateFormat("dd.MM.yyyy HH:mm:ss.S"); + private final String protocol; + protected BrokerService firstBroker; + protected BrokerService secondBroker; + private StompConnection firstBrokerConnection; + private ConnectionFactory secondBrokerConnectionFactory; + private Connection secondBrokerConnection; + + @Before + public void setUp() throws Exception { + firstBroker = setUpBrokerService(PRIMARY_BROKER_CONFIG); + secondBroker = setUpBrokerService(REPLICA_BROKER_CONFIG); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + firstBrokerConnection = new StompConnection(); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(SECOND_BROKER_BINDING_ADDRESS); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + } + + @After + public void tearDown() throws Exception { + firstBrokerConnection.disconnect(); + secondBrokerConnection.stop(); + if (firstBroker != null) { + try { + firstBroker.stop(); + firstBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + if (secondBroker != null) { + try { + secondBroker.stop(); + secondBroker.waitUntilStopped(); + } catch (Exception e) { + } + } + } + + @Parameterized.Parameters(name="protocol={0}") + public static Collection getTestParameters() { + return Arrays.asList(new String[][] { + {"stomp"}, {"stomp+ssl"}, {"stomp+nio+ssl"}, {"stomp+nio"}, + }); + } + + static { + System.setProperty("javax.net.ssl.trustStore", TRUST_KEYSTORE); + System.setProperty("javax.net.ssl.trustStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.trustStoreType", KEYSTORE_TYPE); + System.setProperty("javax.net.ssl.keyStore", SERVER_KEYSTORE); + System.setProperty("javax.net.ssl.keyStorePassword", PASSWORD); + System.setProperty("javax.net.ssl.keyStoreType", KEYSTORE_TYPE); + } + + @Test + public void testMessageSendAndReceive() throws Exception { + startConnection(firstBroker.getTransportConnectorByScheme(protocol), firstBrokerConnection); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + + String type = "queue"; + String body = "testMessageSendAndReceiveOnPrimarySide body"; + String destination = "ReplicaProtocolStompConnectionTestQueue"; + + firstBrokerConnection.begin("tx1"); + String message = String.format("[%s://%s] %s", type, destination, body); + HashMap headers = new HashMap<>(); + headers.put("persistent", "true"); + firstBrokerConnection.send(String.format("/%s/%s", type, destination), message, "tx1", headers); + firstBrokerConnection.commit("tx1"); + Thread.sleep(LONG_TIMEOUT); + + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(new ActiveMQQueue(destination)); + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(message, ((TextMessage) receivedMessage).getText()); + + firstBrokerConnection.subscribe(String.format("/%s/%s", type, destination), Stomp.Headers.Subscribe.AckModeValues.AUTO); + StompFrame receivedStompMessage = firstBrokerConnection.receive(LONG_TIMEOUT); + assertNotNull(receivedStompMessage); + assertEquals(message, receivedStompMessage.getBody()); + + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerSession.close(); + } + + private void startConnection(TransportConnector connector, StompConnection brokerConnection) throws Exception { + if (protocol.contains("ssl")) { + SocketFactory factory = SSLSocketFactory.getDefault(); + Socket socket = factory.createSocket("localhost", connector.getConnectUri().getPort()); + brokerConnection.open(socket); + brokerConnection.connect(null, null, UUID.randomUUID().toString()); + } else { + brokerConnection.open("localhost", connector.getConnectUri().getPort()); + brokerConnection.connect(null, null, UUID.randomUUID().toString()); + } + } + + public ReplicaProtocolStompConnectionTest(String protocol) { + this.protocol = protocol; + } + + protected BrokerService setUpBrokerService(String configurationUri) throws Exception { + BrokerService broker = createBroker(configurationUri); + broker.setAdvisorySupport(false); + broker.setSchedulerSupport(false); + return broker; + } + + protected BrokerService createBroker(String uri) throws Exception { + LOG.info("Loading broker configuration from the classpath with URI: " + uri); + return BrokerFactory.createBroker(new URI("xbean:" + uri)); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaRedeliveryPluginTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaRedeliveryPluginTest.java new file mode 100644 index 00000000000..cfdce8aaa33 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaRedeliveryPluginTest.java @@ -0,0 +1,196 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.RedeliveryPolicy; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.region.policy.RedeliveryPolicyMap; +import org.apache.activemq.broker.region.policy.SharedDeadLetterStrategy; +import org.apache.activemq.broker.util.RedeliveryPlugin; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQQueue; +import org.junit.Test; + +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import java.util.List; +import java.util.concurrent.TimeUnit; + +public class ReplicaRedeliveryPluginTest extends ReplicaPluginTestSupport { + protected ActiveMQConnection firstBrokerConnection; + protected ActiveMQConnection secondBrokerConnection; + final long redeliveryDelayMillis = 2000; + long initialRedeliveryDelayMillis = 4000; + int maxBrokerRedeliveries = 2; + @Override + protected void setUp() throws Exception { + firstBroker = createFirstBroker(); + secondBroker = createSecondBroker(); + firstBroker.setSchedulerSupport(true); + secondBroker.setSchedulerSupport(true); + destination = createDestination(); + + RedeliveryPlugin redeliveryPlugin = new RedeliveryPlugin(); + RedeliveryPolicy redeliveryPolicy = new RedeliveryPolicy(); + redeliveryPolicy.setQueue(destination.getPhysicalName()); + redeliveryPolicy.setRedeliveryDelay(redeliveryDelayMillis); + redeliveryPolicy.setInitialRedeliveryDelay(initialRedeliveryDelayMillis); + redeliveryPolicy.setMaximumRedeliveries(maxBrokerRedeliveries); + + RedeliveryPolicy defaultPolicy = new RedeliveryPolicy(); + defaultPolicy.setRedeliveryDelay(1000); + defaultPolicy.setInitialRedeliveryDelay(1000); + defaultPolicy.setMaximumRedeliveries(0); + + RedeliveryPolicyMap redeliveryPolicyMap = new RedeliveryPolicyMap(); + redeliveryPolicyMap.setRedeliveryPolicyEntries(List.of(redeliveryPolicy)); + redeliveryPolicyMap.setDefaultEntry(defaultPolicy); + redeliveryPlugin.setRedeliveryPolicyMap(redeliveryPolicyMap); + redeliveryPlugin.setFallbackToDeadLetter(true); + redeliveryPlugin.setSendToDlqIfMaxRetriesExceeded(true); + + BrokerPlugin firstBrokerReplicaPlugin = firstBroker.getPlugins()[0]; + firstBroker.setPlugins(new BrokerPlugin[]{redeliveryPlugin, firstBrokerReplicaPlugin}); + startFirstBroker(); + startSecondBroker(); + + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + firstBrokerConnection = (ActiveMQConnection) firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = (ActiveMQConnection) secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + @Test + public void testMessageRedelivery() throws Exception { + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + RedeliveryPolicy firstBrokerRedeliveryPolicy = new RedeliveryPolicy(); + firstBrokerRedeliveryPolicy.setInitialRedeliveryDelay(0); + firstBrokerRedeliveryPolicy.setMaximumRedeliveries(0); + ActiveMQConnection firstBrokerConsumerConnection = (ActiveMQConnection) firstBrokerConnectionFactory.createConnection(); + firstBrokerConsumerConnection.setRedeliveryPolicy(firstBrokerRedeliveryPolicy); + firstBrokerConsumerConnection.start(); + Session producerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer producer = producerSession.createProducer(destination); + + Message message = producerSession.createMessage(); + message.setStringProperty("data", getName()); + producer.send(message); + + Session consumerSession = firstBrokerConsumerConnection.createSession(true, Session.SESSION_TRANSACTED); + MessageConsumer firstBrokerConsumer = consumerSession.createConsumer(destination); + + Message secondBrokerReceivedMsg = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull("second broker got message", secondBrokerReceivedMsg); + + Message receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull("got message", receivedMessage); + consumerSession.rollback(); + + for (int i=0; i < maxBrokerRedeliveries; i++) { + Message shouldBeNullMessage = firstBrokerConsumer.receive(redeliveryDelayMillis / 4); + assertNull(shouldBeNullMessage); + shouldBeNullMessage = secondBrokerConsumer.receive(redeliveryDelayMillis / 4); + assertNull(shouldBeNullMessage); + TimeUnit.SECONDS.sleep(4); + + Message brokerRedeliveryMessage = firstBrokerConsumer.receive(1500); + assertNotNull("got message via broker redelivery after delay", brokerRedeliveryMessage); + assertEquals("message matches", message.getStringProperty("data"), brokerRedeliveryMessage.getStringProperty("data")); + System.out.println("received message: " + brokerRedeliveryMessage); + assertEquals("has expiryDelay specified - iteration:" + i, i == 0 ? initialRedeliveryDelayMillis : redeliveryDelayMillis, brokerRedeliveryMessage.getLongProperty(RedeliveryPlugin.REDELIVERY_DELAY)); + + brokerRedeliveryMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull("should not receive message", brokerRedeliveryMessage); + + consumerSession.rollback(); + } + + producerSession.close(); + secondBrokerSession.close(); + firstBrokerConsumerConnection.close(); + } + + @Test + public void testMessageDeliveredToDlq() throws Exception { + ActiveMQDestination testDestination = new ActiveMQQueue(getName()); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(testDestination); + RedeliveryPolicy firstBrokerRedeliveryPolicy = new RedeliveryPolicy(); + firstBrokerRedeliveryPolicy.setInitialRedeliveryDelay(0); + firstBrokerRedeliveryPolicy.setMaximumRedeliveries(0); + ActiveMQConnection firstBrokerConsumerConnection = (ActiveMQConnection) firstBrokerConnectionFactory.createConnection(); + firstBrokerConsumerConnection.setRedeliveryPolicy(firstBrokerRedeliveryPolicy); + firstBrokerConsumerConnection.start(); + Session producerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer producer = producerSession.createProducer(testDestination); + + Message message = producerSession.createMessage(); + message.setStringProperty("data", getName()); + producer.send(message); + + Session consumerSession = firstBrokerConsumerConnection.createSession(true, Session.SESSION_TRANSACTED); + MessageConsumer firstBrokerConsumer = consumerSession.createConsumer(testDestination); + + Message secondBrokerReceivedMsg = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull("second broker got message", secondBrokerReceivedMsg); + + Message receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull("got message", receivedMessage); + consumerSession.rollback(); + + MessageConsumer firstDlqConsumer = consumerSession.createConsumer(new ActiveMQQueue(SharedDeadLetterStrategy.DEFAULT_DEAD_LETTER_QUEUE_NAME)); + Message dlqMessage = firstDlqConsumer.receive(SHORT_TIMEOUT); + assertNotNull("Got message from dql", dlqMessage); + assertEquals("message matches", message.getStringProperty("data"), dlqMessage.getStringProperty("data")); + + MessageConsumer secondDlqConsumer = secondBrokerSession.createConsumer(new ActiveMQQueue(SharedDeadLetterStrategy.DEFAULT_DEAD_LETTER_QUEUE_NAME)); + dlqMessage = secondDlqConsumer.receive(LONG_TIMEOUT); + assertNotNull("Got message from dql", dlqMessage); + assertEquals("message matches", message.getStringProperty("data"), dlqMessage.getStringProperty("data")); + consumerSession.commit(); + + producerSession.close(); + secondBrokerSession.close(); + consumerSession.close(); + firstBrokerConsumerConnection.close(); + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaSoftFailoverTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaSoftFailoverTest.java new file mode 100644 index 00000000000..7da60a75bfb --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicaSoftFailoverTest.java @@ -0,0 +1,358 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQDestination; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.replica.jmx.ReplicationViewMBean; +import org.apache.activemq.util.Wait; +import org.junit.Ignore; +import org.junit.Test; + +import javax.jms.Connection; +import javax.jms.Message; +import javax.jms.MessageConsumer; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; +import javax.management.MBeanServer; +import javax.management.MBeanServerInvocationHandler; +import javax.management.MalformedObjectNameException; +import javax.management.ObjectName; + +public class ReplicaSoftFailoverTest extends ReplicaPluginTestSupport { + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + private ReplicationViewMBean firstBrokerReplicationView; + private ReplicationViewMBean secondBrokerReplicationView; + protected static String SECOND_REPLICA_BINDING_ADDRESS = "tcp://localhost:61611"; + private static int MESSAGES_TO_SEND = 500; + private static int MAX_RETRY = 10; + + @Override + protected void setUp() throws Exception { + firstBroker = setUpFirstBroker(); + secondBroker = setUpSecondBroker(); + + ReplicaPlugin firstBrokerPlugin = new ReplicaPlugin(); + firstBrokerPlugin.setRole(ReplicaRole.source); + firstBrokerPlugin.setTransportConnectorUri(firstReplicaBindAddress); + firstBrokerPlugin.setOtherBrokerUri(SECOND_REPLICA_BINDING_ADDRESS); + firstBrokerPlugin.setControlWebConsoleAccess(false); + firstBrokerPlugin.setHeartBeatPeriod(0); + firstBroker.setPlugins(new BrokerPlugin[]{firstBrokerPlugin}); + + ReplicaPlugin secondBrokerPlugin = new ReplicaPlugin(); + secondBrokerPlugin.setRole(ReplicaRole.replica); + secondBrokerPlugin.setTransportConnectorUri(SECOND_REPLICA_BINDING_ADDRESS); + secondBrokerPlugin.setOtherBrokerUri(firstReplicaBindAddress); + secondBrokerPlugin.setControlWebConsoleAccess(false); + secondBrokerPlugin.setHeartBeatPeriod(0); + secondBroker.setPlugins(new BrokerPlugin[]{secondBrokerPlugin}); + + firstBroker.start(); + secondBroker.start(); + firstBroker.waitUntilStarted(); + secondBroker.waitUntilStarted(); + + firstBrokerReplicationView = getReplicationViewMBean(firstBroker); + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + + destination = createDestination(); + + waitUntilReplicationQueueHasConsumer(firstBroker); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + super.tearDown(); + } + + @Test + public void testSoftFailover() throws Exception { + firstBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), false); + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + firstBrokerReplicationView = getReplicationViewMBean(firstBroker); + return firstBrokerReplicationView.getReplicationRole().equals(ReplicaRole.replica.name()); + } + }, Wait.MAX_WAIT_MILLIS*2); + + Thread.sleep(SHORT_TIMEOUT); + assertFalse(secondBroker.isStopped()); + waitUntilReplicationQueueHasConsumer(secondBroker); + + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); +// TODO: fix this +// assertEquals(ReplicaRole.source, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerProducer.send(message); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + receivedMessage.acknowledge(); + + firstBrokerSession.close(); + secondBrokerSession.close(); + firstBrokerConnection.stop(); + secondBrokerConnection.stop(); + } + + @Test + public void testPutMessagesBeforeFailover() throws Exception { + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + + int retryCounter = 1; + QueueViewMBean firstBrokerIntermediateQueueView = getQueueView(firstBroker, ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + while (firstBrokerIntermediateQueueView.getInFlightCount() <= 1) { + sendMessages(firstBrokerProducer, MESSAGES_TO_SEND * retryCounter); + retryCounter++; + if (retryCounter == MAX_RETRY) { + fail(String.format("MAX RETRY [%d] times reached! Failed to put load onto source broker!", MAX_RETRY)); + } + } + + firstBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), false); + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + firstBrokerReplicationView = getReplicationViewMBean(firstBroker); + return firstBrokerReplicationView.getReplicationRole().equals(ReplicaRole.replica.name()); + } + }, Wait.MAX_WAIT_MILLIS*2); + + Thread.sleep(SHORT_TIMEOUT); + assertFalse(secondBroker.isStopped()); + + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + waitUntilReplicationQueueHasConsumer(secondBroker); + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + ActiveMQDestination destination2 = createDestination(getDestinationString() + "No.2"); + + + firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + firstBrokerProducer = firstBrokerSession.createProducer(destination2); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination2); + + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination2); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination2); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + + Message receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + secondBrokerProducer.send(message); + receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + receivedMessage.acknowledge(); + + firstBrokerSession.close(); + secondBrokerSession.close(); + firstBrokerConnection.stop(); + secondBrokerConnection.stop(); + } + + @Ignore + @Test + public void doubleFailover() throws Exception { + firstBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), false); + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + firstBrokerReplicationView = getReplicationViewMBean(firstBroker); + return firstBrokerReplicationView.getReplicationRole().equals(ReplicaRole.replica.name()); + } + }, Wait.MAX_WAIT_MILLIS*2); + + Thread.sleep(SHORT_TIMEOUT); + assertFalse(secondBroker.isStopped()); + waitUntilReplicationQueueHasConsumer(secondBroker); + + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(firstBrokerReplicationView.getReplicationRole())); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.AUTO_ACKNOWLEDGE); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination); + int retryCounter = 1; + QueueViewMBean secondBrokerIntermediateQueueView = getQueueView(secondBroker, ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + while (secondBrokerIntermediateQueueView.getInFlightCount() <= 1) { + sendMessages(secondBrokerProducer, MESSAGES_TO_SEND * retryCounter); + retryCounter++; + if (retryCounter == MAX_RETRY) { + fail(String.format("MAX RETRY [%d] times reached! Failed to put load onto source broker!", MAX_RETRY)); + } + } + + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + secondBrokerReplicationView.setReplicationRole(ReplicaRole.replica.name(), false); + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + secondBrokerReplicationView = getReplicationViewMBean(secondBroker); + return secondBrokerReplicationView.getReplicationRole().equals(ReplicaRole.replica.name()); + } + }, Wait.MAX_WAIT_MILLIS*2); + + Thread.sleep(SHORT_TIMEOUT); + assertFalse(firstBroker.isStopped()); + secondBrokerReplicationView = getReplicationView(secondBroker); + assertEquals(ReplicaRole.replica, ReplicaRole.valueOf(secondBrokerReplicationView.getReplicationRole())); + waitUntilReplicationQueueHasConsumer(firstBroker); + + // firstBroker now is primary + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + ActiveMQDestination destination2 = createDestination(getDestinationString() + "No.2"); + + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination2); + MessageConsumer firstBrokerConsumer = firstBrokerSession.createConsumer(destination2); + + secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageConsumer secondBrokerConsumer = secondBrokerSession.createConsumer(destination2); + secondBrokerProducer = secondBrokerSession.createProducer(destination2); + + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + secondBrokerProducer.send(message); + + Message receivedMessage = firstBrokerConsumer.receive(LONG_TIMEOUT); + assertNull(receivedMessage); + + firstBrokerProducer.send(message); + receivedMessage = secondBrokerConsumer.receive(LONG_TIMEOUT); + assertNotNull(receivedMessage); + assertTrue(receivedMessage instanceof TextMessage); + assertEquals(getName(), ((TextMessage) receivedMessage).getText()); + receivedMessage.acknowledge(); + + firstBrokerSession.close(); + secondBrokerSession.close(); + firstBrokerConnection.stop(); + secondBrokerConnection.stop(); + + } + + private void sendMessages(MessageProducer producer, int messagesToSend) throws Exception { + for (int i = 0; i < messagesToSend; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName() + " No. " + i); + producer.send(message); + } + } + + private BrokerService setUpSecondBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(secondBindAddress); + answer.setDataDirectory(SECOND_KAHADB_DIRECTORY); + answer.setBrokerName("secondBroker"); + return answer; + } + + private BrokerService setUpFirstBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(firstBindAddress); + answer.setDataDirectory(FIRST_KAHADB_DIRECTORY); + answer.setBrokerName("firstBroker"); + return answer; + } + + private ReplicationViewMBean getReplicationViewMBean(BrokerService broker) throws Exception { + MBeanServer mbeanServer = broker.getManagementContext().getMBeanServer(); + String objectNameStr = broker.getBrokerObjectName().toString(); + objectNameStr += ",service=Plugins,instanceName=ReplicationPlugin"; + ObjectName replicaViewMBeanName = assertRegisteredObjectName(mbeanServer, objectNameStr); + return MBeanServerInvocationHandler.newProxyInstance(mbeanServer, replicaViewMBeanName, ReplicationViewMBean.class, true); + } + + public ObjectName assertRegisteredObjectName(MBeanServer mbeanServer, String name) throws MalformedObjectNameException, NullPointerException { + ObjectName objectName = new ObjectName(name); + if (mbeanServer.isRegistered(objectName)) { + System.out.println("Bean Registered: " + objectName); + } else { + System.err.println("Could not find MBean!: " + objectName); + } + return objectName; + } + +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationEventHandlingTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationEventHandlingTest.java new file mode 100644 index 00000000000..eba39ee553f --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationEventHandlingTest.java @@ -0,0 +1,272 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.ActiveMQConnection; +import org.apache.activemq.ActiveMQConnectionFactory; +import org.apache.activemq.ActiveMQXAConnectionFactory; +import org.apache.activemq.advisory.DestinationSource; +import org.apache.activemq.broker.Broker; +import org.apache.activemq.broker.BrokerPlugin; +import org.apache.activemq.broker.BrokerService; +import org.apache.activemq.broker.TransportConnector; +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQMessage; +import org.apache.activemq.command.ActiveMQQueue; +import org.apache.activemq.command.DestinationInfo; +import org.apache.activemq.command.MessageId; +import org.apache.activemq.replica.ReplicaEvent; +import org.apache.activemq.replica.ReplicaEventSerializer; +import org.apache.activemq.replica.ReplicaEventType; +import org.apache.activemq.replica.ReplicaPlugin; +import org.apache.activemq.replica.ReplicaPolicy; +import org.apache.activemq.replica.ReplicaRole; +import org.apache.activemq.replica.ReplicaRoleManagementBroker; +import org.apache.activemq.replica.ReplicaStatistics; +import org.apache.activemq.replica.ReplicaSupport; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; +import org.mockito.ArgumentCaptor; + +import javax.jms.Connection; +import javax.jms.MessageProducer; +import javax.jms.Session; +import javax.jms.TextMessage; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doReturn; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.spy; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; + +public class ReplicationEventHandlingTest extends ReplicaPluginTestSupport { + + private final ReplicaEventSerializer eventSerializer = new ReplicaEventSerializer(); + private final ActiveMQQueue sequenceQueue = new ActiveMQQueue(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + private Broker nextBrokerSpy; + private ActiveMQQueue mockMainQueue; + private TransportConnector replicationConnector; + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + ActiveMQConnectionFactory mockConnectionFactorySpy; + + ActiveMQConnection mockConnectionSpy; + + ReplicaPolicy mockReplicaPolicy; + + @Before + public void setUp() throws Exception { + firstBroker = new BrokerService(); + firstBroker.setUseJmx(true); + firstBroker.setPersistent(false); + firstBroker.getManagementContext().setCreateConnector(false); + firstBroker.addConnector(firstBindAddress); + firstBroker.setDataDirectory(FIRST_KAHADB_DIRECTORY); + firstBroker.setBrokerName("firstBroker"); + replicationConnector = firstBroker.addConnector(firstReplicaBindAddress); + replicationConnector.setName("replication"); + firstBroker.start(); + firstBrokerConnectionFactory = new ActiveMQConnectionFactory(firstBindAddress); + firstBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(firstBindAddress); + + mockReplicaPolicy = spy(ReplicaPolicy.class); + mockConnectionFactorySpy = spy(new ActiveMQConnectionFactory(firstReplicaBindAddress)); + mockConnectionSpy = spy((ActiveMQConnection) mockConnectionFactorySpy.createConnection()); + doReturn(mockConnectionFactorySpy).when(mockReplicaPolicy).getOtherBrokerConnectionFactory(); + + mockMainQueue = new ActiveMQQueue(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + + + doReturn(getMainReplicationQueue()).when(mockConnectionSpy).getDestinationSource(); + doReturn(mockConnectionSpy).when(mockConnectionFactorySpy).createConnection(); + + if (secondBroker == null) { + secondBroker = createSecondBroker(); + } + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + } + + @After + protected void tearDown() throws Exception { + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + + super.tearDown(); + } + + @Test + public void testReplicaBrokerHasOutOfOrderReplicationEvent() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(mockMainQueue); + + startSecondBroker(); + destination = createDestination(); + Thread.sleep(SHORT_TIMEOUT); + + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + secondBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(secondBindAddress); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + ActiveMQMessage replicaEventMessage = new ActiveMQMessage(); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_UPSERT) + .setEventData(eventSerializer.serializeReplicationData(destination)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "0"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + firstBrokerProducer.send(mockMainQueue, replicaEventMessage); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + + replicaEventMessage = spy(new ActiveMQMessage()); + + event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "100"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + firstBrokerProducer.send(mockMainQueue, replicaEventMessage); + Thread.sleep(LONG_TIMEOUT); + + verify(nextBrokerSpy, times(1)).send(any(), any()); + verify(replicaEventMessage, never()).acknowledge(); + } + + @Test + public void testReplicaBrokerHasDuplicateReplicationEvent() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(mockMainQueue); + + startSecondBroker(); + destination = createDestination(); + Thread.sleep(SHORT_TIMEOUT); + + secondBrokerConnectionFactory = new ActiveMQConnectionFactory(secondBindAddress); + secondBrokerXAConnectionFactory = new ActiveMQXAConnectionFactory(secondBindAddress); + + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + secondBrokerConnection.start(); + + ActiveMQMessage replicaEventMessage = new ActiveMQMessage(); + ReplicaEvent event = new ReplicaEvent() + .setEventType(ReplicaEventType.DESTINATION_UPSERT) + .setEventData(eventSerializer.serializeReplicationData(destination)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "20"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + firstBrokerProducer.send(mockMainQueue, replicaEventMessage); + Thread.sleep(LONG_TIMEOUT); + + QueueViewMBean secondBrokerSequenceQueueView = getQueueView(secondBroker, ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + assertEquals(secondBrokerSequenceQueueView.browseMessages().size(), 1); + TextMessage sequenceQueueMessage = (TextMessage) secondBrokerSequenceQueueView.browseMessages().get(0); + String[] textMessageSequence = sequenceQueueMessage.getText().split("#"); + assertEquals(Integer.parseInt(textMessageSequence[0]), 20); + + MessageId messageId = new MessageId("1:1"); + + ActiveMQMessage message = new ActiveMQMessage(); + message.setMessageId(messageId); + message.setDestination(destination); + + replicaEventMessage = new ActiveMQMessage(); + + event = new ReplicaEvent() + .setEventType(ReplicaEventType.MESSAGE_SEND) + .setEventData(eventSerializer.serializeMessageData(message)); + replicaEventMessage.setContent(event.getEventData()); + replicaEventMessage.setStringProperty(ReplicaEventType.EVENT_TYPE_PROPERTY, event.getEventType().name()); + replicaEventMessage.setStringProperty(ReplicaSupport.SEQUENCE_PROPERTY, "10"); + replicaEventMessage.setIntProperty(ReplicaSupport.VERSION_PROPERTY, ReplicaSupport.CURRENT_VERSION); + + System.out.println("sending first MESSAGE_SEND..."); + firstBrokerProducer.send(mockMainQueue, replicaEventMessage); + Thread.sleep(LONG_TIMEOUT); + + ArgumentCaptor messageArgumentCaptor = ArgumentCaptor.forClass(ActiveMQMessage.class); + verify(nextBrokerSpy, times(2)).send(any(), messageArgumentCaptor.capture()); + messageArgumentCaptor.getAllValues().stream() + .forEach(msg -> assertEquals(msg.getDestination(), sequenceQueue)); + } + + private DestinationSource getMainReplicationQueue() throws Exception { + DestinationSource destination = new DestinationSource(mockConnectionSpy); + DestinationInfo destinationInfo = new DestinationInfo(); + destinationInfo.setDestination(mockMainQueue); + ActiveMQMessage activeMQMessage = new ActiveMQMessage(); + activeMQMessage.setDataStructure(destinationInfo); + destination.onMessage(activeMQMessage); + + return destination; + } + + @Override + protected BrokerService createSecondBroker() throws Exception { + BrokerService answer = new BrokerService(); + answer.setUseJmx(true); + answer.setPersistent(false); + answer.getManagementContext().setCreateConnector(false); + answer.addConnector(secondBindAddress); + answer.setDataDirectory(SECOND_KAHADB_DIRECTORY); + answer.setBrokerName("secondBroker"); + + ReplicaPlugin replicaPlugin = new ReplicaPlugin() { + @Override + public Broker installPlugin(final Broker broker) { + nextBrokerSpy = spy(broker); + return new ReplicaRoleManagementBroker(nextBrokerSpy, replicaPolicy, ReplicaRole.replica, new ReplicaStatistics()); + } + }; + replicaPlugin.setRole(ReplicaRole.replica); + replicaPlugin.setTransportConnectorUri(secondReplicaBindAddress); + replicaPlugin.setOtherBrokerUri(firstReplicaBindAddress); + replicaPlugin.setControlWebConsoleAccess(false); + replicaPlugin.setHeartBeatPeriod(0); + + answer.setPlugins(new BrokerPlugin[]{replicaPlugin}); + answer.setSchedulerSupport(true); + return answer; + } +} diff --git a/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationQueueOperationsTest.java b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationQueueOperationsTest.java new file mode 100644 index 00000000000..0c60de40640 --- /dev/null +++ b/activemq-unit-tests/src/test/java/org/apache/activemq/broker/replica/ReplicationQueueOperationsTest.java @@ -0,0 +1,169 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.activemq.broker.replica; + +import org.apache.activemq.broker.jmx.QueueViewMBean; +import org.apache.activemq.command.ActiveMQTextMessage; +import org.apache.activemq.replica.ReplicaSupport; +import org.apache.activemq.util.Wait; +import org.junit.Test; +import org.springframework.jms.core.JmsTemplate; + +import javax.jms.Connection; +import javax.jms.MessageProducer; +import javax.jms.Session; + +public class ReplicationQueueOperationsTest extends ReplicaPluginTestSupport { + protected Connection firstBrokerConnection; + protected Connection secondBrokerConnection; + + @Override + protected void setUp() throws Exception { + super.setUp(); + + firstBrokerConnection = firstBrokerConnectionFactory.createConnection(); + secondBrokerConnection = secondBrokerConnectionFactory.createConnection(); + firstBrokerConnection.start(); + secondBrokerConnection.start(); + } + + @Override + protected void tearDown() throws Exception { + if (firstBrokerConnection != null) { + firstBrokerConnection.close(); + firstBrokerConnection = null; + } + if (secondBrokerConnection != null) { + secondBrokerConnection.close(); + secondBrokerConnection = null; + } + super.tearDown(); + } + + @Test + public void testSendMessageToReplicationQueues() throws Exception { + JmsTemplate firstBrokerJmsTemplate = new JmsTemplate(); + firstBrokerJmsTemplate.setConnectionFactory(firstBrokerConnectionFactory); + assertFunctionThrows(() -> firstBrokerJmsTemplate.convertAndSend(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> firstBrokerJmsTemplate.convertAndSend(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> firstBrokerJmsTemplate.convertAndSend(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + + JmsTemplate secondBrokerJmsTemplate = new JmsTemplate(); + secondBrokerJmsTemplate.setConnectionFactory(secondBrokerConnectionFactory); + assertFunctionThrows(() -> secondBrokerJmsTemplate.convertAndSend(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> secondBrokerJmsTemplate.convertAndSend(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> secondBrokerJmsTemplate.convertAndSend(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME, getName()), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + } + + @Test + public void testConsumeMessageFromReplicationQueues() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination); + final int NUM_OF_MESSAGE_SEND = 50; + + for (int i = 0; i < NUM_OF_MESSAGE_SEND; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + secondBrokerProducer.send(message); + } + + JmsTemplate firstBrokerJmsTemplate = new JmsTemplate(); + firstBrokerJmsTemplate.setConnectionFactory(firstBrokerConnectionFactory); + assertFunctionThrows(() -> firstBrokerJmsTemplate.receive(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> firstBrokerJmsTemplate.receive(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> firstBrokerJmsTemplate.receive(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + + JmsTemplate secondBrokerJmsTemplate = new JmsTemplate(); + secondBrokerJmsTemplate.setConnectionFactory(secondBrokerConnectionFactory); + assertFunctionThrows(() -> secondBrokerJmsTemplate.receive(ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> secondBrokerJmsTemplate.receive(ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + assertFunctionThrows(() -> secondBrokerJmsTemplate.receive(ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME), + "JMSException: Not authorized to access destination: queue://" + ReplicaSupport.SEQUENCE_REPLICATION_QUEUE_NAME); + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + @Test + public void testPurgeReplicationQueues() throws Exception { + Session firstBrokerSession = firstBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer firstBrokerProducer = firstBrokerSession.createProducer(destination); + Session secondBrokerSession = secondBrokerConnection.createSession(false, Session.CLIENT_ACKNOWLEDGE); + MessageProducer secondBrokerProducer = secondBrokerSession.createProducer(destination); + final int NUM_OF_MESSAGE_SEND = 50; + + for (int i = 0; i < NUM_OF_MESSAGE_SEND; i++) { + ActiveMQTextMessage message = new ActiveMQTextMessage(); + message.setText(getName()); + firstBrokerProducer.send(message); + secondBrokerProducer.send(message); + } + + QueueViewMBean firstBrokerMainQueue = getQueueView(firstBroker, ReplicaSupport.MAIN_REPLICATION_QUEUE_NAME); + QueueViewMBean firstBrokerIntermediateQueue = getQueueView(firstBroker, ReplicaSupport.INTERMEDIATE_REPLICATION_QUEUE_NAME); + + waitForQueueHasMessage(firstBrokerMainQueue); + firstBrokerMainQueue.purge(); + Thread.sleep(LONG_TIMEOUT); + assertEquals(0, firstBrokerMainQueue.getInFlightCount()); + + + waitForQueueHasMessage(firstBrokerIntermediateQueue); + firstBrokerIntermediateQueue.purge(); + Thread.sleep(LONG_TIMEOUT); + assertEquals(0, firstBrokerIntermediateQueue.getInFlightCount()); + + firstBrokerSession.close(); + secondBrokerSession.close(); + } + + private void assertFunctionThrows(Function testFunction, String expectedMessage) { + try { + testFunction.apply(); + fail("Should have thrown exception on " + testFunction); + } catch (Exception e) { + assertTrue(e.getMessage().contains(expectedMessage)); + } + } + + private void waitForQueueHasMessage(QueueViewMBean queue) throws Exception { + Wait.waitFor(new Wait.Condition() { + @Override + public boolean isSatisified() throws Exception { + return queue.getEnqueueCount() > 0; + } + }); + } + + @FunctionalInterface + public interface Function { + void apply() throws Exception; + } +} diff --git a/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-primary.xml b/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-primary.xml new file mode 100644 index 00000000000..ffa2cf0187c --- /dev/null +++ b/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-primary.xml @@ -0,0 +1,58 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-replica.xml b/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-replica.xml new file mode 100644 index 00000000000..9c9fd62d676 --- /dev/null +++ b/activemq-unit-tests/src/test/resources/org/apache/activemq/broker/replica/transport-protocol-test-replica.xml @@ -0,0 +1,58 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +