Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STAR-1342: Changes from DSEDB - DO NOT MERGE #60

Draft
wants to merge 29 commits into
base: trunk
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
16f64c7
STAR-564 Check only MODIFY on base when updating table with MV (#17)
jtgrabowski Jun 1, 2021
12d4292
Add pytest cache and vscode folders to gitignore (#21)
k-rus Jun 8, 2021
2ce0618
STAR-582 fix repair error on one node cluster (#20)
k-rus Jun 11, 2021
626db14
STAR-247: Allow to easily run tests against big or bti sstable format
jacek-lewandowski Apr 2, 2021
2225baf
STAR-247: remove format specific parts of assertions
jtgrabowski Apr 6, 2021
8c34af1
STAR-247: Standalone scrubber should use the same default sstable for…
jacek-lewandowski Jun 23, 2021
200a8da
STAR-247: Add allowed warning for running scrub test
jacek-lewandowski Jun 23, 2021
d048221
STAR-14: Fix expectations to include memtable table parameter
blambov Apr 12, 2021
b45414e
STAR-254: add DateRange and Geo tests (#9)
jtgrabowski Apr 20, 2021
71ed787
STAR-452: add EverywhereStrategy smoke test (#10)
jtgrabowski Apr 21, 2021
b65026e
STAR-431: Add option to prevent any file-I/O from cqlsh
mfleming May 24, 2021
eccee2d
STAR-431: Add more tests to make sure commands work without --no-file…
mfleming May 25, 2021
571f3da
STAR-432: Add tests for consistency level options (#18)
tlasica Jun 1, 2021
ece3c5c
STAR-543: Port guardrail tests and changes (#19)
djatnieks Jun 8, 2021
0b99fa9
STAR-765: Add tests for cloud connection. (#23)
tlasica Jun 16, 2021
0c48297
STAR-386: Add logging around failure to get timestamp info (#28)
djatnieks Jun 24, 2021
2cdb98e
STAR-517. Skip as it is not possible to test it reliably. (#34)
tlasica Jul 2, 2021
848bdaf
STAR-386 Increase client timeout for test_simultaneous_bootstrap (#37)
djatnieks Jul 2, 2021
b98d21d
STAR-385 Retry cluster stop after exception stopping 'gently' (#36)
djatnieks Jul 2, 2021
8cbe83f
STAR-13 Run tests for UnifiedCompactionStrategy (#22)
Gerrrr Jul 15, 2021
440e9cc
STAR-826 Add missing import (#40)
Gerrrr Jul 16, 2021
fcbc4d4
STAR-836 Fix TestCompaction_with_UnifiedCompactionStrategy.bloomfilte…
Gerrrr Aug 5, 2021
f707b3e
STAR-822: Update test_consistent_repair dtest
jacek-lewandowski Aug 16, 2021
3d83ece
STAR-843: Update dtests for ULID based generation ID
jacek-lewandowski Aug 20, 2021
e5672df
STAR-899: Fix merge_schema_failure_4x.btm script to reflect our chang…
jacek-lewandowski Mar 21, 2022
76028a2
STAR-1200: Better logging for dtests (#55)
jacek-lewandowski Mar 21, 2022
f6c9534
STAR-394: Fix various timeouts (#57)
jacek-lewandowski Mar 21, 2022
0e78a6e
STAR-386: Increase client timeout (#56)
jacek-lewandowski Mar 22, 2022
b7a284a
STAR-899 Fix class name used from SchemaManager to Schema (#58)
djatnieks May 23, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,6 @@ upgrade
html/
doxygen/doxypy-0.4.2/
.pytest_cache/
.vscode/

pytest.ini
42 changes: 36 additions & 6 deletions auth_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -552,10 +552,16 @@ def test_materialized_views_auth(self):
* Create a new user, 'cathy', with no permissions
* Create a ks, table
* Connect as cathy
*
* Try CREATE MV without ALTER permission on base table, assert throws Unauthorized
* Grant cathy ALTER permissions, then CREATE MV successfully
*
* Try to MODIFY base without WRITE permission on base, assert throws Unauthorized
* Grant cathy WRITE permissions on base, and modify base successfully
*
* Try to SELECT from the mv, assert throws Unauthorized
* Grant cathy SELECT permissions, and read from the MV successfully
* Grant cathy SELECT permissions on base, and read from the MV successfully
*
* Revoke cathy's ALTER permissions, assert DROP MV throws Unauthorized
* Restore cathy's ALTER permissions, DROP MV successfully
"""
Expand All @@ -576,12 +582,36 @@ def test_materialized_views_auth(self):
cassandra.execute("GRANT ALTER ON ks.cf TO cathy")
cathy.execute(create_mv)

# TRY SELECT MV without SELECT permission on base table
assert_unauthorized(cathy, "SELECT * FROM ks.mv1", "User cathy has no SELECT permission on <table ks.cf> or any of its parents")
# Try MODIFY base without WRITE permission on base
assert_unauthorized(cathy, "INSERT INTO ks.cf(id, value) VALUES(1, '1')", "User cathy has no MODIFY permission on <table ks.cf> or any of its parents")

# Grant SELECT permission and CREATE MV
cassandra.execute("GRANT SELECT ON ks.cf TO cathy")
cathy.execute("SELECT * FROM ks.mv1")
if self.cluster.version() >= LooseVersion('4.0'):
# From 4.0 onward, only base MODIFY permission is required to update base with MV
# Grant WRITE permission on Base
cassandra.execute("GRANT MODIFY ON ks.cf TO cathy")
cathy.execute("INSERT INTO ks.cf(id, value) VALUES(1, '1')")

# TRY SELECT MV without SELECT permission on base table
assert_unauthorized(cathy, "SELECT * FROM ks.cf", "User cathy has no SELECT permission on <table ks.cf> or any of its parents")
assert_unauthorized(cathy, "SELECT * FROM ks.mv1", "User cathy has no SELECT permission on <table ks.cf> or any of its parents")

# Grant SELECT permission
cassandra.execute("GRANT SELECT ON ks.cf TO cathy")
assert_one(cathy, "SELECT * FROM ks.cf", [1, '1'])
assert_one(cathy, "SELECT * FROM ks.mv1", ['1', 1])
else:
# Before 4.0, MODIFY on MV is required to insert to base
# Grant WRITE permission on Base
cassandra.execute("GRANT MODIFY ON ks.cf TO cathy")
assert_unauthorized(cathy, "INSERT INTO ks.cf(id, value) VALUES(1, '1')", "User cathy has no SELECT permission on <table ks.cf> or any of its parents")
cassandra.execute("GRANT SELECT ON ks.cf TO cathy")
assert_unauthorized(cathy, "INSERT INTO ks.cf(id, value) VALUES(1, '1')", "User cathy has no MODIFY permission on <table ks.mv1> or any of its parents")

# Grant WRITE permission on MV
cassandra.execute("GRANT MODIFY ON ks.mv1 TO cathy")
cathy.execute("INSERT INTO ks.cf(id, value) VALUES(1, '1')")
assert_one(cathy, "SELECT * FROM ks.cf", [1, '1'])
assert_one(cathy, "SELECT * FROM ks.mv1", ['1', 1])

# Revoke ALTER permission and try DROP MV
cassandra.execute("REVOKE ALTER ON ks.cf FROM cathy")
Expand Down
61 changes: 60 additions & 1 deletion bootstrap_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -849,6 +849,12 @@ def test_simultaneous_bootstrap(self):
" cannot bootstrap while cassandra.consistent.rangemovement is true"

cluster = self.cluster
configuration_options = {
'request_timeout_in_ms': 120000,
'read_request_timeout_in_ms': 120000,
'range_request_timeout_in_ms': 120000
}
cluster.set_configuration_options(configuration_options)
cluster.set_environment_variable('CASSANDRA_TOKEN_PREGENERATION_DISABLED', 'True')
cluster.populate(1)
cluster.start()
Expand Down Expand Up @@ -884,7 +890,10 @@ def test_simultaneous_bootstrap(self):
# bugs like 9484, where count(*) fails at higher
# data loads.
for _ in range(5):
assert_one(session, "SELECT count(*) from keyspace1.standard1", [500000], cl=ConsistencyLevel.ONE)
logger.info("Querying: SELECT count(*) from keyspace1.standard1")
# Improve reliability for slower/loaded test systems by using larger client timeout
assert_one(session, "SELECT count(*) from keyspace1.standard1", [500000], cl=ConsistencyLevel.ONE, timeout=180)
logger.info("Querying completed")

def test_cleanup(self):
"""
Expand Down Expand Up @@ -1019,6 +1028,56 @@ def test_bootstrap_binary_disabled(self):
assert_bootstrap_state(self, node3, 'COMPLETED', user='cassandra', password='cassandra')
node3.wait_for_binary_interface()

@since('4.0')
@pytest.mark.no_vnodes
def test_simple_bootstrap_with_everywhere_strategy(self):
cluster = self.cluster
tokens = cluster.balanced_tokens(2)
cluster.set_configuration_options(values={'num_tokens': 1})

logger.debug("[node1, node2] tokens: %r" % (tokens,))

keys = 10000

# Create a single node cluster
cluster.populate(1)
node1 = cluster.nodelist()[0]
node1.set_configuration_options(values={'initial_token': tokens[0]})
cluster.start()

session = self.patient_cql_connection(node1)
create_ks(session, 'ks', 'EverywhereStrategy')
create_cf(session, 'cf', columns={'c1': 'text', 'c2': 'text'})

insert_statement = session.prepare("INSERT INTO ks.cf (key, c1, c2) VALUES (?, 'value1', 'value2')")
execute_concurrent_with_args(session, insert_statement, [['k%d' % k] for k in range(keys)])

node1.flush()
node1.compact()

# Reads inserted data all during the bootstrap process. We shouldn't
# get any error
query_c1c2(session, random.randint(0, keys - 1), ConsistencyLevel.ONE)
session.shutdown()

# Bootstrapping a new node in the current version
node2 = new_node(cluster)
node2.set_configuration_options(values={'initial_token': tokens[1]})
node2.start(wait_for_binary_proto=True)
node2.compact()

node1.cleanup()
logger.debug("node1 size for ks.cf after cleanup: %s" % float(data_size(node1,'ks','cf')))
node1.compact()
logger.debug("node1 size for ks.cf after compacting: %s" % float(data_size(node1,'ks','cf')))

logger.debug("node2 size for ks.cf after compacting: %s" % float(data_size(node2,'ks','cf')))

size1 = float(data_size(node1,'ks','cf'))
size2 = float(data_size(node2,'ks','cf'))
assert_almost_equal(size1, size2, error=0.3)

assert_bootstrap_state(self, node2, 'COMPLETED')

class TestBootstrap(BootstrapTester):
"""
Expand Down
8 changes: 8 additions & 0 deletions byteman/guardrails/disk_usage_full.btm
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
RULE return FULL disk usage
CLASS org.apache.cassandra.service.disk.usage.DiskUsageMonitor
METHOD getState
AT EXIT
IF TRUE
DO
return org.apache.cassandra.service.disk.usage.DiskUsageState.FULL;
ENDRULE
8 changes: 8 additions & 0 deletions byteman/guardrails/disk_usage_stuffed.btm
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
RULE return STUFFED disk usage
CLASS org.apache.cassandra.service.disk.usage.DiskUsageMonitor
METHOD getState
AT EXIT
IF TRUE
DO
return org.apache.cassandra.service.disk.usage.DiskUsageState.STUFFED;
ENDRULE
2 changes: 1 addition & 1 deletion byteman/merge_schema_failure_4x.btm
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#
RULE inject node failure on merge schema exit
CLASS org.apache.cassandra.schema.Schema
METHOD merge
METHOD mergeAndUpdateVersion
AT EXIT
# set flag to only run this rule once.
IF TRUE
Expand Down
3 changes: 2 additions & 1 deletion client_request_metrics_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,15 @@ def fixture_add_additional_log_patterns(self, fixture_dtest_setup):
fixture_dtest_setup.ignore_log_patterns = (
'Testing write failures', # The error to simulate a write failure
'ERROR WRITE_FAILURE', # Logged in DEBUG mode for write failures
f"Scanned over {TOMBSTONE_FAILURE_THRESHOLD + 1} tombstones during query" # Caused by the read failure tests
f"Scanned over {TOMBSTONE_FAILURE_THRESHOLD + 1} (tombstones|tombstone rows) during query" # Caused by the read failure tests
)

def setup_once(self):
cluster = self.cluster
cluster.set_configuration_options({'read_request_timeout_in_ms': 3000,
'write_request_timeout_in_ms': 3000,
'phi_convict_threshold': 12,
'tombstone_warn_threshold': -1,
'tombstone_failure_threshold': TOMBSTONE_FAILURE_THRESHOLD,
'enable_materialized_views': 'true'})
cluster.populate(2, debug=True)
Expand Down
29 changes: 23 additions & 6 deletions compaction_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
since = pytest.mark.since
logger = logging.getLogger(__name__)

strategies = ['LeveledCompactionStrategy', 'SizeTieredCompactionStrategy', 'DateTieredCompactionStrategy']
strategies = ['LeveledCompactionStrategy', 'SizeTieredCompactionStrategy', 'DateTieredCompactionStrategy', 'UnifiedCompactionStrategy']


class TestCompaction(Tester):
Expand Down Expand Up @@ -114,25 +114,30 @@ def test_bloomfilter_size(self, strategy):
else:
if strategy == "DateTieredCompactionStrategy":
strategy_string = "strategy=DateTieredCompactionStrategy,base_time_seconds=86400" # we want a single sstable, so make sure we don't have a tiny first window
elif self.strategy == "UnifiedCompactionStrategy":
strategy_string = "strategy=UnifiedCompactionStrategy,max_sstables_to_compact=4" # disable layout-preserving compaction which can leave more than one sstable
else:
strategy_string = "strategy={}".format(strategy)
min_bf_size = 100000
max_bf_size = 150000
cluster = self.cluster
cluster.populate(1).start()
[node1] = cluster.nodelist()
logger.debug("Compaction: " + strategy_string)

for x in range(0, 5):
node1.stress(['write', 'n=100K', "no-warmup", "cl=ONE", "-rate",
"threads=300", "-schema", "replication(factor=1)",
"compaction({},enabled=false)".format(strategy_string)])
node1.flush()
logger.debug(node1.nodetool('cfstats keyspace1.standard1').stdout)

node1.nodetool('enableautocompaction')
node1.wait_for_compactions()

table_name = 'standard1'
output = node1.nodetool('cfstats').stdout
output = node1.nodetool('cfstats keyspace1.standard1').stdout
logger.debug(output)
output = output[output.find(table_name):]
output = output[output.find("Bloom filter space used"):]
bfSize = int(output[output.find(":") + 1:output.find("\n")].strip())
Expand All @@ -153,7 +158,12 @@ def test_bloomfilter_size(self, strategy):

logger.debug("bloom filter size is: {}".format(bfSize))
logger.debug("size factor = {}".format(size_factor))
assert bfSize >= size_factor * min_bf_size
# In the case where the number of sstables is greater than the number of directories, it's possible this to be
# both with unique keys (where the bf size will remain close to the unadjusted limit) or with repetitions
# of keys (where the bf size will be a multiple of the expected). Permit both by only using the size factor on
# the maximum size. Note that the test is designed to end up with size_factor == 1 and most runs do so, thus
# this is not a loosening of the test in the common case, only ensures that we don't end up with flakes.
assert bfSize >= min_bf_size
assert bfSize <= size_factor * max_bf_size

@pytest.mark.parametrize("strategy", strategies)
Expand Down Expand Up @@ -298,7 +308,7 @@ def test_compaction_strategy_switching(self, strategy):
Ensure that switching strategies does not result in problems.
Insert data, switch strategies, then check against data loss.
"""
strategies = ['LeveledCompactionStrategy', 'SizeTieredCompactionStrategy', 'DateTieredCompactionStrategy']
strategies = ['LeveledCompactionStrategy', 'SizeTieredCompactionStrategy', 'DateTieredCompactionStrategy', 'UnifiedCompactionStrategy']

if strategy in strategies:
strategies.remove(strategy)
Expand All @@ -307,6 +317,7 @@ def test_compaction_strategy_switching(self, strategy):
[node1] = cluster.nodelist()

for strat in strategies:
logger.debug("Switching to {}".format(strat))
session = self.patient_cql_connection(node1)
create_ks(session, 'ks', 1)

Expand Down Expand Up @@ -339,7 +350,10 @@ def test_large_compaction_warning(self):
Check that we log a warning when the partition size is bigger than compaction_large_partition_warning_threshold_mb
"""
cluster = self.cluster
cluster.set_configuration_options({'compaction_large_partition_warning_threshold_mb': 1})
if self.supports_guardrails:
cluster.set_configuration_options({'guardrails': {'partition_size_warn_threshold_in_mb': 1}})
else:
cluster.set_configuration_options({'compaction_large_partition_warning_threshold_mb': 1})
cluster.populate(1).start()
[node] = cluster.nodelist()

Expand All @@ -361,7 +375,10 @@ def test_large_compaction_warning(self):
node.nodetool('compact ks large')
verb = 'Writing' if self.cluster.version() > '2.2' else 'Compacting'
sizematcher = '\d+ bytes' if self.cluster.version() < LooseVersion('3.6') else '\d+\.\d{3}(K|M|G)iB'
node.watch_log_for('{} large partition ks/large:user \({}'.format(verb, sizematcher), from_mark=mark, timeout=180)
log_message = '{} large partition ks/large:user \({}'.format(verb, sizematcher)
if self.supports_guardrails:
log_message = "Detected partition 'user' in ks.large of size 2MB is greater than the maximum recommended size \(1MB\)"
node.watch_log_for(log_message, from_mark=mark, timeout=180)

ret = list(session.execute("SELECT properties from ks.large where userid = 'user'"))
assert_length_equal(ret, 1)
Expand Down
10 changes: 10 additions & 0 deletions conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ def check_required_loopback_interfaces_available():


def pytest_addoption(parser):
parser.addoption("--sstable-format", action="store", default="bti",
help="SSTable format to be used by default for all newly created SSTables: "
"big or bti (default: bti)")
parser.addoption("--use-vnodes", action="store_true", default=False,
help="Determines wither or not to setup clusters using vnodes for tests")
parser.addoption("--use-off-heap-memtables", action="store_true", default=False,
Expand Down Expand Up @@ -175,6 +178,13 @@ class level seems to work, and I guess it's not that much extra overhead to setu
if pytest.config.inicfg.get("log_format") is not None:
logging_format = pytest.config.inicfg.get("log_format")

# ccm logger is configured to spit everything to console
# we want it to use logging setup configured for tests
# unless we do that, we get duplicated log records from ccm module
ccmLogger = logging.getLogger("ccm")
for handler in ccmLogger.handlers:
logging.getLogger("ccm").removeHandler(handler)

logging.basicConfig(level=log_level,
format=logging_format)

Expand Down
17 changes: 17 additions & 0 deletions cqlsh_tests/cqlshrc.sample.cloud
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
; Copyright DataStax, Inc.
;
; Licensed under the Apache License, Version 2.0 (the "License");
; you may not use this file except in compliance with the License.
; You may obtain a copy of the License at
;
; http://www.apache.org/licenses/LICENSE-2.0
;
; Unless required by applicable law or agreed to in writing, software
; distributed under the License is distributed on an "AS IS" BASIS,
; WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
; See the License for the specific language governing permissions and
; limitations under the License.
;
; Sample ~/.cqlshrc file with cloud configuration.
[connection]
secure_connect_bundle = /path/to/creds.zip
Binary file added cqlsh_tests/secure-connect-test.zip
Binary file not shown.
Loading