cassandra-stress: counter_write workload got stuck with small population seq #54
Assignees
Comments
|
Reproduced with recent master. c-s processes status: |
This was referenced Jul 3, 2019
amoskong
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
from: scylladb/scylladb#2790 (comment)
I will retest with laster scylla.
Installation details
Scylla version (or git commit hash): 1.7.4-0.20170726.ff643e3 2.0.rc4-0.20170903.6e6de34
Cluster size: 4
OS (RHEL/CentOS/Ubuntu/AWS AMI): CentOS7
Description
cassandra-stress got stuck after the total ops number reached to population seq.
If we use a short duration (such as 10s), it won't exit.
Prepare
Create keyspace2 (tables: counter1, standard1)
CREATE KEYSPACE IF NOT EXISTS keyspace2
WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true;
CREATE TABLE IF NOT EXISTS keyspace2.counter1 (
key blob PRIMARY KEY,
"C0" counter,
"C1" counter,
"C2" counter,
"C3" counter,
"C4" counter
) WITH COMPACT STORAGE
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL","rows_per_partition":"ALL"}'
AND comment = ''
AND compaction = {'class': 'SizeTieredCompactionStrategy'}
AND compression = {}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
Result
No output from commandline, and raise an exception after some minutes.
$ cassandra-stress counter_write no-warmup cl=QUORUM duration=5m -schema 'replication(factor=1) compaction(strategy=DateTieredCompactionStrategy)' keyspace=keyspace2 -port jmx=6868 -mode cql3 native -rate threads=10 -pop seq=1..10000 -node 10.240.0.4
Connected to cluster: longevity-50gb-4d-amosread-db-cluster-76310bd4, max pending requests per connection 128, max connections per host 8
Datatacenter: datacenter1; Host: /10.240.0.30; Rack: rack1
Datatacenter: datacenter1; Host: /10.240.0.19; Rack: rack1
Datatacenter: datacenter1; Host: /10.240.0.4; Rack: rack1
Datatacenter: datacenter1; Host: /10.240.0.27; Rack: rack1
Created keyspaces. Sleeping 1s for propagation.
Sleeping 2s...
Running COUNTER_WRITE with 10 threads 5 minutes
Failed to connect over JMX; not collecting these stats
type, total ops, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, errors, gc: #, max ms, sum ms, sdv ms, mb
total, 840, 840, 840, 840, 10.4, 3.6, 35.5, 87.1, 136.1, 136.1, 1.0, 0.00000, 0, 0, 0, 0, 0, 0
total, 1641, 767, 767, 767, 13.0, 9.7, 28.7, 209.9, 263.6, 263.6, 2.0, 0.04577, 0, 0, 0, 0, 0, 0
total, 3026, 1380, 1380, 1380, 7.2, 2.4, 22.0, 33.0, 44.0, 84.0, 3.0, 0.12653, 0, 0, 0, 0, 0, 0
total, 4213, 1133, 1133, 1133, 8.5, 9.3, 22.0, 32.6, 44.0, 65.1, 4.1, 0.09444, 0, 0, 0, 0, 0, 0
total, 5630, 1365, 1365, 1365, 7.5, 2.7, 22.0, 34.9, 76.0, 83.4, 5.1, 0.08485, 0, 0, 0, 0, 0, 0
total, 6626, 977, 977, 977, 10.2, 9.0, 26.5, 86.4, 122.9, 122.9, 6.2, 0.07556, 0, 0, 0, 0, 0, 0
total, 7659, 1020, 1020, 1020, 9.9, 10.2, 22.9, 45.9, 96.4, 102.6, 7.2, 0.06565, 0, 0, 0, 0, 0, 0
total, 9205, 1520, 1520, 1520, 6.6, 1.5, 21.9, 32.3, 46.1, 46.2, 8.2, 0.06942, 0, 0, 0, 0, 0, 0
java.lang.RuntimeException: Timed out waiting for a timer thread - seems one got stuck. Check GC/Heap size
at org.apache.cassandra.stress.util.Timing.snap(Timing.java:98)
at org.apache.cassandra.stress.StressMetrics.update(StressMetrics.java:156)
at org.apache.cassandra.stress.StressMetrics.access$300(StressMetrics.java:37)
at org.apache.cassandra.stress.StressMetrics$2.run(StressMetrics.java:104)
at java.lang.Thread.run(Thread.java:748)
The text was updated successfully, but these errors were encountered: