Testing JuiceFS Sharding Scalability?

[centminmod]

I've started playing with JuiceFS mounts with sharded Cloudflare R2 buckets and while it's scaling somewhat at 5x and 10x shard tests. I'm curious

1. where are the theoretical limits when it comes to adding more sharded Cloudflare R2 buckets?
2. for folks sharding JuiceFS mount S3 buckets, how has the scalability been for you?

From https://github.com/centminmod/centminmod-juicefs#redis-metadata-cache--sharded-r2-mount-on-intel-xeon-e-2276g-6c12t-32gb-memory-and-2x-960gb-nvme-raid-1 on dedicated server with 2x 960GB NVMe raid 1.

Default 1024MB big file.

ITEM	VALUE (10x R2 Sharded + Redis)	COST (10x R2 Sharded + Redis)	VALUE (10x R2 Sharded)	COST (10x R2 Sharded)	VALUE (5x R2 Sharded)	COST (5x R2 Sharded)	VALUE (1x R2 Default)	COST (1x R2 Default)
Write big file	1904.61 MiB/s	2.15 s/file	906.04 MiB/s	4.52 s/file	960.47 MiB/s	4.26 s/file	1374.08 MiB/s	2.98 s/file
Read big file	201.00 MiB/s	20.38 s/file	223.19 MiB/s	18.35 s/file	174.17 MiB/s	23.52 s/file	152.23 MiB/s	26.91 s/file
Write small file	1319.8 files/s	3.03 ms/file	701.2 files/s	5.70 ms/file	777.4 files/s	5.15 ms/file	780.3 files/s	5.13 ms/file
Read small file	10279.8 files/s	0.39 ms/file	6378.3 files/s	0.63 ms/file	7940.0 files/s	0.50 ms/file	8000.9 files/s	0.50 ms/file
Stat file	15890.1 files/s	0.25 ms/file	21123.7 files/s	0.19 ms/file	29344.7 files/s	0.14 ms/file	27902.2 files/s	0.14 ms/file
FUSE operation	71338 operations	2.23 ms/op	71555 operations	2.16 ms/op	71597 operations	2.67 ms/op	71649 operations	3.06 ms/op
Update meta	1740 operations	0.27 ms/op	6271 operations	9.01 ms/op	6041 operations	4.09 ms/op	6057 operations	2.50 ms/op
Put object	1083 operations	390.88 ms/op	1152 operations	403.23 ms/op	1136 operations	428.27 ms/op	1106 operations	547.32 ms/op
Get object	1024 operations	294.63 ms/op	1034 operations	278.61 ms/op	1049 operations	299.50 ms/op	1030 operations	301.80 ms/op
Delete object	754 operations	125.28 ms/op	316 operations	124.32 ms/op	60 operations	120.73 ms/op	29 operations	234.02 ms/op
Write into cache	1424 operations	4.85 ms/op	1424 operations	24.92 ms/op	1424 operations	83.12 ms/op	1424 operations	12.91 ms/op
Read from cache	400 operations	0.05 ms/op	400 operations	0.05 ms/op	400 operations	0.05 ms/op	400 operations	0.04 ms/op

Default 1MB big file.

ITEM	VALUE (10x R2 Sharded + Redis)	COST (10x R2 Sharded + Redis)	VALUE (10x R2 Sharded)	COST (10x R2 Sharded)	VALUE (5x R2 Sharded)	COST (5x R2 Sharded)	VALUE (1x R2 Default)	COST (1x R2 Default)
Write big file	530.10 MiB/s	0.01 s/file	452.66 MiB/s	0.01 s/file	448.20 MiB/s	0.01 s/file	230.82 MiB/s	0.02 s/file
Read big file	1914.40 MiB/s	0.00 s/file	1545.95 MiB/s	0.00 s/file	1376.38 MiB/s	0.00 s/file	1276.38 MiB/s	0.00 s/file
Write small file	2715.4 files/s	1.47 ms/file	682.8 files/s	5.86 ms/file	792.5 files/s	5.05 ms/file	675.7 files/s	5.92 ms/file
Read small file	10069.0 files/s	0.40 ms/file	6299.4 files/s	0.63 ms/file	7827.1 files/s	0.51 ms/file	7833.1 files/s	0.51 ms/file
Stat file	16545.3 files/s	0.24 ms/file	21365.2 files/s	0.19 ms/file	24308.1 files/s	0.16 ms/file	28226.1 files/s	0.14 ms/file
FUSE operation	5767 operations	0.09 ms/op	5757 operations	0.42 ms/op	5750 operations	0.38 ms/op	5756 operations	0.41 ms/op
Update meta	1617 operations	0.19 ms/op	5814 operations	0.72 ms/op	5740 operations	0.74 ms/op	5770 operations	0.70 ms/op
Put object	37 operations	290.94 ms/op	107 operations	282.68 ms/op	94 operations	286.35 ms/op	118 operations	242.35 ms/op
Get object	0 operations	0.00 ms/op	0 operations	0.00 ms/op	0 operations	0.00 ms/op	0 operations	0.00 ms/op
Delete object	48 operations	103.83 ms/op	133 operations	116.84 ms/op	59 operations	117.93 ms/op	95 operations	83.94 ms/op
Write into cache	404 operations	0.11 ms/op	404 operations	0.12 ms/op	404 operations	0.12 ms/op	404 operations	0.14 ms/op
Read from cache	408 operations	0.06 ms/op	408 operations	0.06 ms/op	408 operations	0.05 ms/op	408 operations	0.06 ms/op