How to tune JuiceFS to deal with “excessive” amount of small files

[myc724]

We’re trying JuiceFS with several file IO patterns common in our domain. For single process write a big file then multiple remote processes read the big file, JuiceFS’s default works very well, especially when the number of remote processes goes up. However, when multiple remote processes write “excessive” amount of smaller files, JuiceFS seems stressed more in comparison w. NFS. The volume was formatted using JuiceFS’s default setup, so I’m trying different configuration/format-options (eg. "--writeback") now. Do you have any suggestion or general tuning guidance? Thanks!

400 remote processes write
  ->   8,192   4M files (total 32GB): NFS took 25s vs JuiceFS took  10s.
  -> 131,072 256K files (total 32GB): NFS took 81s vs JuiceFS took 128s.

[xiaogaozi]

-> 131,072 256K files (total 32GB): NFS took 81s vs JuiceFS took 128s.

Is this result obtained after enabling the --writeback option?

[davies]

For data safety, JuiceFS will call Fsync() for every blocks written in writeback mode, so it's much faster than S3 if the caching disk is HDD. Can you use a SSD as the caching disk (--cache-dir xxx)

What's the disk used by NFS? NFS may have different persistency story than JuiceFS, right now JuiceFS prefer durability rather than latency for small files.

[myc724]

Thanks a lot for the suggestion and information! I will deep dive and get back here.

[myc724]

We did some experiments (using default JuiceFS options) and collected data using JuiceFS accesslog. Following paragraphs summarize the result. We’re thinking to create different volumes and/or separated mountpoints to accommodate various scenarios. Based on the data shown below, do you suggest any “juicefs format” and/or “juicefs mount” option which could minimize the performance differences between newly-started-servers vs servers-loaded-with-131-thousands-256KB-files? Thanks!

================

Before an "excessive" amount of small files being added, we added 30 tests using this dd command (where ${idx} just to ensure unique output files so that every lookup results the same. After the 131 thousands 256KB thousands files being added, we, again, added another 30 tests using the same dd command.

dd if=/dev/zero of=test_${idx} bs=256K count=1 oflag=direct

For each of these dd command (which writes 256KB), JuiceFS accesslog profiles for getattr, lookup, create, flush, write, write, flush, release. We’re seeing performance differences in 3 operations (lookup, create, and the flush after write).

The flush after write:

New servers               avg 0.0084 ## ranging from <0.002271> to <0.017847> 
Servers with 131K files   avg 0.0154 ## ranging from <0.008675> to <0.024759>

Create:

New servers               avg 0.0079 ## ranging from <0.002013> to <0.022999>
Servers with 131K files   avg 0.0165 ## ranging from <0.008419> to <0.022304>

Lookup:

New servers               avg 0.0014 ## ranging from <0.002271> to <0.017847>
Servers with 131K files   avg 0.0027 ## ranging from <0.008675> to <0.024759>

================

In case you’re interested in server side statistic:

Before 131 thousands files being added, …

   PID %CPU %MEM     VSZ     RSS TTY    STAT START  TIME COMMAND
108795  0.0  0.0  48,456   3,088 pts/2  Sl   12:43  0:00 redis -port 4111 ## object storage
108798  0.0  0.0  51,028   3,976 pts/2  Sl   12:43  0:00 redis -port 6111 ## metadata

After 131 thousands files being added, …

   PID %CPU %MEM     VSZ     RSS TTY    STAT START  TIME COMMAND
108795  0.9  0.0 190,824 143,668 pts/2  Sl   12:43  0:06 redis -port 4111 ## object storage
108798 19.5  0.0 130,980  76,636 pts/2  Sl   12:43  2:00 redis -port 6111 ## metadata

[davies]

Thanks for all the details.

The difference came from Redis, I'm surprised that there are this much difference on response time, from 2m to 8ms. Did you create the 30 files in sequential or parallel?

We don't need to change the option of format based on number of files. For mount, we may only need to change the size of local cache, or maybe the --buffer-size based on your workload.

[myc724]

I used a script to create the 30 files in sequential (... 10 seconds between each file creation).

The difference came from Redis

I assume your meant the Redis for metadata operations.

Thanks a lot for the suggestions. I will continue benchmark these options.