ext4's use of lazytime causes juicefs to consume a lot of cpu

[wangzhhbj]

What happened:
JuiceFS is mounted using writeback mode, with the cache-dir located on an ext4 filesystem. The ext4 is mounted with the "lazytime" option (requires kernel version 4.0 or higher).

When writing files with 8 concurrent processes, there is a large number of cache tmp files being created, leading to JuiceFS consuming a significant amount of CPU resources (60 CPU cores).

From the stack trace, it can be seen that the high CPU usage is due to ext4_file_write calling find_inode_nowait which in turn calls _raw_spin_lock, continuously contending for the lock.

mount cmd：
juicefs mount -d badger:///root/metadata jfs --writeback --socket jfs.sock --writeback-size 1024 --cache-size 1024 --log jfs_client.log

xtrabackup24(mysql5.7) backup cmd:
More than 100 tables, each table more than 50MB
port=3306;innobackupex --defaults-file=/etc/my$port.cnf --user=root --host=127.0.0.1 --port=$port --parallel=8 jfs/data

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: 1.0.0
• OS (e.g cat /etc/os-release): centos
• Kernel (e.g. uname -a): 4.19.91
• Object storage (cloud provider and region, or self maintained): aliyun OSS
• Metadata engine info (version, cloud provider managed or self maintained): badger
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage): internal network , 10Gb net

[davies]

This should be a issue from ext4 rather than JuiceFS, which creates files concurrently. It it possible that the ext4 has run out of inodes?

[davies]

There is no much we can do here.

[wangzhhbj]

This should be a issue from ext4 rather than JuiceFS, which creates files concurrently. It it possible that the ext4 has run out of inodes?

Inodes are not exhausted, and ext4 enables lazytime to improve performance by delaying the writing of timestamp updates to disk. The current cache usage of juicefs causes ext4 to scramble for locks in find_inode_nowait, which consumes a lot of cpu.

[davies]

Understand, What's the better way to use ext4 to avoid his issue?

[wangzhhbj]

Understand, What's the better way to use ext4 to avoid his issue?

One way is to try to avoid using lazytime to mount ext4, but perhaps juicefs is not the highest priority, other components need lazytime, and they share the same disk. I was wondering if there was a way to provide caching without creating files too often, not sure if that would work.

[davies]

We don't have a plan to change the format in writeback mode, so no much we can do now, let's move it into discussion.