Skip to content

Latest commit

 

History

History
162 lines (105 loc) · 13.7 KB

redis_best_practices.md

File metadata and controls

162 lines (105 loc) · 13.7 KB
Raw
sidebar_label sidebar_position slug
Redis
1
/redis_best_practices

Redis Best Practices

This is a guide about Redis best practices. Redis is a critical component in JuiceFS architecture. It stores all the file system metadata and serve metadata operation from client. If Redis has any problem (either service unavailable or lose data), it will affect the user experience.

:::tip It's highly recommended to use Redis service managed by public cloud provider if possible. See "Recommended Managed Redis Service" for more information. :::

If you still want to operate Redis by yourself in production environment, please keep in mind that JuiceFS requires Redis version 4.0+. Moreover, it's recommended to pick an official stable version, and read the following contents before deploying Redis.

:::note Part of the content in this article comes from the Redis official website. If there is any inconsistency, please refer to the official Redis document. :::

Memory usage

The space used by the JuiceFS metadata engine is mainly related to the number of files in the file system. According to our experience, the metadata of each file occupies approximately 300 bytes of memory. Therefore, if you want to store 100 million files, approximately 30 GiB of memory is required.

You can check the specific memory usage through Redis's INFO memory command, for example:

> INFO memory
used_memory: 19167628056
used_memory_human: 17.85G
used_memory_rss: 20684886016
used_memory_rss_human: 19.26G
...
used_memory_overhead: 5727954464
...
used_memory_dataset: 13439673592
used_memory_dataset_perc: 70.12%

Among them, used_memory_rss is the total memory size actually used by Redis, which includes not only the size of data stored in Redis (that is, used_memory_dataset above), but also some Redis system overhead (that is, used_memory_overhead above). As mentioned earlier, the metadata of each file occupies about 300 bytes and is calculated by used_memory_dataset. If you find that the metadata of a single file in your JuiceFS file system occupies much more than 300 bytes, you can try to run juicefs gc command to clean up possible redundant data.

High Availability

:::caution JuiceFS uses "Redis transactions" to guarantee the atomicity of metadata operations. However, since Redis Cluster does not support transaction, Redis Cluster cannot be used as a JuiceFS metadata store. For high availability of Redis, please use Redis Sentinel. :::

Redis Sentinel is the official high availability solution for Redis. It provides following capabilities:

  • Monitoring. Sentinel constantly checks if your master and replica instances are working as expected.
  • Notification. Sentinel can notify the system administrator, or other computer programs, via an API, that something is wrong with one of the monitored Redis instances.
  • Automatic failover. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications using the Redis server are informed about the new address to use when connecting.
  • Configuration provider. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address.

A stable release of Redis Sentinel is shipped since Redis 2.8. Redis Sentinel version 1, shipped with Redis 2.6, is deprecated and should not be used.

There're some fundamental things you need to know about Redis Sentinel before using it:

  1. You need at least three Sentinel instances for a robust deployment.
  2. The three Sentinel instances should be placed into computers or virtual machines that are believed to fail in an independent way. So for example different physical servers or Virtual Machines executed on different availability zones.
  3. Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures, since Redis uses asynchronous replication. However there are ways to deploy Sentinel that make the window to lose writes limited to certain moments, while there are other less secure ways to deploy it.
  4. There is no HA setup which is safe if you don't test from time to time in development environments, or even better if you can, in production environments, if they work. You may have a misconfiguration that will become apparent only when it's too late (at 3am when your master stops working).
  5. Sentinel, Docker, or other forms of Network Address Translation or Port Mapping should be mixed with care: Docker performs port remapping, breaking Sentinel auto discovery of other Sentinel processes and the list of replicas for a master.

Please read the official documentation for more information.

Once Redis servers and Sentinels are deployed, the META-URL can be specified as redis[s]://[[USER]:PASSWORD@]MASTER_NAME,SENTINEL_ADDR[,SENTINEL_ADDR]:SENTINEL_PORT[/DB], for example:

$ ./juicefs mount redis://:password@masterName,1.2.3.4,1.2.5.6:26379/2 ~/jfs

:::tip For v0.16+, the PASSWORD in the URL will be used to connect Redis server, the password for Sentinel should be provided using environment variable SENTINEL_PASSWORD. For early versions, the PASSWORD is used for both Redis server and Sentinel, they can be overrode by environment variables SENTINEL_PASSWORD and REDIS_PASSWORD. :::

:::tip Since JuiceFS v1.0.0, it is supported to connect only Redis replica nodes when mounting file systems to reduce the load on Redis master node. In order to enable this feature, you must mount the JuiceFS file system in read-only mode (that is, set the --read-only mount option), and connect to the metadata engine through the Redis Sentinel. Finally, you need to add ?route-read=replica to the end of the metadata URL. For example: redis://:password@masterName,1.2.3.4,1.2.5.6:26379/2?route-read=replica.

It should be noted that since the data of the Redis master node is asynchronously replicated to the replica nodes, it is possible that the metadata read may not be the latest. :::

Data Durability

Redis provides a different range of persistence options:

  • RDB: The RDB persistence performs point-in-time snapshots of your dataset at specified intervals.
  • AOF: The AOF persistence logs every write operation received by the server, that will be played again at server startup, reconstructing the original dataset. Commands are logged using the same format as the Redis protocol itself, in an append-only fashion. Redis is able to rewrite the log in the background when it gets too big.
  • RDB+AOF Recommended: It is possible to combine both AOF and RDB in the same instance. Notice that, in this case, when Redis restarts the AOF file will be used to reconstruct the original dataset since it is guaranteed to be the most complete.

When using AOF, you can have different fsync policies:

  1. No fsync
  2. fsync every second Default
  3. fsync at every query

With the default policy of fsync every second write performances are still great (fsync is performed using a background thread and the main thread will try hard to perform writes when no fsync is in progress.) but you can only lose one second worth of writes.

The disk may be damaged, the virtual machine may disappear, even if the RBD+AOF mode is adopted, Redis data needs to be backed up regularly.

Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename only when the new snapshot is complete. You can also copy the AOF file in order to create backups.

Please read the official documentation for more information.

Backing up Redis Data

Make Sure to Backup Your Database. Disks break, instances in the cloud disappear, and so forth.

By default Redis saves snapshots of the dataset on disk, in a binary file called dump.rdb. You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually call the SAVE or BGSAVE commands.

Redis is very data backup friendly since you can copy RDB files while the database is running: the RDB is never modified once produced, and while it gets produced it uses a temporary name and is renamed into its final destination atomically using rename(2) only when the new snapshot is complete.

This means that copying the RDB file is completely safe while the server is running. This is what we suggest:

  • Create a cron job in your server creating hourly snapshots of the RDB file in one directory, and daily snapshots in a different directory.
  • Every time the cron script runs, make sure to call the find command to make sure too old snapshots are deleted: for instance you can take hourly snapshots for the latest 48 hours, and daily snapshots for one or two months. Make sure to name the snapshots with data and time information.
  • At least one time every day make sure to transfer an RDB snapshot outside your data center or at least outside the physical machine running your Redis instance.

Please read the official documentation for more information.

Restore Redis Data

After generating the AOF or RDB backup file, you can restore the data by copying the backup file to the path corresponding to the dir configuration of the new Redis instance, which you can get by using the CONFIG GET dir command.

If both AOF and RDB persistence are enabled, Redis will start using the AOF file first to recover the data because AOF is guaranteed to be the most complete data.

After recovering Redis data, you can continue to use the JuiceFS file system with the new Redis address. It is recommended to run juicefs fsck command to check the integrity of the file system data.

Recommended Managed Redis Service

Amazon ElastiCache for Redis

Amazon ElastiCache for Redis is a fully managed, Redis-compatible in-memory data store built for the cloud. It provides automatic failover, automatic backup features to ensure availability and durability.

:::info Amazon ElastiCache for Redis has two type: cluster mode disabled and cluster mode enabled. Because JuiceFS uses transaction to guarantee the atomicity of metadata operations, so you couldn't use "cluster mode enabled" type. :::

Google Cloud Memorystore for Redis

Google Cloud Memorystore for Redis is a fully managed Redis service for the Google Cloud. Applications running on Google Cloud can achieve extreme performance by leveraging the highly scalable, available, secure Redis service without the burden of managing complex Redis deployments.

Azure Cache for Redis

Azure Cache for Redis is a fully managed, in-memory cache that enables high-performance and scalable architectures. Use it to create cloud or hybrid deployments that handle millions of requests per second at sub-millisecond latency-all with the configuration, security, and availability benefits of a managed service.

Alibaba Cloud ApsaraDB for Redis

Alibaba Cloud ApsaraDB for Redis is a database service that is compatible with native Redis protocols. It supports a hybrid of memory and hard disks for data persistence. ApsaraDB for Redis provides a highly available hot standby architecture and can scale to meet requirements for high-performance and low-latency read/write operations.

:::info ApsaraDB for Redis supports 3 type architectures: standard, cluster and read/write splitting. Because JuiceFS uses transaction to guarantee the atomicity of metadata operations, so you couldn't use cluster type architecture. :::

Tencent Cloud TencentDB for Redis

Tencent Cloud TencentDB for Redis is a caching and storage service compatible with the Redis protocol. It features a rich variety of data structure options to help you develop different types of business scenarios, and offers a complete set of database services such as primary-secondary hot backup, automatic switchover for disaster recovery, data backup, failover, instance monitoring, online scaling and data rollback.

:::info TencentDB for Redis supports 2 type architectures: standard and cluster. Because JuiceFS uses transaction to guarantee the atomicity of metadata operations, so you couldn't use cluster type architecture. :::