Skip to content

Commit

Permalink
address comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@ChenSammi
ChenSammi committed Dec 9, 2024
1 parent 54503f5 commit 08f6298
Show file tree
Hide file tree
Showing 2 changed files with 24 additions and 27 deletions.
19 changes: 9 additions & 10 deletions hadoop-hdds/docs/content/feature/Short-Circuit-Read.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ summary: Introduction to Ozone Datanode Short Circuit Local Read Feature
limitations under the License.
-->

Current in Ozone, client reads data over GRPC from Datanode. When the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over GRPC connection.
By default, client reads data over GRPC from the Datanode. When the client asks the Datanode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a GRPC connection.

This “short-circuit” local read feature will bypass the DataNode, allowing the client to read the file from local disk directly when the client is co-located with the data on the same server.

Expand All @@ -33,13 +33,12 @@ Short-circuit local read can provide a substantial performance boost to many app

Short-circuit local reads make use of a UNIX domain socket. This is a special path in the filesystem that allows the client and the DataNodes to communicate.

The Hadoop native library "libhadoop.so" provides the support to use the Unix domain socket. Please refer to Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html") for details of this library.
The Hadoop native library `libhadoop.so` provides support to for Unix domain sockets. Please refer to Hadoop's [Native Libraries Guide](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html) for details.

Before enabling short-circuit local reads, you must have a proper libhadoop.so, and make sure it's under the directory where Java can find and load it through "System.loadLibrary()" call.
The Hadoop version used in Ozone is defined by `hadoop.version` in pom.xml. Before enabling short-circuit local reads, find the `libhadoop.so` from the corresponding version Hadoop release package, put it under one of the directories specified by Java `java.library.path` property. The default value of `java.library.path` depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.

The paths that Java will search for libraries are specified by the "java.library.path" property. The default value of "java.library.path" depends on the OS and Java version. For example, on Linux with OpenJDK 8 it is `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`.
The `ozone checknative` command can be used to detect whether `libhadoop.so` can be found and loaded successfully by Ozone service.

Command "ozone checknative" can be used to detect whether libhadoop.so can be loaded successfully by Ozone service.

## Configuration

Expand All @@ -63,14 +62,14 @@ It makes use of a UNIX domain socket, a special path in the filesystem. You will
</property>
```

The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under /var/run or /var/lib are often used， just like the current default value "/var/lib/ozone_dn_socket".
The DataNode needs to be able to create this path. On the other hand, it should not be possible for any user except the Ozone user(user who launches Ozone service) or root to create this path. For this reason, paths under `/var/run` or `/var/lib` are often used， just like the current default value `/var/lib/ozone_dn_socket`.

If you configure the "ozone.domain.socket.path" to other value, for example "/dir1/dir2/ozone_dn_socket", please make sure that both dir1 and dir2 are exiting directories, and there is no ozone_dn_socket under dir2. ozone_dn_socket will be created by Ozone Datanode later during Datanode start.
If you configure the `ozone.domain.socket.path` to another value, for example `/dir1/dir2/ozone_dn_socket`, please make sure that both `dir1` and `dir2` are exiting directories, but the file `ozone_dn_socket` does not exist under `dir2`. `ozone_dn_socket` will be created by Ozone Datanode later during its startup.

### Security Consideration

To ensure data security and integrity, Ozone will follow the rule as Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") on "ozone.domain.socket.path" path permission check.
It will fail the "ozone.domain.socket.path" verification and disable the feature if the filesystem permissions of the specified path are inadequate.
To ensure data security and integrity, Ozone will follow the same rules as Hadoop to check permission on the `ozone.domain.socket.path` path as documented in [Socket Path Security](https://wiki.apache.org/hadoop/SocketPathSecurity).
It will fail the `ozone.domain.socket.path` verification and disable the feature if the filesystem permissions of the specified path are inadequate.
The verification failure message carries detail instruction about how to fix the problem. Following is an example,

"The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity"
`The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity`
32 changes: 15 additions & 17 deletions hadoop-hdds/docs/content/feature/Short-Circuit-Read.zh.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ weight: 2
menu:
main:
parent: 特性
summary: Ozone DataNode 本地短路读功能介绍
summary: Ozone Datanode 本地短路读功能介绍
---
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
Expand All @@ -23,26 +23,24 @@ summary: Ozone DataNode 本地短路读功能介绍
limitations under the License.
-->

当前在 Ozone 中，客户端使用 GRPC 通道从 DataNode 读取数据。当客户端向 DataNode 请求读取一个文件时，DataNode 将文件从本次磁盘读到内存，然后通过 GRPC 通道发回给客户端。
当前在 Ozone 中，客户端使用 GRPC 通道从 Datanode 读取数据。当客户端向 Datanode 请求读取一个文件时，Datanode 将文件从本次磁盘读到内存，然后通过 GRPC 通道发回给客户端。

DataNode 本地短路读功能，当客户端和 DataNode 在同一个机器时，允许客户端绕过 DataNode，直接从本地磁盘读取文件内容。通过绕过 DataNode，去掉网络通信带来的开销，DataNode 本地短路读功能将帮助许多 Ozone 应用，提升读性能。
Datanode 本地短路读功能，当客户端和 Datanode 在同一个机器时，允许客户端绕过 Datanode，直接从本地磁盘读取文件内容。通过绕过 Datanode，去掉网络通信带来的开销，Datanode 本地短路读功能将帮助许多 Ozone 应用，提升读性能。

## 前提

DataNode 本地短路读功能基于 Unix domain socket 实现。 Unix domain socket 是一个特殊的文件系统路径，支持客户端和 DataNode 通过它交互传递信息。
Datanode 本地短路读功能基于 Unix domain socket 实现。 Unix domain socket 是一个特殊的文件系统路径，支持客户端和 Datanode 通过它交互传递信息。

DataNode 本地短路读功能需要用到 Hadoop 本地库 libhadoop.so。 libhadoop.so 提供了调用 Unix domain socket 的功能。该本地库的详细信息，详见 Native Libraries ("https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html")。
Datanode 本地短路读功能需要用到 Hadoop 本地库 `libhadoop.so`。 `libhadoop.so` 提供了调用 Unix domain socket 的功能。该本地库的详细信息，详见 [Native Libraries](https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/NativeLibraries.html)。

在启用 DataNode 本地短路读功能前，确保获取一个可用的 libhadoop.so 文件，并将该文件放置在 Java System.loadLibrary() 调用能搜寻到路径下。
Ozone 依赖的 Hadoop 版本，由 pom.xml 里的 `hadoop.version` 变量定义. 在启用 Datanode 本地短路读功能前，从对应的 Hadoop 版本发布获取对应的libhadoop.so 文件，将该文件放置在任一 Java 变量 `java.library.path` 定义的目录下。`java.library.path` 的默认值取决于操作系统和 JAVA 版本。例如，在 Linux 上 OpenJDK 8 的默认值是 `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`。

Java 环境变量 "java.library.path" 定义了 Java 的动态库搜寻路径。"java.library.path" 的默认值取决于操作系统和 JAVA 版本。例如，在 Linux 上 OpenJDK 8 的默认值是 `/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib`。

当放置好 libhadoop.so 后，可使用命令 "ozone checknative" 来查看 libhadoop.so 是否能被 Ozone的服务进程正确的搜寻和加载到。
在放置好 `libhadoop.so` 后，可使用命令 `ozone checknative` 来查看 `libhadoop.so` 是否能被 Ozone的服务进程正确的搜寻和加载到。


## 配置

DataNode 本地短路读功能需要在客户端和 DataNode 端同时配置。 默认情况下，它是关闭的。
Datanode 本地短路读功能需要在客户端和 Datanode 端同时配置。 默认情况下，它是关闭的。

```XML
<property>
Expand All @@ -52,7 +50,7 @@ DataNode 本地短路读功能需要在客户端和 DataNode 端同时配置。
</property>
```

DataNode 本地短路读基于 UNIX domain socket。以下变量将配置 domain socket 路径。
Datanode 本地短路读基于 UNIX domain socket。以下变量将配置 domain socket 路径。

```XML
<property>
Expand All @@ -62,17 +60,17 @@ DataNode 本地短路读基于 UNIX domain socket。以下变量将配置 domain
</property>
```

DataNode 需要能创建该路径. 同时，除了启动 Ozone 服务的用户和 root 用户，其他用户不能创建该路径。 由于有这些限制，路径经常使用 /var/run 或者 /var/lib 下的子目录, 正如当前的默认值 "/var/lib/ozone_dn_socket" 一样。
Datanode 需要能创建该路径. 同时，除了启动 Ozone 服务的用户和 root 用户，其他用户不能创建该路径。 由于有这些限制，路径经常使用 `/var/run` 或者 `/var/lib` 下的子目录, 正如当前的默认值 `/var/lib/ozone_dn_socket` 一样。

如果修改了 "ozone.domain.socket.path" 的值，比如设置成 "/dir1/dir2/ozone_dn_socket"，请确保 dir1 和 dir2 是已存在的目录，并且 dir2 下还没有 ozone_dn_socket 文件。 ozone_dn_socket 将在 DataNode 启动的时候由 DataNode 创建。
如果修改了 `ozone.domain.socket.path` 的值，比如设置成 `/dir1/dir2/ozone_dn_socket`，请确保 `dir1` 和 `dir2` 是已存在的目录，并且 `dir2` 下还没有 `ozone_dn_socket` 文件。 `ozone_dn_socket` 将在 Datanode 启动的时候由 Datanode 创建。


### 安全考量

为了确保数据的安全和完整性，Ozone 在 "ozone.domain.socket.path" 路径的权限上，将遵守和 Hadoop ("https://cwiki.apache.org/confluence/display/HADOOP2/SocketPathSecurity") 一样的规则。
为了确保数据的安全和完整性，Ozone 在 `ozone.domain.socket.path` 路径的权限检查上，将遵守和 Hadoop [Socket路径安全](https://wiki.apache.org/hadoop/SocketPathSecurity) 一样的规则。

如果 "ozone.domain.socket.path" 路径权限验证失败， 该功能将自动关闭。
如果 `ozone.domain.socket.path` 路径权限检查失败，该功能将自动关闭。

验证失败返回的信息包含修复问题的指引，例如
检查失败返回的信息包含修复问题的指引，例如

"The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity"
`The path component: '/etc/hadoop' in '/etc/hadoop/ozone_dn_socket' has permissions 0777 uid 0 and gid 0. It is not protected because it is world-writable. This might help: 'chmod o-w /etc/hadoop'. For more information: https://wiki.apache.org/hadoop/SocketPathSecurity`

0 comments on commit 08f6298

Please sign in to comment.