Skip to content

Commit

Permalink
add documents
Browse files Browse the repository at this point in the history
  • Loading branch information
yangxudong committed Sep 19, 2023
1 parent 02d4dce commit 8672e47
Show file tree
Hide file tree
Showing 18 changed files with 887 additions and 29 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,10 +61,10 @@ Running Platform:
### A variety of models

- [DSSM](docs/source/models/dssm.md) / [MIND](docs/source/models/mind.md) / [DropoutNet](docs/source/models/dropoutnet.md) / [CoMetricLearningI2I](docs/source/models/co_metric_learning_i2i.md) / [PDN](docs/source/models/pdn.md)
- [W&D](docs/source/models/wide_and_deep.md) / [DeepFM](docs/source/models/deepfm.md) / [MultiTower](docs/source/models/multi_tower.md) / [DCN](docs/source/models/dcn.md) / [FiBiNet](docs/source/models/fibinet.md) / [MaskNet](docs/source/models/masknet.md)
- [W&D](docs/source/models/wide_and_deep.md) / [DeepFM](docs/source/models/deepfm.md) / [MultiTower](docs/source/models/multi_tower.md) / [DCN](docs/source/models/dcn.md) / [FiBiNet](docs/source/models/fibinet.md) / [MaskNet](docs/source/models/masknet.md) / [CDN](docs/source/models/cdn.md)
- [DIN](docs/source/models/din.md) / [BST](docs/source/models/bst.md)
- [MMoE](docs/source/models/mmoe.md) / [ESMM](docs/source/models/esmm.md) / [DBMTL](docs/source/models/dbmtl.md) / [PLE](docs/source/models/ple.md)
- [CMBF](docs/source/models/cmbf.md) / [UNITER](docs/source/models/uniter.md)
- [HighwayNetwork](docs/source/models/highway.md) / [CMBF](docs/source/models/cmbf.md) / [UNITER](docs/source/models/uniter.md)
- More models in development

### Easy to customize
Expand Down
Binary file added docs/images/models/cdn.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/models/dcn_cross.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 18 additions & 12 deletions docs/source/component/backbone.md
Original file line number Diff line number Diff line change
Expand Up @@ -963,17 +963,23 @@ MovieLens-1M数据集效果:
- DIN模型配置文件:[DIN_backbone.config](https://github.com/alibaba/EasyRec/blob/master/samples/model_config/din_backbone_on_taobao.config)
- BST模型配置文件:[BST_backbone.config](https://github.com/alibaba/EasyRec/blob/master/samples/model_config/bst_backbone_on_taobao.config)

其他模型:

- Highway Network: [highway network](../models/highway.md)

# 组件库介绍

## 1.基础组件

| 类名 | 功能 | 说明 | 示例 |
| ----------------- | ------ | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| MLP | 多层感知机 | 支持配置激活函数、初始化方法、Dropout、是否使用BN等 | [案例1](#wide-deep) |
| Highway | 类似残差链接 | 可用来对预训练embedding做增量微调,来自Highway Network | |
| Gate | 门控 | 多个输入的加权求和 | |
| PeriodicEmbedding | 周期激活函数 | 数值特征Embedding | [案例5](#dlrm-embedding) |
| AutoDisEmbedding | 自动离散化 | 数值特征Embedding | [dlrm_on_criteo_with_autodis.config](https://github.com/alibaba/EasyRec/tree/master/examples/configs/dlrm_on_criteo_with_autodis.config) |
| 类名 | 功能 | 说明 | 示例 |
| ----------------- | ------ | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| MLP | 多层感知机 | 可定制激活函数、initializer、Dropout、BN等 | [案例1](#wide-deep) |
| Highway | 类似残差链接 | 可用来对预训练embedding做增量微调 | [highway network](../models/highway.html) |
| Gate | 门控 | 多个输入的加权求和 | [Cross Decoupling Network](../models/cdn.html#id2) |
| PeriodicEmbedding | 周期激活函数 | 数值特征Embedding | [案例5](#dlrm-embedding) |
| AutoDisEmbedding | 自动离散化 | 数值特征Embedding | [dlrm_on_criteo_with_autodis.config](https://github.com/alibaba/EasyRec/tree/master/examples/configs/dlrm_on_criteo_with_autodis.config) |

**备注**:Gate组件的第一个输入是权重向量,后面的输入拼凑成一个列表,权重向量的长度应等于列表的长度

## 2.特征交叉组件

Expand All @@ -987,11 +993,11 @@ MovieLens-1M数据集效果:

## 3.特征重要度学习组件

| 类名 | 功能 | 说明 | 示例 |
| --------- | ----------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------- |
| SENet | 建模特征重要度 | FiBiNet模型的组件 | [fibinet_on_movielens.config](https://github.com/alibaba/EasyRec/tree/master/examples/configs/fibinet_on_movielens.config) |
| MaskBlock | 建模特征重要度 | MaskNet模型的组件 | [masknet_on_movielens.config](https://github.com/alibaba/EasyRec/tree/master/examples/configs/masknet_on_movielens.config) |
| MaskNet | 多个串行或并行的MaskBlock | MaskNet模型 | [masknet_on_movielens.config](https://github.com/alibaba/EasyRec/tree/master/examples/configs/masknet_on_movielens.config) |
| 类名 | 功能 | 说明 | 示例 |
| --------- | ----------------- | ------------ | ----------------------------------------------------- |
| SENet | 建模特征重要度 | FiBiNet模型的组件 | [MMoE](../models/mmoe.html#id4) |
| MaskBlock | 建模特征重要度 | MaskNet模型的组件 | [Cross Decoupling Network](../models/cdn.html#id2) |
| MaskNet | 多个串行或并行的MaskBlock | MaskNet模型 | [DBMTL](../models/dbmtl.html#dbmtl-based-on-backbone) |

## 4. 序列特征编码组件

Expand Down
13 changes: 8 additions & 5 deletions docs/source/component/component.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,13 @@

- HighWay

| 参数 | 类型 | 默认值 | 说明 |
| ------------ | ------ | ---- | ------------ |
| emb_size | uint32 | | embedding维度 |
| activation | str | gelu | 激活函数 |
| dropout_rate | float | | dropout rate |
| 参数 | 类型 | 默认值 | 说明 |
| -------------- | ------ | ---- | ------------ |
| emb_size | uint32 | None | embedding维度 |
| activation | str | gelu | 激活函数 |
| dropout_rate | float | 0 | dropout rate |
| init_gate_bias | float | -3.0 | 门控网络的bias初始值 |
| num_layers | int | 1 | 网络层数 |

- PeriodicEmbedding

Expand Down Expand Up @@ -116,4 +118,5 @@
| ----------- | ------ | --- | --------------------------- |
| loss_type | string | | 损失函数类型,包括:l2_loss, info_nce |
| loss_weight | float | 1.0 | 损失函数权重 |
| temperature | float | 0.1 | info_nce loss 的参数 |
| 其他 | | | 根据loss_type决定 |
142 changes: 142 additions & 0 deletions docs/source/models/cdn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# Cross Decoupling Network

### 简介

推荐系统中的item反馈通常是长尾分布的,即满足二八定律,少数item接收了大部分用户的反馈。在长尾分布下,推荐的item是有偏的,这会导致推荐效果变差。
交叉解耦网络CDN针对这类问题提出了相应的方法。

- 通过混合专家结构,在item段解耦记忆和泛化的学习过程
- 通过正则化的BBN(BBN是图像分类领域提出的方法)从不同分布中解耦用户表征(可参考这篇文章:BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition)
- 使用一种新的适配器来聚合解耦的向量,并将训练注意力转移到长尾item上。

![cdn](../../images/models/cdn.jpg)

### 配置说明

```protobuf
model_config: {
model_name: 'Cross Decoupling Network'
model_class: 'RankModel'
feature_groups: {
group_name: 'memorize'
feature_names: 'user_id'
feature_names: 'adgroup_id'
wide_deep: DEEP
}
feature_groups: {
group_name: 'general'
feature_names: 'cms_segid'
feature_names: 'cms_group_id'
feature_names: 'age_level'
feature_names: 'pvalue_level'
feature_names: 'shopping_level'
feature_names: 'occupation'
feature_names: 'new_user_class_level'
feature_names: 'cate_id'
feature_names: 'campaign_id'
feature_names: 'customer'
feature_names: 'brand'
feature_names: 'price'
feature_names: 'tag_category_list'
feature_names: 'tag_brand_list'
wide_deep: DEEP
}
feature_groups: {
group_name: 'frequency'
feature_names: 'pid'
wide_deep: DEEP
}
backbone {
blocks {
name: "mem_expert"
inputs {
feature_group_name: "memorize"
}
keras_layer {
class_name: "MLP"
mlp {
hidden_units: [512, 256]
}
}
}
blocks {
name: "gen_experts"
inputs {
feature_group_name: "general"
input_fn: "lambda x: [x, x]"
}
repeat {
num_repeat: 3
keras_layer {
class_name: "MaskBlock"
mask_block {
output_size: 256
aggregation_size: 1024
}
}
}
}
blocks {
name: "gate_weight"
inputs {
feature_group_name: "frequency"
}
keras_layer {
class_name: "MLP"
mlp {
hidden_units: 4
use_final_bn: false
final_activation: "softmax"
}
}
}
blocks {
name: "gate"
inputs {
block_name: "gate_weight"
input_fn: "lambda x: [x]"
}
inputs {
block_name: "mem_expert"
input_fn: "lambda x: [x]"
}
inputs {
block_name: "gen_experts"
}
keras_layer {
class_name: "Gate"
}
}
top_mlp {
hidden_units: [128, 64]
}
}
model_params {
l2_regularization: 1e-6
}
embedding_regularization: 1e-5
}
```

- model_name: 任意自定义字符串,仅有注释作用
- model_class: 'RankModel', 不需要修改, 通过组件化方式搭建的单目标排序模型都叫这个名字
- feature_groups: 配置一组特征。
- backbone: 通过组件化的方式搭建的主干网络,[参考文档](../component/backbone.md)
- blocks: 由多个`组件块`组成的一个有向无环图(DAG),框架负责按照DAG的拓扑排序执行个`组件块`关联的代码逻辑,构建TF Graph的一个子图
- name/inputs: 每个`block`有一个唯一的名字(name),并且有一个或多个输入(inputs)和输出
- keras_layer: 加载由`class_name`指定的自定义或系统内置的keras layer,执行一段代码逻辑;[参考文档](../component/backbone.md#keraslayer)
- repeat: 重复组件块,可多次执行某个keras_layer,默认输出一个list,[参考文档](../component/backbone.md#id21)
- mask_block: MaskNet的基础组件,参数详见[参考文档](../component/component.md#id4)
- Gate: 门控组件,实现对输入的加权求和;第一个输入是权重向量,后面的输入拼凑成一个列表,权重向量的长度应等于列表的长度
- concat_blocks: DAG的输出节点由`concat_blocks`配置项定义,如果不配置`concat_blocks`,框架会自动拼接DAG的所有叶子节点并输出。
- model_params:
- l2_regularization: (可选) 对DNN参数的regularization, 减少overfit
- embedding_regularization: 对embedding部分加regularization, 减少overfit

### 示例Config

[cdn_on_taobao.config](https://github.com/alibaba/EasyRec/tree/master/samples/model_config/cdn_on_taobao.config)

### 参考论文

[Cross Decoupling Network](https://arxiv.org/pdf/2210.14309.pdf)
1 change: 1 addition & 0 deletions docs/source/models/dcn.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
Deep&Cross Network(DCN)是在DNN模型的基础上,引入了一种新型的交叉网络,该网络在学习某些特征交叉时效率更高。特别是,DCN显式地在每一层应用特征交叉,不需要人工特征工程,并且只增加了很小的额外复杂性。

![dcn.png](../../images/models/dcn.png)
![dcn_cross.png](../../images/models/dcn_cross.jpg)

DCN-V2相对于前一个版本的模型,主要的改进点在于:

Expand Down
85 changes: 85 additions & 0 deletions docs/source/models/highway.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Highway Network

### 简介

传统的神经网络随着深度的增加,训练越来越困难。Highway Network使用简单的SGD就可以训练很深的网络,收敛速度更快。并且Highway Network还可以用来以增量的方式微调预训练好的embedding特征。

### 配置说明

```protobuf
model_config: {
model_name: 'HighWayNetwork'
model_class: 'RankModel'
feature_groups: {
group_name: 'image'
feature_names: 'embedding'
wide_deep: DEEP
}
feature_groups: {
group_name: 'general'
feature_names: 'user_id'
feature_names: 'movie_id'
feature_names: 'gender'
feature_names: 'age'
feature_names: 'occupation'
feature_names: 'zip_id'
feature_names: 'movie_year_bin'
feature_names: 'title'
feature_names: 'genres'
feature_names: 'score_year_diff'
feature_names: 'score_time'
wide_deep: DEEP
}
backbone {
blocks {
name: 'highway'
inputs {
feature_group_name: 'image'
}
keras_layer {
class_name: 'Highway'
}
}
blocks {
name: 'top_mlp'
inputs {
feature_group_name: 'general'
}
inputs {
block_name: 'highway'
}
keras_layer {
class_name: 'MLP'
mlp {
hidden_units: [256, 128, 64]
}
}
}
}
model_params {
l2_regularization: 1e-6
}
embedding_regularization: 1e-6
}
```

- model_name: 任意自定义字符串,仅有注释作用
- model_class: 'RankModel', 不需要修改, 通过组件化方式搭建的单目标排序模型都叫这个名字
- feature_groups: 配置一组特征。
- backbone: 通过组件化的方式搭建的主干网络,[参考文档](../component/backbone.md)
- blocks: 由多个`组件块`组成的一个有向无环图(DAG),框架负责按照DAG的拓扑排序执行个`组件块`关联的代码逻辑,构建TF Graph的一个子图
- name/inputs: 每个`block`有一个唯一的名字(name),并且有一个或多个输入(inputs)和输出
- keras_layer: 加载由`class_name`指定的自定义或系统内置的keras layer,执行一段代码逻辑;[参考文档](../component/backbone.md#keraslayer)
- Highway: 使用Highway Network微调图像embedding。组件的参数,详见[参考文档](../component/component.md#id2)
- concat_blocks: DAG的输出节点由`concat_blocks`配置项定义,如果不配置`concat_blocks`,框架会自动拼接DAG的所有叶子节点并输出。
- model_params:
- l2_regularization: (可选) 对DNN参数的regularization, 减少overfit
- embedding_regularization: 对embedding部分加regularization, 减少overfit

### 示例Config

[highway_on_movielens.config](https://github.com/alibaba/EasyRec/tree/master/samples/model_config/highway_on_movielens.config)

### 参考论文

[Highway Network](https://arxiv.org/pdf/1505.00387.pdf)
2 changes: 2 additions & 0 deletions docs/source/models/rank.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
rocket_launching
masknet
fibinet
cdn
regression
multi_cls

Expand All @@ -25,5 +26,6 @@
.. toctree::
:maxdepth: 2

highway
cmbf
uniter
28 changes: 24 additions & 4 deletions easy_rec/python/layers/backbone.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,24 @@ def load_keras_layer(self, layer_conf, name, reuse=None):
else:
pb_params = getattr(layer_conf, param_type)
params = Parameter(pb_params, False, l2_reg=self._l2_reg)
layer = layer_cls(params, name=name, reuse=reuse)

has_reuse = True
try:
from funcsigs import signature
sig = signature(layer_cls.__init__)
has_reuse = 'reuse' in sig.parameters.keys()
except ImportError:
try:
from sklearn.externals.funcsigs import signature
sig = signature(layer_cls.__init__)
has_reuse = 'reuse' in sig.parameters.keys()
except ImportError:
logging.warning('import funcsigs failed')

if has_reuse:
layer = layer_cls(params, name=name, reuse=reuse)
else:
layer = layer_cls(params, name=name)
return layer, customize
elif param_type is None: # internal keras layer
layer = layer_cls(name=name)
Expand Down Expand Up @@ -274,9 +291,12 @@ def call_keras_layer(self, inputs, name, training):
if customize:
output = layer(inputs, training=training, **kwargs)
else:
output = layer(inputs, training=training)
if cls == 'BatchNormalization':
add_elements_to_collection(layer.updates, tf.GraphKeys.UPDATE_OPS)
try:
output = layer(inputs, training=training)
if cls == 'BatchNormalization':
add_elements_to_collection(layer.updates, tf.GraphKeys.UPDATE_OPS)
except TypeError:
output = layer(inputs)
return output

def call_layer(self, inputs, config, name, training):
Expand Down
Loading

0 comments on commit 8672e47

Please sign in to comment.