Clip gradient by global norm

add clip_by='global_norm' option for GradientClipping optimizer wrapper

Spectral Weight Decay

1. Add SpectralWeightDecay optimizer wrapper combining
2. Decouple Weight Decay https://arxiv.org/abs/1711.05101
3. Spectral Regularization https://arxiv.org/abs/1705.10941
4. Change sparse_update of WeightDecay to default False
5. Remove split mechanism of spectral norm
6. Add lipschitz option for spectral norm

Sparse Weight Decay

• WeightDecay supports sparse weight decay for discrete operation like tf.nn.embedding_lookup

Weight Decay Variable Filter

• WeightDecay supports variable_filter: can be container or callable

Advanced Optimizer

Implement Optimizer

• Radam: https://arxiv.org/pdf/1908.03265.pdf

And Optimizer Wrapper

• LookAhead: https://arxiv.org/abs/1907.08610
• GradientClipping
• WeightDecay: https://arxiv.org/abs/1711.05101

fix GRUCell initial state

Let GRUCell get_initial_state compatible with call

Fix extend dimension

• Fix the wrong dimension bug of extend_dimension.
• Reduce the computation of embedding dropout.
• Add new property to Embedding.configs

Embedder can extend dimensions

Main modifications:

• Embedder can have some trainable dimensions. (#103)
• Modified status badge. (#105)

Fix Embedding dropout

Merge pull request #102 from Yoctol/pass-dropout-in-cls

Fix emb dropout by sending dropout in cls

Fix Spectral Norm for Gate RNN

apply spectral norm separately on sub-kernel for different gate