Feature Proposal: optionloader for Kitex client/server

[felix021]

Currently Kitex users have to manually intialize a client/server with options, for example:

cli, err := somService.NewClient(svcName, client.WithHostPorts(addr))

If there's a need to change the options, one has to modify the code and recompile it, which is no doubt tedious work.

In this proposal, we suggest implementing a library optionloader for loading options from certain source (e.g. some local yaml file), which is capable of loading frequently used options out-of-the-box, and is also extensible by allowing users to register config-to-option translators without modify the library itself.

With this library, users will be able to initialize a client (or server) this way:

loader := optionloader.NewClientOptionLoader()

loader.RegisterTranslator("timeout", func (config map[string]interface{}) client.Option {
    // custom implementation
})

reader := optionloader.NewFileReader(someFile)

loader.SetSource(reader)

options, err := optionloader.Load()

cli, err := somService.NewClient(svcName, options...)

Note: the above is just pseudo code to show the core concept, and is not strict stardard for final design. For example, some user may prefer a default value (other than the default value in Kitex) when a config item does not appear in the yaml file.

To make this library handier, it's preferably better to support more config sources like etcd/consul, and also the translation for callopt.Option, streamclient.Option and streamcall.Option.

This library shall be released under https://github.com/kitex-contrib .

If you are interested in implementing this feature, please kindly prepare a detailed tech plan and reply with your lark id for us.

[Printemps417]

I want to try to solve this issue.

[felix021]

I want to try to solve this issue.

Please prepare a detailed tech plan.

[kaori-seasons]

@felix021
When implementing an options loader, we can follow these steps:

Define the core interface:

• 1. Define the core interface of the option loader, including registering the translator, setting the configuration source, loading options, etc.
• 2. Implement the option loader: Implement the option loader to load options according to the registered translator and configuration source.
• 3. Register configurations and translators: Allows users to register configurations and custom translators to map configurations to Kitex options.
• 4. Support different configuration sources: implement different configuration sources, such as file readers, etcd readers, etc.

The following is the sample code

package optionloader

import (
	"io/ioutil"
	"gopkg.in/yaml.v2"
	"github.com/kitex-contrib/kitex-client/client"
)

// 选项加载器的接口
type OptionLoader interface {
	RegisterTranslator(name string, translator Translator)
	SetSource(source ConfigSource)
	Load() ([]client.Option, error)
}

// 翻译器的接口
type Translator func(config map[string]interface{}) client.Option

// 配置源的接口
type ConfigSource interface {
	ReadConfig() (map[string]interface{}, error)
}

// 文件配置源的实现
type FileConfigSource struct {
	Filepath string
}

// 从文件中读取配置
func (f *FileConfigSource) ReadConfig() (map[string]interface{}, error) {
	data, err := ioutil.ReadFile(f.Filepath)
	if err != nil {
		return nil, err
	}

	var config map[string]interface{}
	err = yaml.Unmarshal(data, &config)
	if err != nil {
		return nil, err
	}

	return config, nil
}

//  OptionLoader 接口的实现
type OptionLoaderImpl struct {
	translators map[string]Translator
	source      ConfigSource
}

//  创建一个新的选项加载器
func NewOptionLoader() OptionLoader {
	return &OptionLoaderImpl{
		translators: make(map[string]Translator),
	}
}

//  注册翻译器
func (loader *OptionLoaderImpl) RegisterTranslator(name string, translator Translator) {
	loader.translators[name] = translator
}

//  设置配置源
func (loader *OptionLoaderImpl) SetSource(source ConfigSource) {
	loader.source = source
}

//  加载选项
func (loader *OptionLoaderImpl) Load() ([]client.Option, error) {
	config, err := loader.source.ReadConfig()
	if err != nil {
		return nil, err
	}

	var options []client.Option
	for key, value := range config {
		translator, ok := loader.translators[key]
		if !ok {
			continue
		}

		option := translator(value.(map[string]interface{}))
		options = append(options, option)
	}

	return options, nil
}

// 创建一个新的客户端选项加载器
func NewClientOptionLoader() ClientOptionLoader {
	return &OptionLoaderImpl{
		translators: make(map[string]Translator),
	}
}

func NewClientOptionLoader(translators map[string]Translator) OptionLoader {
	loader := NewOptionLoader()

	for name, translator := range translators {
		loader.RegisterTranslator(name, translator)
	}

	return loader
}

// 创建一个新的文件读取器
func NewFileReader(filepath string) ConfigSource {
	return &FileConfigSource{
		Filepath: filepath,
	}
}


func main(){
        //方式1
	loader := NewClientOptionLoader()

	// 注册 timeout 翻译器
	loader.RegisterTranslator("timeout", func(config map[string]interface{}) client.Option {
		timeout := config["timeout"].(int) // 假设 timeout 是一个整数
		return client.WithTimeout(timeout)
	});
       //方式2
       translators := map[string]Translator{
	"timeout": func(config map[string]interface{}) client.Option {
		timeout := config["timeout"].(int) // 假设 timeout 是一个整数
		return client.WithTimeout(timeout)
	},
	// 可以注册更多的选项翻译器...
     }

     loader := NewClientOptionLoader(translators)
      
}

The above is just a simple interface abstraction. Is there anything else you need to add?

[felix021]

@felix021 When implementing an options loader, we can follow these steps:

Define the core interface:

• 1. Define the core interface of the option loader, including registering the translator, setting the configuration source, loading options, etc.
• 2. Implement the option loader: Implement the option loader to load options according to the registered translator and configuration source.
• 3. Register configurations and translators: Allows users to register configurations and custom translators to map configurations to Kitex options.
• 4. Support different configuration sources: implement different configuration sources, such as file readers, etcd readers, etc.

The following is the sample code

package optionloader

import (
	"io/ioutil"
	"gopkg.in/yaml.v2"
	"github.com/kitex-contrib/kitex-client/client"
)

// 选项加载器的接口
type OptionLoader interface {
	RegisterTranslator(name string, translator Translator)
	SetSource(source ConfigSource)
	Load() ([]client.Option, error)
}

// 翻译器的接口
type Translator func(config map[string]interface{}) client.Option

// 配置源的接口
type ConfigSource interface {
	ReadConfig() (map[string]interface{}, error)
}

// 文件配置源的实现
type FileConfigSource struct {
	Filepath string
}

// 从文件中读取配置
func (f *FileConfigSource) ReadConfig() (map[string]interface{}, error) {
	data, err := ioutil.ReadFile(f.Filepath)
	if err != nil {
		return nil, err
	}

	var config map[string]interface{}
	err = yaml.Unmarshal(data, &config)
	if err != nil {
		return nil, err
	}

	return config, nil
}

//  OptionLoader 接口的实现
type OptionLoaderImpl struct {
	translators map[string]Translator
	source      ConfigSource
}

//  创建一个新的选项加载器
func NewOptionLoader() OptionLoader {
	return &OptionLoaderImpl{
		translators: make(map[string]Translator),
	}
}

//  注册翻译器
func (loader *OptionLoaderImpl) RegisterTranslator(name string, translator Translator) {
	loader.translators[name] = translator
}

//  设置配置源
func (loader *OptionLoaderImpl) SetSource(source ConfigSource) {
	loader.source = source
}

//  加载选项
func (loader *OptionLoaderImpl) Load() ([]client.Option, error) {
	config, err := loader.source.ReadConfig()
	if err != nil {
		return nil, err
	}

	var options []client.Option
	for key, value := range config {
		translator, ok := loader.translators[key]
		if !ok {
			continue
		}

		option := translator(value.(map[string]interface{}))
		options = append(options, option)
	}

	return options, nil
}

// 创建一个新的客户端选项加载器
func NewClientOptionLoader() ClientOptionLoader {
	return &OptionLoaderImpl{
		translators: make(map[string]Translator),
	}
}

func NewClientOptionLoader(translators map[string]Translator) OptionLoader {
	loader := NewOptionLoader()

	for name, translator := range translators {
		loader.RegisterTranslator(name, translator)
	}

	return loader
}

// 创建一个新的文件读取器
func NewFileReader(filepath string) ConfigSource {
	return &FileConfigSource{
		Filepath: filepath,
	}
}


func main(){
        //方式1
	loader := NewClientOptionLoader()

	// 注册 timeout 翻译器
	loader.RegisterTranslator("timeout", func(config map[string]interface{}) client.Option {
		timeout := config["timeout"].(int) // 假设 timeout 是一个整数
		return client.WithTimeout(timeout)
	});
       //方式2
       translators := map[string]Translator{
	"timeout": func(config map[string]interface{}) client.Option {
		timeout := config["timeout"].(int) // 假设 timeout 是一个整数
		return client.WithTimeout(timeout)
	},
	// 可以注册更多的选项翻译器...
     }

     loader := NewClientOptionLoader(translators)
      
}

The above is just a simple interface abstraction. Is there anything else you need to add?

Sorry for the late reply. This proposal is already claimed by @Printemps417 and I forgot to update the asignee.

[BaiZe1998]

I have been following this issue for some time. How is it going? May I join it?

[DMwangnima]

@BaiZe1998 The previous student's implementation has merit, but still needs more complete details. We can discuss it together.

[Printemps417]

I have been following this issue for some time. How is it going? May I join it?

Hi, we have completed most implementations of the optionloader for yml, etcd, and consul, which is being developed in this repository: https://github.com/Printemps417/optionloader. I'm glad we can exchange ideas and learn from each other.