proxy: added multithreaded worker support for proxy plugins

[gautampunhani]

This applies the same code change brought in for making output plugins performant in v1.7.0 to the proxy plugins of Go.
Without this change if the go output plugin gets slow/unresponsive, other input plugins and output plugins also were getting stuck because of single-threaded nature of invocation.

This addresses issue fluent-bit-go#45

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change
• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found
  Please note that, Valgrind with golang plugins show leaks without this change as well

Documentation

• [N/A] Documentation required for this feature (Documentation will be same as any output plugin with workers)

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nokute78]

Could you fix warning ?

[nokute78]

I confirmed below delay plugin.
It waits 1 minute before flushing.

diff --git a/examples/out_gstdout/out_gstdout.go b/examples/out_gstdout/out_gstdout.go
index 8a0acd9..9c99a74 100644
--- a/examples/out_gstdout/out_gstdout.go
+++ b/examples/out_gstdout/out_gstdout.go
@@ -31,6 +31,8 @@ func FLBPluginFlush(data unsafe.Pointer, length C.int, tag *C.char) int {
        var ts interface{}
        var record map[interface{}]interface{}
 
+       time.Sleep(time.Second * 60)
+
        // Create Fluent Bit decoder
        dec := output.NewDecoder(data, int(length))

a.conf

[SERVICE]
    Plugins_File ./b.conf

[INPUT]
    Name dummy

[OUTPUT]
    Name stdout
    Match *
#    Workers 2

[OUTPUT]
    Name gstdout
    Match *
    Workers 3

b.conf:

[PLUGINS]
    Path /path/to/out_gstdout.so

Native out_stdout flushes without preventing by go plugin.

 bin/fluent-bit -c a.conf 
Fluent Bit v1.9.0
* Copyright (C) 2019-2021 The Fluent Bit Authors
* Copyright (C) 2015-2018 Treasure Data
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[flb-go] plugin parameter = ''
[2021/07/29 17:45:16] [ info] [engine] started (pid=5821)
[2021/07/29 17:45:16] [ info] [storage] version=1.1.1, initializing...
[2021/07/29 17:45:16] [ info] [storage] in-memory
[2021/07/29 17:45:16] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2021/07/29 17:45:16] [ info] [cmetrics] version=0.1.6
[2021/07/29 17:45:16] [ info] [sp] stream processor started
[2021/07/29 17:45:16] [ info] [output:gstdout:gstdout.1] worker #2 started
[2021/07/29 17:45:16] [ info] [output:gstdout:gstdout.1] worker #1 started
[2021/07/29 17:45:16] [ info] [output:gstdout:gstdout.1] worker #0 started
[0] dummy.0: [1627548317.209731386, {"message"=>"dummy"}]
[1] dummy.0: [1627548318.209646778, {"message"=>"dummy"}]
[2] dummy.0: [1627548319.210304577, {"message"=>"dummy"}]
[3] dummy.0: [1627548320.209884035, {"message"=>"dummy"}]
[0] dummy.0: [1627548321.210551449, {"message"=>"dummy"}]
[1] dummy.0: [1627548322.209664054, {"message"=>"dummy"}]
[2] dummy.0: [1627548323.209793680, {"message"=>"dummy"}]
[3] dummy.0: [1627548324.209815591, {"message"=>"dummy"}]
[4] dummy.0: [1627548325.210364678, {"message"=>"dummy"}]
[0] dummy.0: [1627548326.210564617, {"message"=>"dummy"}]
[1] dummy.0: [1627548327.209859292, {"message"=>"dummy"}]
[2] dummy.0: [1627548328.210035259, {"message"=>"dummy"}]
[3] dummy.0: [1627548329.209700478, {"message"=>"dummy"}]
[4] dummy.0: [1627548330.209783094, {"message"=>"dummy"}]
^C[2021/07/29 17:45:31] [engine] caught signal (SIGINT)

[nokute78]

There are 3 golang plugin threads. 👍
flb-out-gstdout

$ grep Name /proc/`pidof fluent-bit`/task/*/status
/proc/5862/task/5862/status:Name:	fluent-bit
/proc/5862/task/5863/status:Name:	fluent-bit
/proc/5862/task/5864/status:Name:	fluent-bit
/proc/5862/task/5865/status:Name:	fluent-bit
/proc/5862/task/5866/status:Name:	fluent-bit
/proc/5862/task/5867/status:Name:	fluent-bit
/proc/5862/task/5868/status:Name:	flb-pipeline
/proc/5862/task/5869/status:Name:	flb-logger
/proc/5862/task/5870/status:Name:	flb-out-gstdout
/proc/5862/task/5871/status:Name:	flb-out-gstdout
/proc/5862/task/5872/status:Name:	flb-out-gstdout
$ 

[nokute78]

@gautampunhani Thank you.

Could you modify commit message ?
proxy: added multithreaded worker support for proxy plugins
moving function to the right place to fix warning of implicit declaration

In this case, you updated flb_output.c.
e.g.
output: added multithreaded worker support for proxy plugins
output: moving function to the right place to fix warning of implicit declaration
https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#commit-changes

Warning is removed.👍

[nokute78]

@gautampunhani Please modify commit message.

Could you modify commit message ?
proxy: added multithreaded worker support for proxy plugins
moving function to the right place to fix warning of implicit declaration

[gautampunhani]

Hi @nokute78 , I was trying to reword the commit messages but it was showing the files that I have not committed to the diff as well. I am closing this PR because of this goof-up. Have raised another PR ( #3908 ) with the fixed comments.

[gautampunhani]

Closing this PR, have created a new one : #3908