[Packaging] Use protobuf v3.8

[again4you]

This patch uses protobuf v3.8 instead of v3.6.1.

Signed-off-by: Sangjung Woo [email protected]

Known issue

After applying this patch, Unit testcase test_tensor_filter.framework_auto_ext_pt_01 of NNStreamer is failed.

• Test envrionment

  • Apply this patch to pytorch, build and install pytorch debian package
  • Disable tensorflow and tensorflow-lite and Enable pytorch and caffe2 when building NNStreamer since Tensorflow also has protobuf version problem.
  • Build NNStreamer and run unit tests

• Test result

...

[ RUN      ] test_tensor_filter.framework_auto_ext_pt_01
** Message: 08:23:25.262: gpu = 0, accl = default
[libprotobuf FATAL ../third_party/protobuf/src/google/protobuf/generated_message_reflection.cc:2321] CHECK failed: file != NULL: 
unknown file: Failure
C++ exception with description "CHECK failed: file != NULL: " thrown in the test body.
[  FAILED  ] test_tensor_filter.framework_auto_ext_pt_01 (24 ms)
[----------] 3 tests from test_tensor_filter (137 ms total)

[----------] Global test environment tear-down
[==========] 61 tests from 3 test cases ran. (156 ms total)
[  PASSED  ] 60 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] test_tensor_filter.framework_auto_ext_pt_01

[again4you]

Testcase framework_auto_ext_pb_03 was passed but the log looks like a failure when loading model file. Should we have to update the model file for protobuf v3.8?

[ RUN      ] test_tensor_filter.framework_auto_ext_pb_03

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2

** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: the file of init_model_path is not valid: ../tests/test_models/models/caffe2_init_net.pb"


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: Failed to load model


** (unittest_plugins:15326): CRITICAL **: 08:58:57.110: failed to initialize the object: Caffe2
[       OK ] test_tensor_filter.framework_auto_ext_pb_03 (12 ms)

[myungjoo]

As the description of this fork says this repo is deprecated.
Please submit to review.tizen.org and alter PPA to get sources from git.tizen.org

DEPRECATED. Please migrate launchpad git repo from github to tizen.org.

[again4you]

After talking with @helloahn , basic opreation of caffe2 filter works as below.

$ cd tests/nnstreamer_filter_caffe2/
$ ./runTest.sh
Independent Mode
==================================================
    Test Group (Unspecified) Starts.
[PASSED] 1:gst-launch of case 1
[PASSED] 1:Golden test comparison
==================================================
[PASSED] Test Group (Unspecified) Passed

@myungjoo
After verification, we are going to move tizen.org. Thanks.

[again4you]

After applying this PR(nnstreamer/nnstreamer#2399),
caffe2 test case(i.e. test_tensor_filter.framework_auto_ext_pb_03) is fixed.

However, pytorch test case (i.e. test_tensor_filter.framework_auto_ext_pt_01) is not fixed yet.

@kparichay ,
After upgrading Protobuf v3.8, your runTest.sh test script in nnstreamer/tests/nnstreamer_filter_pytorch does not work as below.

// Run only "Golden test comparison"

$ bash ./runTest.sh 
Independent Mode
==================================================
    Test Group (Unspecified) Starts.
** Message: 15:04:42.625: gpu = 0, accl = default
[libprotobuf FATAL ../third_party/protobuf/src/google/protobuf/generated_message_reflection.cc:2321] CHECK failed: file != NULL: 

terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: file != NULL: 
[FAILED][Critical] 1:gst-launch of case 1
Traceback (most recent call last):
  File "checkLabel.py", line 26, in <module>
    onehot = readbyte(sys.argv[1])
  File "checkLabel.py", line 21, in readbyte
    with open(filename, 'rb') as f:
IOError: [Errno 2] No such file or directory: 'tensorfilter.out.log'
[FAILED][Critical] 1:Golden test comparison
==================================================
[FAILED] Test Group (Unspecified) has failed cases (2)

All errors occurs in the same point of the Protobuf.
The error code from Protobuf v3.8 is as below.

// ../third_party/protobuf/src/google/protobuf/generated_message_reflection.cc:2321
void AssignDescriptorsImpl(const DescriptorTable* table) {
    ...
    // Fill the arrays with pointers to descriptors and reflection classes.
    const FileDescriptor* file =
        DescriptorPool::generated_pool()->FindFileByName(table->filename);
    GOOGLE_CHECK(file != NULL);

I am still fighting to solve this issue but I don't know why.
I would be happy when anyone give a hint to fix this issue.

[again4you]

The stable pytorch v1.5.0 also use protobuf v3.6.1.

pytorch$ cat version.txt 
1.5.0a0
pytorch$ cd third_party/protobuf/
pytorch/third_party/protobuf$ head -n 10 CHANGES.txt 
2018-07-27 version 3.6.1 (C++/Java/Python/PHP/Objective-C/C#/Ruby/JavaScript)

  C++
  * Introduced workaround for Windows issue with std::atomic and std::once_flag
    initialization (#4777, #4773).
...

[kparichay]

@again4you sorry for the late reply.
Below is my observation from a quick look. I will be able to spend more time tomorrow, so I will investigate and share.

In my understanding, BUILD_CUSTOM_PROTOBUF is ON by default, which builds custom protobuf present in this package (from packaging directory containing protobuf 3.6.1). Further, CAFFE2_LINK_LOCAL_PROTOBUF is also enabled by default. So, I dont understand why protobuf 3.8 installed in the system should affect pytorch unittest.

Further, as per cmakelists.txt from pytorch/torch, protobuf does not seem to be dependency (not sure though).

Also, pytorch latest versions might not work with protobuf 3.8 (pytorch#22437)