Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "./onnxruntime_test_all --help" segfault #22839

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

amancini-N
Copy link
Contributor

Description

Makes sure the GTest parameters are processed before ORT Env is set up in unittests binary. In this way, binary won't segfault on exit if --help arg is used

Motivation and Context

Fixes #22838

::testing::InitGoogleTest(&argc, argv);
ortenv_setup();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very tricky. The ::testing::InitGoogleTest function also calls a lot of initialization functions in the tests, and the these functions may call into ONNX Runtime. Therefore ortenv should be initialized before that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I pushed an alternative restoring the original order, but making sure ortenv teardown is always done. This one also seems working with "--help" case

@@ -93,6 +93,8 @@ int TEST_MAIN(int argc, char** argv) {

ORT_TRY {
ortenv_setup();
// TODO: Fix the C API issue
atexit(ortenv_teardown); // If we don't do this, it will crash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Atexit functions get executed after the main function is returned, especially , it is after the "::google::protobuf::ShutdownProtobufLibrary" , which means in the teardown function we cannot call all protobuf functions, which is not possible. Also, it will also change the destruction order between function local statics and this "ortenv_teardown" function. Because basically the function local statics will use atexit function to register their destructors. Therefore, here the ordering is really very tricky. I would suggest keeping it unchanged unless there is an urgent need.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would using gsl::finally instead of atexit work?

e.g., #include <gsl/gsl>, then,

Suggested change
atexit(ortenv_teardown); // If we don't do this, it will crash
auto clean_up_ortenv = gsl::finally(ortenv_teardown); // If we don't do this, it will crash

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edgchen1 tried with gsl::finally, no luck. The problem is abseil parsing is just calling for exit() if --help flag is used, so only atexit has effect in this way.
@snnn I see your point, I agree this ordering is indeed tricky, and unfortunately abseil and gtest don't leave enough control on possible hooks on help exit.

I've pushed one last tentative. This time, ortenv_setup is called only if --help is not specified. This should not alter the ordering for test execution and --help is not crashing, as ortenv_setup is not called in that case

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Dev] "./onnxruntime_test_all --help" gives segmentation fault
3 participants