Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed multiprocessing issue during RoBERTa prediction - Solution to #1558 #1559

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

etqadkhan
Copy link

With reference to #1558
When using a RoBERTa model for doing prediction, when the model is loaded using ClassificationModel() it prompts a warning stating,

UserWarning: use_multiprocessing automatically disabled as xlmroberta fails when using multiprocessing for feature conversion.

When prediction is performed on text data, if the number of records are a little more than a handful, the prediction progress bar takes infinite execution.

The issue occurs because in the classification_model.py file under the ClassificationModel class, there are two arguments that concern with multiprocessing, them being, args.use_multiprocessing and args.use_multiprocessing_for_evaluation. While one is set to False by default, the other remains to be True,as can be seen in the screenshot attached:

image




To be able to perform prediction by successfully disabling multiprocessing, we need to disable args.use_multiprocessing_for_evaluation = False and it should work fine. The approach has been tested locally and has proven to be working. Screenshot attached:

working

@wesngoh
Copy link

wesngoh commented Oct 26, 2024

Faced a similar issue and am glad to have found your workaround, im surprised this have not been pushed after a year.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants