Error with cplex persistent : "ValueError: Cannot load a SolverResults object with bad status"

[svkawtikwar]

Summary

I am using the CPLEXPersistent solver interface for my problem. Here is what I do:

1. I build a pyomo concrete model and solve it
2. I update the model objective and do the solver.set_instance(model) to update the model
3. I solve it again.

When I solve it again, the cplex solver log clearly shows that cplex solved the model but then I get an error saying:
ValueError: Cannot load a SolverResults object with bad status

I am very sure that the CPLEX is able to solve the model to optimality (there is no infeasibility or unboundedness or any other error).
To troubleshoot, I replace the cplex_persistent solver with gurobi_persistent and the error goes away. So it's fair to think that the issue is with the cplex_persistent plugin and not with my model.

Also, one more important thing to mention: This error doesn't appear 100% of time, I did 10 runs of my code and it was successful 6 times.

Steps to reproduce the issue

from pyomo.opt import SolverFactory
solver = SolverFactory("cplex_persistent")
model: pyo.ConcreteModel = get_model(input_data=data)
solver.set_instance(model)
result = solver.solve(model)
model = update_model(model)
solver.set_instance(model)
result = solver.solve(model)

Since, this error doesn't happen all the time, I would recommend running it multiple times.

Error Message

ValueError: Cannot load a SolverResults object with bad status`

$ # Output message here, including entire stack trace, if available
line 98, in main
    result = solver.solve(model, report_timing=False)
  File "C:\Users\xxx\Downloads\.venv\lib\site-packages\pyomo\solvers\plugins\solvers\persistent_solver.py", line 567, in solve
    _model.solutions.load_from(
  File "C:\Users\xxx\Downloads\.venv\lib\site-packages\pyomo\core\base\PyomoModel.py", line 228, in load_from        
    raise ValueError(
ValueError: Cannot load a SolverResults object with bad status: error

Information on your system

Pyomo version: Pyomo 6.7.3 (CPython 3.8.0 on Windows 10)
Python version: Python 3.8.0
Operating system: Windows 10
How Pyomo was installed (PyPI, conda, source): pip
Solver (if applicable): cplex_persistent (cplex version 20.1.0.0)

Additional information

This error doesn't appear 100% of time, I did 10 runs of my code and it was successful 6 times. When I replace the cplex_persistent solver with gurobi_persistent the error goes away. So it's fair to think that the issue is with the cplex_persistent solve solution load APU and not with my model.

[mrmundt]

@svkawtikwar - Does it happen with all models, or a specific model?

[svkawtikwar]

@svkawtikwar - Does it happen with all models, or a specific model?
@mrmundt - Thanks for the response, it happens only to the model I have right now. It doesn't happen with toy models. But I can see from cplex logs that it is solved to optimality. Unfortunately I cannot share the model here.

I did another test, I am adding the details here:

What I tried?

result1 = solver.solve(model, report_timing=True, load_solutions=True)
model = myclass.update_model(model)
solver.set_instance(model)
result2 = solver.solve(
        model, report_timing=True, load_solutions=False, keepfiles=True
    )
pyo.assert_optimal_termination(result2)
logging.info(f"Optimal Objective value: {pyo.value(model.OBJ):.2f} ")

The code fails at assert_optimal_termination(result2), with the error:

Traceback (most recent call last):
  File "c:/Users/xxx/Downloads/xxx/runLocalOpt.py", line 29, in <module>
    main(
  File "c:\Users\xxx\Downloads\xxx\runMM.py", line 107, in main
    pyo.assert_optimal_termination(result2)
  File "C:\Users\xxx\Downloads\xxx\.venv\lib\site-packages\pyomo\opt\results\solver.py", line 178, in assert_opt_optimal_termination
    raise RuntimeError(msg)
RuntimeError: Solver failed to return an optimal solution. Solver status: error, Termination condition: error

However, the solver logs for the second solve command (generated from keepfiles=True) show that model was solved by cplex. I have attached the solver log.
cplex-solver.log

[svkawtikwar]

Turns out the exit code from CPLEX for my model is 5.
According to CPLEX, it means "Optimal solution is available, but with infeasibilities after unscaling" https://www.ibm.com/docs/en/icos/20.1.0?topic=api-cpx-stat-optimal-infeas.
I can see that the way postsolve works is by using different if conditions on status from CPLEX, as in the code present at pyomo->solvers->plugins->solvers->cplex_direct.py:L694 (here is the code snippet)

 if status in [1, 101, 102]:
            self.results.solver.status = SolverStatus.ok
            self.results.solver.termination_condition = TerminationCondition.optimal
            soln.status = SolutionStatus.optimal
        elif status in [2, 40, 118, 133, 134]:
            self.results.solver.status = SolverStatus.warning
            self.results.solver.termination_condition = TerminationCondition.unbounded
            soln.status = SolutionStatus.unbounded
        elif status in [4, 119, 134]:
            # Note: status of 4 means infeasible or unbounded
            #       and 119 means MIP infeasible or unbounded
            self.results.solver.status = SolverStatus.warning
            self.results.solver.termination_condition = (
                TerminationCondition.infeasibleOrUnbounded
            )
            soln.status = SolutionStatus.unsure
        elif status in [3, 103]:
            self.results.solver.status = SolverStatus.warning
            self.results.solver.termination_condition = TerminationCondition.infeasible
            soln.status = SolutionStatus.infeasible
        elif status in [10]:
            self.results.solver.status = SolverStatus.aborted
            self.results.solver.termination_condition = (
                TerminationCondition.maxIterations
            )
            soln.status = SolutionStatus.stoppedByLimit
        elif status in [11, 25, 107, 131]:
            self.results.solver.status = SolverStatus.aborted
            self.results.solver.termination_condition = (
                TerminationCondition.maxTimeLimit
            )
            soln.status = SolutionStatus.stoppedByLimit
        else:
            self.results.solver.status = SolverStatus.error
            self.results.solver.termination_condition = TerminationCondition.error
            soln.status = SolutionStatus.error

It has no check for error code 5 and there is no mechanism to output the solver status message from CPLEX. It will be helpful to add that mechanism to the CPLEXDIRECT class.

[mrmundt]

Hello, @svkawtikwar ! Thanks for the extra information - that is very helpful. In case you haven't been tracking it (it's not horribly transparent), we are in the process of reworking the solver interfaces (see #1030). We haven't started on CPLEX yet, but I will add to the backlog for CPLEX when we start its conversion to the new interface.