BUG (string dtype): comparison of string column to mixed object column fails #60228 (fixed) #60392
Conversation
| if (is_string_dtype(lvalues) and is_object_dtype(rvalues)) or ( | ||
| is_object_dtype(lvalues) and is_string_dtype(rvalues) |
There was a problem hiding this comment.
Checking for string dtype for an array can be expensive in case the array is object dtype (at that point it will scan all values to check if they are strings). So we might want to try avoid that at this level.
I think we could handle the issue specifically for the ArrowExtensionArray itself (see the code I referenced in #60228 (comment))
| ): | ||
| if lvalues.dtype.name == "string" and rvalues.dtype == object: | ||
| lvalues = lvalues.astype("string") | ||
| rvalues = pd_array(rvalues, dtype="string") |
There was a problem hiding this comment.
We might need to do the casting the other way around. Instead of casting the object to string and then compare both as strings, I think we have to cast the string to object and compare both as object dtype.
The reason for this is that casting to string might actually convert values to strings, and then we are no longer doing the comparison for the original values.
>>> ser_string = pd.Series(["1", "b"])
>>> ser_mixed = pd.Series([1, "b"])
>>> ser_string == ser_mixed
0 False
1 True
dtype: bool
>>> ser_string == ser_mixed.astype("string")
0 True
1 True
dtype: boolSo if we would do that casting under the hood, the result would change in this case.
There was a problem hiding this comment.
And we should add this case to the tests!
|
|
||
| def test_comparison_string_mixed_object(): | ||
| # Issue https://github.com/pandas-dev/pandas/issues/60228 | ||
| pd.options.future.infer_string = True |
There was a problem hiding this comment.
You don't need to add this for CI, because we have a separate CI build that enables this option for the full test suite.
Now, this can still be useful to test locally, but the way you can do this is with setting an environment variable (on linux I can do PANDAS_FUTURE_INFER_STRING=1 pytest ... to run the test with the option enabled.
Changes :
comparison_opfunction inpandas/core/ops/array_ops.py:stringandobjectdtypes.stringandobjecttypes are casted to a common type (string) before comparison.stringvsobjectwith homogeneous and mixed types.pd.options.future.infer_string = True).--
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.