NULLs sometimes rendering as "None" and sometimes as "∅ null"

[sgpeter1]

Hello,

I'm developing an adapter and noticed that sometimes NULLs are rendering as "None" and sometimes as "∅ null". I can't quite trace the cause, although I can write queries that change from "None" to "∅ null" by changing what rows are returned.

Any ideas what might be happening or places to look. I'm currently stumped. :)

Thanks!

[tconbeer]

If the Python representation of the data is None, then you get the dimmed ∅ null. For anything else, you’re probably displaying a string that says None or NaT, etc.

Depending on how your adapter works, there might be other intermediate representations of the data, like a PyArrow scalar, and there would be some other casting going on to get to and from that intermediate representation.

For reference, the code for the final formatter is here (and then the rich markup gets rendered):

https://github.com/tconbeer/textual-fastdatatable/blob/main/src/textual_fastdatatable/formatter.py#L17-L21

Before that formatting step, we're getting Python values from Arrow using .as_py(). It's possible this is a bug - id be interested in examples that show different behavior that you mention.

[sgpeter1]

Thanks for the response!

Well, shoot. I actually can't replicate this now because the underlying data changed and I didn't snapshot anything.

That said, I'm guessing something odd was happening here with type detection.

https://github.com/tconbeer/textual-fastdatatable/blob/main/src/textual_fastdatatable/backend.py#L247-L257

I suppose further investigations would have to look at PyArrow type detection strategies.

[sgpeter1]

FWIW I determined what was happening. There was a single value (0.97) in a column where all of the other values were 1. The response from the database looked like this:

[(None, 1), ('test', Decimal('0.97'))]

Because this causes a mixed type error when pushing into a PyArrow table, everything (all columns) get cast as strings. This ultimately causes that first-row, first-column None to show as the string None instead of ∅ null.

[tconbeer]

Got it. Thats an easy update to that casting one-liner to use None instead of str(None). Let's call that a bug

[tconbeer]

Created #658
Going to close this discussion. Thanks for raising this issue!