[Python] Interchange pa.Table's Column.null_count doesn't count NaNs

[honno]

Describe the bug, including details regarding any error messages, version, and platform.

It seems the interchange Column.null_count() (relevant spec) has erroneous behaviour

>>> import pyarrow as pa
>>> pa.__version__
'12.0.0.dev304'  # from https://pypi.fury.io/arrow-nightlies/
>>> df = pa.table([pa.array([float("nan")], type=pa.float64())], ["foo"])
>>> dfi = df.__dataframe__()
>>> col = dfi.get_column(0)
>>> col.null_count
0  # should be 1

I assume this is because Arrow does not treat NaNs as nulls, which semantically makes sense, but in the interchange protocol it should—see vaexio/vaex#2120 for a related discussion.

See pandas for expected behaviour

>>> import pandas as pa
>>> df = pd.DataFrame({"foo": [float("nan")]})
>>> dfi = df.__dataframe__()
>>> col = dfi.get_column(0)
>>> col.null_count
1

cc @AlenkaF (let me know if not to tag you on things! coincidentally I was working on data-apis/dataframe-interchange-tests#20 today when Ralf commented heh.)

Component(s)

Python

[AlenkaF]

cc @AlenkaF (let me know if not to tag you on things! coincidentally I was working on data-apis/dataframe-interchange-tests#20 today when Ralf commented heh.)

Do tag me please! =)
Very cool you both jumped on it so quickly hehe.

I assume this is because Arrow does not treat NaNs as nulls, which semantically makes sense, but in the interchange protocol it should—see vaexio/vaex#2120 for a related discussion.

But we agree with Maarten 😊 vaexio/vaex#2120 (comment)
There would be loss of information for all the libraries that actually differentiate between nan and null, right?

[AlenkaF]

Closing this issue as we have received confirmation from the protocol side that nan and null can be treated separately, see discussion in data-apis/dataframe-api#126