export ids during tokenisation

[cmacdonald]

optionally, export ids during tokenisation

[okhat]

Thanks Craig!

I'm not sure of the purpose of this:

if with_ids:
    #the masking code assumes that args.mask_punctuation is false.
    assert len(self.colbert.skiplist) == 0

Seems related to what we discussed by email. But we're moving in the direction of having masking punctuation as a default and discouraging keeping them. This may conflict with that. Thoughts?

[cmacdonald]

Good question, I'll look into this tomorrow.

[okhat]

By the way, and this is a bit off topic for this PR, can we remove the setup.py file? Are there still use-cases where pip is essential?

[cmacdonald]

Sorry for latency - other pressures.

I'm keen to keep setup.py. See for example: https://colab.research.google.com/github/cmacdonald/pyterrier_colbert/blob/main/vaswani.ipynb (or equivalently https://github.com/terrierteam/pyterrier_colbert/blob/main/vaswani.ipynb)

Our pyt_colbert wrapper just expresses a dependency on (our current fork) of Colbert.