Added model_max_length as a tok parameter

[aittalam]

Since 4.40.0, transformers does not get model_max_tokens from the model's family dict but explicitly looks for this parameter in tokenizer_config.json. Not all models provide this, so one possibility is to explicitly allow users to provide this parameter in the config. This PR implements exactly this.

Tested with facebook/bart-large-cnn + pytest unit/integration tests.

[binaryaaron]

approving given the internal constraints; double check a few things before landing though. I'd suggest adding an extra test to the unit tests to check for it.

[aittalam]

approving given the internal constraints; double check a few things before landing though. I'd suggest adding an extra test to the unit tests to check for it.

Thanks! I have two more PRs in queue:

• bump of all libs (this will allow us, anmong other things, to run more recent models)
• adding text generation pipeline + prompts

Will make unit tests for both and add one for this change too (explicitly passing a longer text as input and verifying it does not break with the new setup)