Enable get_token_stream to include LineEnd tokens with optional parameter.

[SurajAralihalli]

Description

This PR adds parameter LineEndTokenOption to the get_token_stream and process_token_stream functions, enabling LineEnd tokens in the output. Also retained original declaration of get_token_stream to maintain backward compatibility.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[karthikeyann]

This adds another step where we need to remove these LineEnd before tree algorithms.

Do we need LineEnd tokens? if this is for finding the row number of tokens, it's possible to calculate using StructBegin, StructEnd.

[shrshi]

One high-level comment -

[karthikeyann]

I and @shrshi discussed about a profile of @revans2 's prototype https://github.com/revans2/spark-rapids-jni/pull/new/get_json_obj_experiment.CUDF

A few outcomes of our meeting:

• 5 simulateDFA calls in total.
• A few smaller FST take longer time than get_stack_context FST. will investigate further.
• Consider fusing recover_from_error with PDA (json_to_tokens_fst FST).
• It is possible to eliminate the process_token_stream function itself, if the post-processing code of tokens after get_token_stream, can handle Error tokens. That new code is going to process tokens row-wise, so it will be easier to nullify the entire row.
  • @SurajAralihalli can the new post processing code of tokens replace Error tokens with null row?
  • @shrshi suggested, FST may not be required to implement the process_token_stream anyway, since it's a stream compaction always. (Related PR [FEA] Adds option to recover from invalid JSON lines in JSON tokenizer  #13344)

[vyasr]

@SurajAralihalli @shrshi @karthikeyann is this PR something we still eventually want to get in once it's suitably updated, or is it a prototype that can be closed now?

[SurajAralihalli]

I think this PR can be dropped.