Text looping when using inference and go full determinism

[Kimiko-AI]

While using text inference for testing my LoRA, when regenerate with changed temperature, Top P and stuff, the output is still the same as before. Tested on unhelpful-ai.

[zetavg]

Hi, thanks for noticing this. It's been bothering me for a while since I saw that the temperature, top_p, and top_k didn't affect the generated text at all. And it seems that not just here, but also on the official alpaca-lora demo the three params have no effect. I'd haven't found an answer yet.

But I just got a new clue that those three works on the demo of databricks-dolly, while it does not work when databricks/dolly-v2-7b is selected as the base model here. Maybe I can figure it out soon...

And to avoid looping output, you can try to increase the Repetition Penalty and/or Beams. Hope it helps!

[zetavg]

Ok I found out why temperature, top_p, and top_k are not working huggingface/transformers#22405 (comment).

the popular temperature, top_p, and top_k are only active when do_sample=True is also passed

[Kimiko-AI]

Not related to this but is adding an option for text only dataset possible? Sorry for bother you but I am not good at Python enough to implemented it.

[zetavg]

Do you mean supporting the "Plain Text" format with datasets loaded from files? If so, I'm also thinking about this as I want to fine-tune some models specialized in writing code, and it'll be more convenient to write code samples in a plain text format instead of dealing with all the \ns in JSON.

BTW don't worry about the familiarity with Python, I'm also not good at it, and many pieces of code in this repo are co-authored by ChatGPT lol.

[zetavg]

Closing this since the issue has been resolved in the main branch (e1c08f4ded). If you're interested in further discussions, let's move on at Discussions!