Model suggestion

[andri-jpg]

I have a suggestion for this project to be even better. we just use koboldcpp. because it can run Pygmalion 6b with only 4gb of ram, it can be accelerated with GPU. it took 12 seconds for the reply response. with this AIwaifu can be smarter because it uses the 6b model
link to koboldcpp https://github.com/LostRuins/koboldcpp

I also made a koboldcpp version of AIwaifu https://github.com/andri-jpg/AIwaifu-png

it's just that, I interact with the localhost using selenium, I haven't been able to integrate directly into localhost with python

for the demo, you can see here https://youtu.be/TzU27v9Hf6Q

[HRNPH]

I've thought of using C++ implementation of those model
but probably wouldn't do it soon since I'm pretty busy with work nowadays
I think a solid pipeline need to be laid for the Language Model Part first since I want us to be able to change for better model(if we have one) in the future

[andri-jpg]

I completely agree with your opinion. We do need to have a solid pipeline for the Language Model Part so that we can change it with a better model in the future. Thank you for providing your response.

[HRNPH]

Following from #44
I'll add up that we can actually quantize model to 4-bit ourself using GGML
which would be the method I prefer since we can change the model and then quantize it

Soon There'll be a released of 4bit Quantization directly from Huggingface itself so maybe we don't really need to do anything? lol