Misc ML related. The power of prompt engineering

[ndrean]

I was wrong about prompt engineering. Ignorance probably....this made me change my mind

You type a desired ffmpeg kind like command in a text input, and the best real approaching command - using ffmpeg - will be run against your file input.

Stunning...

[LuchoTurtle]

Would this be considered prompt engineering? Isn't it just extracting semantics from a given input and creating ffmpeg commands and applying them onto media files? 👀

[ndrean]

When I read the prompt, certainly. I assume the author spent some time to determine how he should formulate the demand for OpenAI to respond correctly. So "just" is maybe a bit severe, isn't it?

https://github.com/acalejos/CinEx/blob/main/cinex.livemd#ffmpeg-instructions

[LuchoTurtle]

Oh, of course! I didn't mean "just" in an derogatory way. It's an impressive tool that is a god-sent since ffmpeg can be daunting at first. It's really interesting seeing how the system prompt is produced 🤔 great share!

For me (at least), prompt engineering is when the end-user (me, for example) tries to formulate prompts to get better results. It's effectively what it was done in the system prompt but that's part of an agent, which is something a bit different, in my opinion :D

[ndrean]

Can you precise what is an agent?

[ndrean]

I found an easy, maybe naive, example to illustrate what correct prompting is about. If you do - or don't - show to the LLM how to "reason", at least what you expect him to do, say by giving a example, then it does - or not - work.

Besides this, he also use Instructor, the Elixir version of the Python Instructor. If I understand correctly, the idea is to parse the LLM's response into a schema for validity. Valid means that the response (a FFMPEG command) is a valid one when it is executable. He tests it with :exec.run(...), cf the dependency erlexec).

What I don't get yet is how this is used then the changeset is invalid. Is there a loop back? Some kind of response back to the LLM saying "hey, this is not valid, please RTF prompt and answer correctly next time 😀. Try again!", until the response "seems" valid, as per the schema at least (but this is a huge advancement).