update docs + version (#85)

CHANGELOG.md

@@ -6,10 +6,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+### Fixed
+
+## [0.14.0]
+
 ### Added
 - Added a new documentation section "How it works" to explain the inner workings of the package. It's a work in progress, but it should give you a good idea of what's happening under the hood.
 - Improved template loading, so if you load your custom templates once with `load_templates!("my/template/folder)`, it will remember your folder for all future re-loads.
-- Added convenience function `create_template` to create templates on the fly without having to deal with `PT.UserMessage` etc. See `?create_template` for more information.
+- Added convenience function `create_template` to create templates on the fly without having to deal with `PT.UserMessage` etc. If you specify the keyword argument `load_as = "MyName"`, the template will be immediately loaded to the template registry. See `?create_template` for more information and examples.
 
 ### Fixed
 

Project.toml

@@ -1,7 +1,7 @@
 name = "PromptingTools"
 uuid = "670122d1-24a8-4d70-bfce-740807c42192"
 authors = ["J S @svilupp and contributors"]
-version = "0.14.0-DEV"
+version = "0.14.0"
 
 [deps]
 AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"

docs/src/frequently_asked_questions.md

@@ -287,4 +287,68 @@ give_me_number("How many car seats are in Porsche 911T?")
 
 We ultimately received our custom type `SmallInt` with the number of car seats in the Porsche 911T (I hope it's correct!).
 
-If you want to access the full conversation history (all the attempts and feedback), simply output the `response` object and explore `response.conversation`.
+If you want to access the full conversation history (all the attempts and feedback), simply output the `response` object and explore `response.conversation`.
+
+## How to quickly create a prompt template?
+
+Many times, you will want to create a prompt template that you can reuse with different inputs (eg, to create templates for AIHelpMe or LLMTextAnalysis). 
+
+Previously, you would have to create a vector of `SystemMessage` and `UserMessage` objects and then save it to a disk and reload. 
+Now, you can use the `create_template` function to do it for you. It's designed for quick prototyping, so it skips the serialization step and loads it directly into the template store (ie, you can use it like any other templates - try `aitemplates()` search).
+
+The syntax is simple: `create_template(;user=<user prompt>, system=<system prompt>, load_as=<template name>)` 
+
+When called it creates a vector of messages, which you can use directly in the `ai*` functions. If you provide `load_as`, it will load the template in the template store (under the `load_as` name).
+
+Let's generate a quick template for a simple conversation (only one placeholder: name)
+```julia
+# first system message, then user message (or use kwargs)
+tpl=PT.create_template("You must speak like a pirate", "Say hi to {{name}}"; load_as="GreatingPirate")
+
+## 2-element Vector{PromptingTools.AbstractChatMessage}:
+## PromptingTools.SystemMessage("You must speak like a pirate")
+##  PromptingTools.UserMessage("Say hi to {{name}}")
+```
+
+You can immediately use this template in `ai*` functions:
+```julia
+aigenerate(tpl; name="Jack Sparrow")
+# Output: AIMessage("Arr, me hearty! Best be sending me regards to Captain Jack Sparrow on the salty seas! May his compass always point true to the nearest treasure trove. Yarrr!")
+```
+
+Since we provided `load_as`, it's also registered in the template store:
+```julia
+aitemplates("pirate")
+
+## 1-element Vector{AITemplateMetadata}:
+## PromptingTools.AITemplateMetadata
+##   name: Symbol GreatingPirate
+##   description: String ""
+##   version: String "1.0"
+##   wordcount: Int64 46
+##   variables: Array{Symbol}((1,))
+##   system_preview: String "You must speak like a pirate"
+##   user_preview: String "Say hi to {{name}}"
+##   source: String ""
+```
+
+So you can use it like any other template:
+```julia
+aigenerate(:GreatingPirate; name="Jack Sparrow")
+# Output: AIMessage("Arr, me hearty! Best be sending me regards to Captain Jack Sparrow on the salty seas! May his compass always point true to the nearest treasure trove. Yarrr!")
+```
+
+If you want to save it in your project folder:
+```julia
+PT.save_template("templates/GreatingPirate.json", tpl; version="1.0") # optionally, add description
+```
+It will be saved and accessed under its basename, ie, `GreatingPirate` (same as `load_as` keyword argument).
+
+Note: If you make any changes to the templates on the disk/in a folder, you need to explicitly reload all templates again!
+
+If you are using the main PromptingTools templates, you can simply call `PT.load_templates!()`.
+If you have a project folder with your templates, you want to add it first:
+```julia
+PT.load_templates!("templates") 
+```
+After the first run, we will remember the folder and you can simply call `PT.load_templates!()` to reload all the templates in the future!

docs/src/how_it_works.md

@@ -76,6 +76,8 @@ When you provide a Symbol (eg, `:AssistantAsk`) to ai* functions, thanks to the
 
 You can discover all available templates with `aitemplates("some keyword")` or just see the details of some template `aitemplates(:AssistantAsk)`.
 
+Note: There is a new way to create and register templates in one go with `create_template(;user=<user prompt>, system=<system prompt>, load_as=<template name>)` (it skips the serialization step where a template previously must have been saved somewhere on the disk). See FAQ for more details or directly `?create_template`.
+
 ## ai* Functions
 
 The above steps are implemented in the `ai*` functions, eg, `aigenerate`, `aiembed`, `aiextract`, etc. They all have the same basic structure: 

src/templates.jl

@@ -119,7 +119,56 @@ function render(schema::Nothing, template::AITemplate; kwargs...)
 end
 
 ## Loading/saving -- see src/serialization.jl
+"""
+    build_template_metadata(
+        template::AbstractVector{<:AbstractMessage}, template_name::Symbol,
+        metadata_msgs::AbstractVector{<:MetadataMessage} = MetadataMessage[])
+
+Builds `AITemplateMetadata` for a given template based on the messages in `template` and other information.
 
+`AITemplateMetadata` is a helper struct for easy searching and reviewing of templates via `aitemplates()`.
+"""
+function build_template_metadata(
+        template::AbstractVector{<:AbstractMessage}, template_name::Symbol,
+        metadata_msgs::AbstractVector{<:MetadataMessage} = MetadataMessage[])
+
+    # prepare the metadata
+    wordcount = 0
+    system_preview = ""
+    user_preview = ""
+    variables = Symbol[]
+    for i in eachindex(template)
+        msg = template[i]
+        wordcount += length(msg.content)
+        if hasproperty(msg, :variables)
+            append!(variables, msg.variables)
+        end
+        # truncate previews to 100 characters
+        if msg isa SystemMessage && length(system_preview) < 100
+            system_preview *= first(msg.content, 100)
+        elseif msg isa UserMessage && length(user_preview) < 100
+            user_preview *= first(msg.content, 100)
+        end
+    end
+    if !isempty(metadata_msgs)
+        # use the first metadata message found if available
+        meta = first(metadata_msgs)
+        metadata = AITemplateMetadata(; name = template_name,
+            meta.description, meta.version, meta.source,
+            wordcount,
+            system_preview = first(system_preview, 100),
+            user_preview = first(user_preview, 100),
+            variables = unique(variables))
+    else
+        metadata = AITemplateMetadata(; name = template_name,
+            wordcount,
+            system_preview = first(system_preview, 100),
+            user_preview = first(user_preview, 100),
+            variables = unique(variables))
+    end
+
+    return metadata
+end
 """
         remove_templates!()
 
@@ -192,41 +241,9 @@ function load_templates!(dir_templates::Union{String, Nothing} = nothing;
                     end
                     store[template_name] = template
 
-                    # prepare the metadata
-                    wordcount = 0
-                    system_preview = ""
-                    user_preview = ""
-                    variables = Symbol[]
-                    for i in eachindex(template)
-                        msg = template[i]
-                        wordcount += length(msg.content)
-                        if hasproperty(msg, :variables)
-                            append!(variables, msg.variables)
-                        end
-                        # truncate previews to 100 characters
-                        if msg isa SystemMessage && length(system_preview) < 100
-                            system_preview *= first(msg.content, 100)
-                        elseif msg isa UserMessage && length(user_preview) < 100
-                            user_preview *= first(msg.content, 100)
-                        end
-                    end
-                    if !isempty(metadata_msgs)
-                        # use the first metadata message found if available
-                        meta = first(metadata_msgs)
-                        metadata = AITemplateMetadata(; name = template_name,
-                            meta.description, meta.version, meta.source,
-                            wordcount,
-                            system_preview = first(system_preview, 100),
-                            user_preview = first(user_preview, 100),
-                            variables = unique(variables))
-                    else
-                        metadata = AITemplateMetadata(; name = template_name,
-                            wordcount,
-                            system_preview = first(system_preview, 100),
-                            user_preview = first(user_preview, 100),
-                            variables = unique(variables))
-                    end
                     # add metadata to store
+                    metadata = build_template_metadata(
+                        template, template_name, metadata_msgs)
                     push!(metadata_store, metadata)
                 end
             end
@@ -352,19 +369,23 @@ end
 
 ## Utility for creating templates
 """
-    create_template(; user::AbstractString, system::AbstractString="Act as a helpful AI assistant.")
+    create_template(; user::AbstractString, system::AbstractString="Act as a helpful AI assistant.", 
+        load_as::Union{Nothing, Symbol, AbstractString} = nothing)
 
-    create_template(system::AbstractString, user::AbstractString)
+    create_template(system::AbstractString, user::AbstractString, 
+        load_as::Union{Nothing, Symbol, AbstractString} = nothing)
 
 Creates a simple template with a user and system message. Convenience function to prevent writing `[PT.UserMessage(...), ...]`
 
 # Arguments
 - `system::AbstractString`: The system message. Usually defines the personality, style, instructions, output format, etc.
 - `user::AbstractString`: The user message. Usually defines the input, query, request, etc.
+- `load_as::Union{Nothing, Symbol, AbstractString}`: If provided, loads the template into the `TEMPLATE_STORE` under the provided name `load_as`. If `nothing`, does not load the template.
 
 Use double handlebar placeholders (eg, `{{name}}`) to define variables that can be replaced by the `kwargs` during the AI call (see example).
 
 Returns a vector of `SystemMessage` and UserMessage objects.
+If `load_as` is provided, it registers the template in the `TEMPLATE_STORE` and `TEMPLATE_METADATA` as well.
 
 # Examples
 
@@ -384,14 +405,16 @@ aigenerate(tpl; name="Jack Sparrow")
 # Output: AIMessage("Arr, me hearty! Best be sending me regards to Captain Jack Sparrow on the salty seas! May his compass always point true to the nearest treasure trove. Yarrr!")
 ```
 
+If you're interested in saving the template in the template registry, jump to the end of these examples!
+
 If you want to save it in your project folder:
 ```julia
 PT.save_template("templates/GreatingPirate.json", tpl; version="1.0") # optionally, add description
 ```
 It will be saved and accessed under its basename, ie, `GreatingPirate`.
 
 Now you can load it like all the other templates (provide the template directory):
-```
+```julia
 PT.load_templates!("templates") # it will remember the folder after the first run
 # Note: If you save it again, overwrite it, etc., you need to explicitly reload all templates again!
 ```
@@ -416,15 +439,45 @@ Now you can use it like any other template (notice it's a symbol, so `:GreatingP
 ```julia
 aigenerate(:GreatingPirate; name="Jack Sparrow")
 # Output: AIMessage("Arr, me hearty! Best be sending me regards to Captain Jack Sparrow on the salty seas! May his compass always point true to the nearest treasure trove. Yarrr!")
+```
+
+If you do not need to save this template as a file, but you want to make it accessible in the template store for all `ai*` functions, you can use the `load_as` (= template name) keyword argument:
+```julia
+# this will not only create the template, but also register it for immediate use
+tpl=PT.create_template("You must speak like a pirate", "Say hi to {{name}}"; load_as="GreatingPirate")
+
+# you can now use it like any other template
+aiextract(:GreatingPirate; name="Jack Sparrow")
 ````
 """
 function create_template(
         system::AbstractString,
-        user::AbstractString)
-    return [SystemMessage(system), UserMessage(user)]
+        user::AbstractString; load_as::Union{Nothing, Symbol, AbstractString} = nothing)
+    ##
+    global TEMPLATE_STORE, TEMPLATE_METADATA
+    ##
+    template = [SystemMessage(system), UserMessage(user)]
+    ## Should it be loaded as well?
+    if !isnothing(load_as)
+        template_name = Symbol(load_as)
+        ## add to store
+        if haskey(TEMPLATE_STORE, template_name)
+            @warn("Template $(template_name) already exists, overwriting!")
+            ## remove from metadata to avoid duplicates
+            filter!(x -> x.name != template_name, TEMPLATE_METADATA)
+        end
+        TEMPLATE_STORE[template_name] = template
+        ## prepare the metadata
+        metadata = build_template_metadata(
+            template, template_name)
+        push!(TEMPLATE_METADATA, metadata)
+    end
+
+    return template
 end
 # Kwarg version
 function create_template(;
-        user::AbstractString, system::AbstractString = "Act as a helpful AI assistant.")
-    create_template(system, user)
+        user::AbstractString, system::AbstractString = "Act as a helpful AI assistant.",
+        load_as::Union{Nothing, Symbol, AbstractString} = nothing)
+    create_template(system, user; load_as)
 end

test/templates.jl

@@ -66,6 +66,15 @@ end
     @test tpl[2].content == "Say hi to {{chef}}"
     @test tpl[2].variables == [:chef]
     @test tpl[2] isa UserMessage
+
+    # use save_as
+    tpl = create_template(
+        "You must speak like a pirate", "Say hi to {{name}}"; load_as = :PirateGreetingX)
+    @test haskey(PT.TEMPLATE_STORE, :PirateGreetingX)
+    @test length(filter(x -> x.name == :PirateGreetingX, PT.TEMPLATE_METADATA)) == 1
+    ## clean up
+    delete!(PT.TEMPLATE_STORE, :PirateGreetingX)
+    filter!(x -> x.name != :PirateGreetingX, PT.TEMPLATE_METADATA)
 end
 
 @testset "Templates - Echo aigenerate call" begin