search-doc-1702194123042.json

{"searchDocs":[{"title":"Data Utils","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/data-utils","content":"","keywords":"","version":"Next"},{"title":"DataUtils​","type":1,"pageTitle":"Data Utils","url":"/YiValApi/docs/API-Reference/data-utils#datautils","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Data Utils","url":"/YiValApi/docs/API-Reference/data-utils#introduction","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Data Utils","url":"/YiValApi/docs/API-Reference/data-utils#class-definition","content":" Description​  Methods (Functions)​  evaluate_condition(condition: str, evaluator_output: EvaluatorOutput) -> bool: Uses the _tokenize_condition and _evaluate_tokenized_condition functions to evaluate a condition against an evaluator output. read_code_from_path_or_module(path_or_module: str) -> Optional[str]: Reads the source code from either an absolute file path or from a module and its function. Returns the source code if found, otherwise None. transform_experiment_result_generic(code: str, exp_result: ExperimentResult) -> dict: Transforms an experiment result into a generic format, where the result is a dictionary containing the Input and Output fields.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Data Utils","url":"/YiValApi/docs/API-Reference/data-utils#example","content":"  # Read the code from the source code = read_code_from_path_or_module("yival.demo.headline_generation") import pickle from yival.schemas.experiment_config import Experiment condition = "name == openai_prompt_based_evaluator AND result >= 3 AND display_name == clarity " with open('test_demo_results.pkl', 'rb') as f: result: Experiment = pickle.load(f) for combo_result in result.combination_aggregated_metrics: results: List[ExperimentResult] = combo_result.experiment_results for result in results: for eo in result.evaluator_outputs: condition_met = evaluate_condition(condition, eo) if condition_met: # Extract the result pair given code. result_pair = transform_experiment_result_generic( code, result ) print(result_pair)  ","version":"Next","tagName":"h3"},{"title":"Data Generation","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/data-generation","content":"","keywords":"","version":"Next"},{"title":"BaseDataGenerator​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#basedatagenerator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#introduction","content":" This class provides a blueprint for data generators and offers methods to register new generators, retrieve registered generators, and fetch their configurations.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#example","content":" ","version":"Next","tagName":"h3"},{"title":"OpenAIPromptDataGenerator​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#openaipromptdatagenerator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#introduction-1","content":" This module provides a mechanism for data generation using OpenAI's models. It aims to generate data examples programmatically based on a given prompt and configuration. The generated examples can be used for various purposes including testing, simulations, and more. The module offers utility functions to process and transform the generated data, ensuring it's structured and usable for subsequent operations.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#class-definition-1","content":" Description​  This configuration object is specifically tailored for the OpenAIPromptDataGenerator class. It provides settings to guide the generation of test cases using OpenAI models based on a given prompt.  Attributes​  model_name(str): Specifies the OpenAI model to be used for generating the test cases.The default value is "gpt-4". prompt (Union[str, List[Dict[str, str]]]): The instruction or set of instructions given to the OpenAI model for generating test cases. It can be a single string or a list of messages.The default value is an empty string or "". input_function (Dict[str, Any]): Details of the function for which test cases are to be generated. Contains information like function name, description, and parameters.The default value is an empty dictionary or {}. diversify(bool): A flag that determines if the generated test cases should be diversified to ensure comprehensive evaluation.The default value is set to True. max_token(int): Specifies the maximum number of tokens allowed for the prompt.The default value is set to 2000. expected_param_name(str): Indicates the name of the parameter that contains expected values in the generated test cases. call_option(Optional[CallOption]): Provides additional options for calling the OpenAI model.The default value is None. output_csv_path(Optional[str]): The path to save the generated test cases in a CSV format.The default value is None.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#example-1","content":" Generate Data and Save Test Case​  # Create a configuration for data generation generator_config = OpenAIPromptBasedGeneratorConfig( prompt="Please provide test cases for...", input_function={ "name": "example_function", "description": "Describe the function...", "parameters": { "param1": "str", "param2": "int" } }, diversify=True, max_token=1500, expected_param_name="expected_output", output_csv_path="path/to/save/test_cases.csv" ) # Initialize the data generator with the configuration data_generator = OpenAIPromptDataGenerator(generator_config) # Generate and save test cases data_generator.generate_examples()   Use OpenAI Prompt Data Generator in the YiVal config​  dataset: data_generators: openai_prompt_data_generator: chunk_size: 2 diversify: true prompt: "Please provide a concrete and realistic test case as a dictionary for function invocation using the ** operator. Only include parameters, excluding description and name. Ensure it's succinct and well-structured. **Only provide the dictionary.**" input_function: description: Given the species of an animal and its character, generate a corresponding story name: animal_story_generation parameters: species: str character: str drawing_style: str number_of_examples: 2 model_name: gpt-4 output_path: animal_story.pkl source_type: machine_generated   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"Custom Data Generator Guide: ListStringDataGenerator​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#custom-data-generator-guide-liststringdatagenerator","content":" This guide will walk you through creating a custom data generator using the provided BaseDataGenerator.  ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#introduction-2","content":" The ability to programmatically generate data is crucial in scenarios where synthetic or mock data is required, such as in testing, simulations, and more. The provided foundational architecture for data generators allows flexibility and extensibility, enabling you to create custom data generators tailored to specific needs.  In this guide, we will demonstrate how to create a custom data generator by extending the BaseDataGenerator. Our custom generator will output a list of predefined strings. By following this guide, you'll gain an understanding of the structure and process, enabling you to develop even more complex generators as needed.  ","version":"Next","tagName":"h3"},{"title":"Subclassing the BaseDataGenerator​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#subclassing-the-basedatagenerator","content":" First, create a ListStringDataGenerator that simply outputs a list of strings as specified in its configuration.  from typing import Iterator, List from list_string_data_generator_config import ListStringGeneratorConfig from yival.data_generators.base_data_generator import BaseDataGenerator from yival.schemas.common_structures import InputData class ListStringDataGenerator(BaseDataGenerator): def __init__(self, config: 'ListStringGeneratorConfig'): super().__init__(config) def generate_examples(self) -> Iterator[List[InputData]]: for string_data in self.config.strings_to_generate: yield [InputData(example_id=self.generate_example_id(string_data), content=string_data)]     ","version":"Next","tagName":"h3"},{"title":"Providing a Configuration Class​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#providing-a-configuration-class","content":" To specify the list of strings our generator should output, define a custom configuration class:  from dataclasses import dataclass, field from typing import List from yival.schemas.data_generator_configs import BaseDataGeneratorConfig @dataclass class ListStringGeneratorConfig(BaseDataGeneratorConfig): """ Configuration for the ListStringDataGenerator. """ strings_to_generate: List[str] = field(default_factory=list)   ","version":"Next","tagName":"h3"},{"title":"Config​","type":1,"pageTitle":"Data Generation","url":"/YiValApi/docs/API-Reference/data-generation#config","content":" In your configuration (YAML), you can now specify the use of this data generator:  custom_data_generators: list_string_data_generator: class: /path/to/list_string_data_generator.ListStringDataGenerator config_cls: /path/to/list_string_data_generator_config.ListStringGeneratorConfig   dataset: data_generators: list_string_data_generator: strings_to_generate: - abc - def source_type: machine_generated  ","version":"Next","tagName":"h3"},{"title":"Selection","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/selection","content":"","keywords":"","version":"Next"},{"title":"SelectionStrategy​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#selectionstrategy","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#introduction","content":" This module defines an abstract base class for selection strategies. A selection strategy determines how to select or prioritize specific experiments, scenarios, or configurations based on certain criteria.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#example","content":" ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"AHPSelectionStrategy​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#ahpselectionstrategy","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#introduction-1","content":" This module provides the AHPSelection class, which is a selection strategy based on the Analytic Hierarchy Process (AHP). The strategy uses the provided criteria, their weights, and data from the experiment to rank the experiment combinations and select the best one.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#class-definition-1","content":" Description​  A data class defining the configuration for the AHPSelection strategy.  Attributes​  criteria(List[str]): A list of criteria names to be considered during selection. criteria_weights(Dict[str, float]): A dictionary mapping each criterion to its weight. criteria_maximization(Dict[str, bool]): A dictionary indicating whether each criterion should be maximized (True) or minimized (False). normalize_func(Optional[str]): The name of the normalization function to be used, if any.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#example-1","content":" AHPSelection Configuration​   combination_A = CombinationAggregatedMetrics( combo_key=str({"name": "A"}), experiment_results=[], aggregated_metrics={}, average_token_usage=120, average_latency=200, combine_evaluator_outputs=[EvaluatorOutput(name="elo", result=1500)] ) # Combination B has a lower elo, but also much lower token usage and # latency combination_B = CombinationAggregatedMetrics( combo_key=str({"name": "B"}), experiment_results=[], aggregated_metrics={}, average_token_usage=50, average_latency=50, combine_evaluator_outputs=[EvaluatorOutput(name="elo", result=1300)] ) #Combination C has highest elo with highest token usage and latency combination_C = CombinationAggregatedMetrics( combo_key=str({"name": "C"}), experiment_results=[], aggregated_metrics={}, average_token_usage=300, average_latency=300, combine_evaluator_outputs=[EvaluatorOutput(name="elo", result=1600)] ) # The experiment data trade_off_test_data = Experiment( combination_aggregated_metrics=[ combination_A, combination_B, combination_C ], group_experiment_results=[] ) config_trade_off = AHPConfig( criteria=["elo", "average_token_usage", "average_latency"], criteria_weights={ "elo": 0.6, "average_token_usage": 0.2, "average_latency": 0.2 }, criteria_maximization={ "elo": True, "average_token_usage": False, "average_latency": False }, normalize_func='z-score' ) context_trade_off = SelectionContext( strategy=AHPSelection(config=config_trade_off) ) best_combo_trade_off = context_trade_off.execute_selection( trade_off_test_data ) print(best_combo_trade_off)   Use OpenAI Prompt Data Generator in the YiVal config​  selection_strategy: ahp_selection: criteria: - "openai_prompt_based_evaluator: clear" - average_token_usage - average_latency criteria_maximization: "openai_prompt_based_evaluator: clear": true average_latency: false average_token_usage: false criteria_weights: "openai_prompt_based_evaluator: clear": 0.6 average_latency: 0.2 average_token_usage: 0.2 normalize_func: "z-score"   ","version":"Next","tagName":"h3"},{"title":"Custom Selection Strategy Guide: CustomSelectionStrategy​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#custom-selection-strategy-guide--customselectionstrategy","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#introduction-2","content":" Selection strategies are paramount in the experimental framework, guiding the selection or prioritization of experiments, scenarios, or configurations. These strategies can be based on a variety of criteria, ranging from past performance to specific business rules. In this guide, we'll outline the process for creating your own custom selection strategy.  ","version":"Next","tagName":"h3"},{"title":"The Essence of Selection Strategy​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#the-essence-of-selection-strategy","content":" The SelectionStrategy class is the backbone of all selection strategies. It encapsulates core methods to:  Register new selection strategies. Retrieve registered strategies. Access their default configurations. At its core, a selection strategy's primary task is to decide how to select or prioritize specific experiments or configurations.  ","version":"Next","tagName":"h3"},{"title":"Crafting a Custom Selection Strategy​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#crafting-a-custom-selection-strategy","content":" To devise a custom selection strategy, you should inherit from the SelectionStrategy class and implement the select abstract method:  class CustomSelectionStrategy(SelectionStrategy): """ Custom strategy for selecting experiments. """ def select(self, experiment): """ Custom logic for selecting or prioritizing experiments. Args: experiment (Experiment): The experiment under consideration. Returns: SelectionOutput: The result of the selection process. """ # Your selection logic goes here pass   ","version":"Next","tagName":"h3"},{"title":"Config​","type":1,"pageTitle":"Selection","url":"/YiValApi/docs/API-Reference/selection#config","content":" custom_selection_strategies: custom_selection_strategy: class: /path/to/custom_selection_strategy.CustomSelectionStrategy config_cls: /path/to/custom_selection_strategy.CustomSelectionStrategyConfig   To use it:  selection_strategy: custom_selection_strategies:  ","version":"Next","tagName":"h3"},{"title":"State","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/state","content":"","keywords":"","version":"Next"},{"title":"ExperimentState​","type":1,"pageTitle":"State","url":"/YiValApi/docs/API-Reference/state#experimentstate","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"State","url":"/YiValApi/docs/API-Reference/state#introduction","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"State","url":"/YiValApi/docs/API-Reference/state#class-definition","content":" Description​  Represents the state of active experiments and their variations. It provides functionality to retrieve the next variation for an experiment, manage and initialize variations, and get all possible combinations of variations.  Attributes​  active(bool): Indicates if the experiment is currently active. current_variations(Dict[str, List[Any]]): Holds the variations for each experiment. counters(Dict[str, int]): Keeps track of the number of variations used for each experiment.  Methods​  get_instance() -> ExperimentState: Retrieves the instance of the ExperimentState. As it's implemented with a singleton pattern, this method ensures only one instance exists. get_default_state() -> ExperimentState: Retrieves the default state instance of the ExperimentState. get_next_variation(name: str) -> Optional[Any]: Retrieves the next variation for a given experiment name. get_all_variation_combinations() -> List[Dict[str, Any]]: Provides all possible combinations of the variations across all experiments. initialize_variations_from_config(): Initializes the variations for the experiments based on the provided ExperimentConfig. set_variations_for_experiment(name: str, variations: Union[List[Any], Iterator[Any]]): Sets the variations for a given experiment name. clear_variations_for_experiment(): Clears all the variations associated with the experiments. set_experiment_config(config: Any): Sets the ExperimentConfig and initializes the variations accordingly. set_specific_variation(name: str, variation: Any): Assigns a specific variation to an experiment without cycling through the available variations.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"State","url":"/YiValApi/docs/API-Reference/state#example","content":" Initialize: Create an instance of the ExperimentState.   state = ExperimentState.get_instance()   Set Experiment Config: Before retrieving variations, you need to set the experiment configuration.   state.set_experiment_config(your_experiment_config)   Retrieve Variations: Get the next variation for an experiment.   variation = state.get_next_variation('experiment_name')   All Variation Combinations: To get all possible combinations of variations across experiments:   combinations = state.get_all_variation_combinations()  ","version":"Next","tagName":"h3"},{"title":"Common Utils","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/common-utils","content":"","keywords":"","version":"Next"},{"title":"HFInference​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#hfinference","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#introduction","content":" The HFInference class facilitates local inference using models from the HuggingFace transformers library. It provides utilities to load and run inference on a specified model, ensuring efficient text generation directly on the host system.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#class-definition","content":" Description​  The HFInference class provides an interface for local inference using HuggingFace models.  Attributes​  model_name(str): The name or path of the HuggingFace model to be loaded.  Methods​  __init__(self, model_name: str): Initializes the HFInference instance and loads the specified model.Parameters: model_name (str): The name or path of the HuggingFace model. load_model(self, model_name: str) -> Tuple[PreTrainedModel, PreTrainedTokenizer]: Loads the model and corresponding tokenizer from the transformers library.Parameters:model_name (str): The name or path of the HuggingFace model.Returns: A tuple containing the loaded model and its tokenizer. generate(self, prompt: str, max_length: int = 200, temperature: float = 0, top_p: float = 0.99, repetition_penalty: float = 1): Generates text based on the provided prompt using the loaded model.Parameters:prompt (str): The input text or prompt for the model.max_length (int): Maximum length of the generated text. The default value is 200.temperature (float): Sampling temperature. The default value is 0.top_p (float): Nucleus sampling's top-p value. The default value is 0.99.repetition_penalty (float): Repetition penalty factor. The default value is 1.Returns:A generator that yields each generated token or word.  Notes​  The HFInference class incorporates a patched greedy_search method from the transformers library for efficient text generation.It is optimized to work on both CPU and GPU environments, allowing for faster inferencing when GPU support is available.The class dynamically detects the model architecture from the given model's configuration, ensuring compatibility with various HuggingFace model architectures.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#example","content":" # Assuming necessary imports are in place # Initialize the HFInference class with a model name hf_inference = HFInference("gpt2-medium") # Generate text using the model prompt_text = "Once upon a time" generated_tokens = hf_inference.generate(prompt_text, max_length=100) # Display the generated text generated_text = " ".join(generated_tokens) print(generated_text)   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"DocSimilarityUtils​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#docsimilarityutils","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#introduction-1","content":" This module provides utilities for obtaining embeddings of textual data using the OpenAI API and for computing the cosine similarity between two sets of embeddings.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#class-definition-1","content":" Description​  Fetches the embedding for a given string using the OpenAI API.  Methods (Functions)​  get_embedding(input_str: str) -> list[float]**:** Fetches the embedding for a given string using the OpenAI API.Parameters: input_str (str): The input text for which the embedding is to be obtained. Returns: list[float]: A list of floats representing the embedding of the input text. cosine_similarity(a: list[float], b: list[float]) -> float: Computes the cosine similarity between two sets of embeddings.Parameters: a (list[float]): The first set of embeddings.b (list[float]): The second set of embeddings. Returns: float: A float value representing the cosine similarity between the two sets of embeddings. get_cosine_simarity(doc1: str, doc2: str) -> float**:** Computes the cosine similarity between the embeddings of two textual documents.Parameters: doc1 (str): The first document text.doc2 (str): The second document text. Returns: float: A float value representing the cosine similarity between the embeddings of the two documents.  Notes​  Ensure that you have properly set up and authenticated your OpenAI API before using these utilities.The cosine similarity function is a general-purpose function and can be used to compute the similarity between any two sets of embeddings, not just textual embeddings.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#example-1","content":" # get_embedding example embedding = get_embedding("Hello, world!") print(embedding)   # cosine_similarity example embedding1 = [0.2, 0.5, 0.8] embedding2 = [0.1, 0.6, 0.9] similarity = cosine_similarity(embedding1, embedding2) print(similarity)   # get_cosine_similarity example document1 = "The sun shines brightly." document2 = "It's a bright and sunny day." similarity = get_cosine_similarity(document1, document2) print(similarity)   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#source-code-1","content":" ","version":"Next","tagName":"h3"},{"title":"Utils​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#utils","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#introduction-2","content":" This module provides common utility functions designed primarily for asynchronous interactions with OpenAI's API, managing rate limits, and obtaining embeddings.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#class-definition-2","content":" RateLimiter​  Description​  The RateLimiter class ensures that the rate of requests and token usage do not exceed specified limits.  Attributes​  max_rate (int): Maximum number of requests allowed per second.max_tokens_per_minute (int): Maximum number of tokens allowed to be used per minute.start_time (float): Time when the rate limiter was initialized.request_count (int): Number of requests made since initialization.token_usage (deque): A deque containing tuples of tokens used and the time they were used.  Example​  import asyncio # Create an instance of RateLimiter with specified limits rate_limiter = RateLimiter(max_rate=5, max_tokens_per_minute=1000) # A mock function that simulates API request and uses rate limiter async def mock_api_request(): await rate_limiter.wait() # Wait for the rate limiter # Add tokens to the rate limiter (simulating token usage) rate_limiter.add_tokens(50) print("API request made!") # Simulate multiple API requests using asyncio async def main(): tasks = [mock_api_request() for _ in range(10)] await asyncio.gather(*tasks) asyncio.run(main())   ","version":"Next","tagName":"h3"},{"title":"Function Definition​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#function-definition","content":" parallel_completions(message_batches, model, max_tokens,temperature=1.3, presence_penalty=0, pbar=None, logit_bias=None) -> list​  Description​  Asynchronously performs parallel completions using OpenAI's API.  Parameters​  message_batches (list): A list containing batches of messages for completion.model (str): Model to be used for completion.max_tokens (int): Maximum tokens to be used for completion.temperature (float, optional): Sampling temperature. The default value is 1.3.presence_penalty (float, optional): Presence penalty. The default value is 0.pbar (optional): A progress bar instance. The default value is None.logit_bias (optional): Bias for the logit. The default value is None.  Example​  import asyncio # Define the message batches for completion message_batches = [ [{"role": "user", "content": "tell me a joke"}], [{"role": "user", "content": "what's the weather like?"}], [{"role": "user", "content": "how are you?"}], ] # Use the parallel_completions function to get completions async def main(): responses = await parallel_completions( message_batches=message_batches, model="gpt-3.5-turbo", max_tokens=50 ) for response in responses: print(response['choices'][0]['message']['content']) asyncio.run(main())   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Common Utils","url":"/YiValApi/docs/API-Reference/common-utils#source-code-2","content":"","version":"Next","tagName":"h3"},{"title":"Variation Generator","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/variation-generator","content":"","keywords":"","version":"Next"},{"title":"BaseVariationImprover​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#basevariationimprover","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#introduction","content":" This module defines the base class for combination improvers. Combination improvers are responsible for improving the combination of experiments based on their experiment results.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#example","content":" ","version":"Next","tagName":"h3"},{"title":"OpenAIPromptBasedVariationGenerator​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#openaipromptbasedvariationgenerator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#introduction-1","content":" A variation generator that uses the GPT-4 model to generate variations based on the provided prompt configurations.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#class-definition-1","content":" Description​  A data class defining the configuration for the OpenAIPromptBasedVariationGenerator.  Attributes​  model_name(str): The name of the model to be used. The default value is "gpt-4". prompt(Union[str, List[Dict[str, str]]]): The prompt or set of prompts that guide the generation of variations. diversify(bool): Whether to ensure diversification in the generated responses. variables(Optional[List[str]]): Specific variables that should be included in the generated variations. max_tokens(int): The maximum number of tokens for the generated response. The default value is 7000.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#example-1","content":" Using OpenAIPromptBasedVariationGenerator​   generator = OpenAIPromptBasedVariationGenerator( OpenAIPromptBasedVariationGeneratorConfig( prompt=[{ "role": "system", "content": SYSTEM_PRMPOT }, { "role": "user", "content": "Here are some test cases: AI, Weapon\\n\\n Here is the description of the use-case: Given \\{area\\}, write a tech startup headline" }], number_of_variations=2, output_path="test_variation.pkl", diversify=False, variables=["area"] ) ) res = generator.generate_variations() for d in res: print(d)   Using the OpenAIPromptBasedCombinationImprover in YiVal config​  variations: - name: task generator_name: openai_prompt_based_variation_generator generator_config: diversify: false max_tokens: 7000 number_of_variations: 2 model_name: gpt-4 output_path: demo_generated_prompt.pkl prompt: - content: |- Your mission is to craft prompts tailored for GPT-4. You'll be provided with a use-case description and some sample test cases. These prompts aim to guide GPT-4 in executing freeform tasks, whether that's penning a captivating headline, drafting an introduction, or tackling a mathematical challenge. In your designed prompt, delineate the AI's role using lucid English. Highlight its perceptual field and the boundaries of its responses. Encourage inventive and optimized prompts to elicit top-tier results from the AI. Remember, GPT-4 is self-aware of its AI nature; no need to reiterate that. The efficacy of your prompt determines your evaluation. Stay authentic! Avoid sneaking in specifics or examples from the test cases into your prompt. Such maneuvers will lead to immediate disqualification. Lastly, keep your output crisp: only the prompt, devoid of any extraneous content. role: system - content: |- Use case description: Given an tech startup business, generate a corresponding landing page headline Test Cases: Food Delivery, AI Developer tools. role: user - content: |- {tech_startup_business} represent the specific test cases. role: user variables: - tech_startup_business   ","version":"Next","tagName":"h3"},{"title":"Custom Variation Generator Subclass Guide: generate_variations​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#custom-variation-generator-subclass-guide-generate_variations","content":" This guide explains how to create a custom variation generator subclass based on the BaseVariationGenerator for experimental variations.  ","version":"Next","tagName":"h2"},{"title":"Understand the Base​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#understand-the-base","content":" The BaseVariationGenerator provides foundational methods and attributes for all variation generators. Subclasses should implement the generate_variations method to define the logic for producing variations.  Example: SimpleVariationGenerator​  Let's design a generator that simply returns variations based on the configurations provided.  ","version":"Next","tagName":"h3"},{"title":"Define the Configuration Class​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#define-the-configuration-class","content":" Firstly, you'll need a configuration class specific to your generator:  from dataclasses import dataclass from yival.schemas.varation_generator_configs import BaseVariationGeneratorConfig @dataclass class SimpleVariationGeneratorConfig(BaseVariationGeneratorConfig): variations: Optional[List[str]] = None # List of variations to generate   This configuration class inherits from BaseVariationGeneratorConfig and has an additional attribute, variations, which is a list of variation strings.  ","version":"Next","tagName":"h3"},{"title":"Implement the Variation Generator​","type":1,"pageTitle":"Variation Generator","url":"/YiValApi/docs/API-Reference/variation-generator#implement-the-variation-generator","content":" Now, let's create the custom variation generator:  from typing import Iterator, List from yival.schemas.experiment_config import WrapperVariation from yival.variation_generators.base_variation_generator import BaseVariationGenerator class SimpleVariationGenerator(BaseVariationGenerator): def __init__(self, config: SimpleVariationGeneratorConfig): super().__init__(config) self.config = config def generate_variations(self) -> Iterator[List[WrapperVariation]]: variations = [WrapperVariation(value_type="str", value=var) for var in self.config.variations] yield variations   Here, the generate_variations method simply converts the list of variation strings from the configuration into a list of WrapperVariation objects and yields it.  Using the Custom Variation Generator in Configuration​  In your configuration (YAML), you can now specify the use of this variation generator:  custom_variation_generators: simple_variation_generator: class: /path/to/simple_variation_generator.SimpleVariationGenerator config_cls: /path/to/simple_variation_generator_config.SimpleVariationGeneratorConfig   variations: - name: task generator_name: simple_variation_generator generator_config: variations: - abc - def   This configuration will use the SimpleVariationGenerator and produce the variations "variation1" and "variation2". ","version":"Next","tagName":"h3"},{"title":"Examples","type":0,"sectionRef":"#","url":"/YiValApi/docs/Examples","content":"","keywords":"","version":"Next"},{"title":"🤖 Headline Generation for a Startup Company​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#-headline-generation-for-a-startup-company","content":" Startup companies often invest a significant amount of time and effort into creating the perfect homepage headline that suits their brand. Yival can assist with generating these headlines using a selected machine learning model such as GPT-4 by compiling through different combinations of the generated data. Later, we can select an evaluator such as the Analytic Hierarchy Process (AHP) to pinpoint the ideal headline for a startup company’s website.    Please take a look at this notebook to view the implementation of headline generation for a startup company.  To generate the proper prompts, we recommend using a step-by-step optimization pipeline mode, a method that continuously adjusts itself based on the test case results. Here you can find an example: Notebook  ","version":"Next","tagName":"h2"},{"title":"🐯 Animal Story Generation with MidJourney​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#-animal-story-generation-with-midjourney","content":" Generating compelling visuals for a narrative can be quite demanding, but with YiVal's AI optimization framework, it becomes a remarkably straightforward task. In this demo, you only need to input the initial prompt, and YiVal will automatically generate a variety of animals with unique personalities, and craft charming and concise animal stories using various templates. In addition, Midjourney is employed to create the accompanying images. What's truly remarkable is that this entire process is fully automated, requiring just your initial prompt.    Please take a look at this notebook to view the implementation of animal story generation with MidJourney.  ","version":"Next","tagName":"h2"},{"title":"📈 Comparison of LLM Model Evaluations​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#-comparison-of-llm-model-evaluations","content":" NLP practitioners often face the challenge of assessing various models' capabilities. Yival provides a variety of generalized evaluation methods, by simply requiring the provision of data and configuration files. In this demo, we performed a side-by-side comparison of the QA abilities of the following four models:  GPT-3.5-TurboLlama-2-13b-chatLlama-2-70b-chatVicuna-13b    Please take a look at this notebook to view the implementation of animal story generation with MidJourney.  ","version":"Next","tagName":"h2"},{"title":"🕹️ YiVal's Basic Interactive Mode​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#️-yivals-basic-interactive-mode","content":" ","version":"Next","tagName":"h2"},{"title":"Basic Interactive Mode​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#basic-interactive-mode","content":" To get started with a demo of the basic interactive mode of YiVal, simply run the following command:  yival demo --basic_interactive   Once started, navigate to the following address in your web browser: http://127.0.0.1:8073/interactive    ","version":"Next","tagName":"h3"},{"title":"Question Answering with Expected Result Evaluator​","type":1,"pageTitle":"Examples","url":"/YiValApi/docs/Examples#question-answering-with-expected-result-evaluator","content":" Similarly, as above, use the command below to get started with Question Answering with Expected Result Evaluator demo:  yival demo --qa_expected_results   Once started, navigate to the following address in your web browser: http://127.0.0.1:8073   ","version":"Next","tagName":"h3"},{"title":"Wrapper","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/wrapper","content":"","keywords":"","version":"Next"},{"title":"BaseWrapper​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#basewrapper","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#introduction","content":" This module introduces the BaseWrapper class, which serves as the fundamental structure for wrappers in the experimental framework. Wrappers manage experiment variations based on the global experiment state. They are crucial components to control and monitor different variations during an experiment's lifecycle.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#class-definition","content":" Description​  Base class for wrappers that manage experiment variations based on the global experiment state. This class provides the fundamental structure and methods for wrappers. Specific wrappers should inherit from this class and implement the necessary methods.  Attributes​  name(str): The name of the wrapper. experiment_state(ExperimentState): The global state of the experiment. config(BaseWrapperConfig): The configuration for the wrapper.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#example","content":" ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"StringWrapper​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#stringwrapper","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#introduction-1","content":" This module provides a StringWrapper class designed to manage string variations based on the current experiment state. It helps in retrieving and formatting string variations in the context of an active experiment.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#class-definition-1","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#example-1","content":" wrapper = StringWrapper(template="Hello, {name}!", name="greeting_experiment", variables={"name": "John"}) print(str(wrapper)) # This will print the appropriate greeting based on the experiment state.   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#source-code-1","content":" ","version":"Next","tagName":"h3"},{"title":"Custom Wrapper Guide: NumberWrapper​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#custom-wrapper-guide-numberwrapper","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#introduction-2","content":" In the experimental framework, wrappers play a vital role in managing variations throughout an experiment's lifecycle. By creating custom wrappers, one can control and monitor variations tailored to specific needs, ensuring that the experiment operates smoothly and efficiently.  In this guide, we will walk you through the process of creating a custom wrapper named NumberWrapper. This wrapper will handle variations specifically for numbers. By the end of this guide, you will have a clear understanding of creating and registering a custom wrapper within the experimental framework.  ","version":"Next","tagName":"h3"},{"title":"Base Wrapper Overview​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#base-wrapper-overview","content":" The BaseWrapper class provides the fundamental structure for wrappers. It comes equipped with methods to register new wrappers, retrieve registered ones, and fetch their configurations. The primary purpose of a wrapper is to manage experiment variations based on the global experiment state.  ","version":"Next","tagName":"h3"},{"title":"Creating a NumberWrapper​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#creating-a-numberwrapper","content":" The NumberWrapper will be a custom wrapper designed to handle variations specifically for numbers.  from typing import Optional from number_wrapper_config import NumberWrapperConfig from yival.wrappers.base_wrapper import BaseWrapper class NumberWrapper(BaseWrapper): """ A wrapper for numbers to manage experiment variations based on the global experiment state. If a variation for the given experiment name exists and the global ExperimentState is active, the variation is used. Otherwise, the original number is returned. """ default_config = NumberWrapperConfig() def __init__( self, value: float, name: str, config: Optional[NumberWrapperConfig] = None state: Optional[ExperimentState] = None ) -> None: super().__init__(name, config, state) self._value = value def get_value(self) -> float: variation = self.get_variation() if variation is not None: return variation return self._value   Here, the NumberWrapper class is responsible for retrieving a variation if one exists, otherwise it returns the original number. The get_value method is used to fetch the number, considering any variations.  ","version":"Next","tagName":"h3"},{"title":"Registering the NumberWrapper​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#registering-the-numberwrapper","content":" To make the NumberWrapper usable within the experimental framework, it needs to be registered. The registration process involves mapping the wrapper's name to its class and configuration.  from dataclasses import dataclass from yival.schemas.wrapper_configs import BaseWrapperConfig @dataclass class NumberWrapperConfig(BaseWrapperConfig): """ Configuration specific to the NumberWrapper. """ pass   By calling the register_wrapper method, the NumberWrapper becomes available for use in the experimental framework.  ","version":"Next","tagName":"h3"},{"title":"Config​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#config","content":" Now you can configure the wrapper in the yml file:  custom_wrappers: number_wrapper: class: /path/to/number_wrapper.NumberWrapper config_cls: /path/to/number_wrapper_config.NumberWrapperConfig   And you should be able to use the wrapper in your code like a string wrapper.  ","version":"Next","tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Wrapper","url":"/YiValApi/docs/API-Reference/wrapper#conclusion","content":" By following this guide, you've successfully created and registered a custom wrapper named NumberWrapper in the experimental framework. This flexibility allows you to tailor experiments to specific needs, ensuring accurate and efficient results. ","version":"Next","tagName":"h3"},{"title":"Evaluation","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/evaluation","content":"","keywords":"","version":"Next"},{"title":"BaseEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#baseevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition","content":" Description​  Evaluators are central components in the experimental framework that interpret experiment results and offer either quantitative or qualitative feedback.  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example","content":" ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"EvaluatorType​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#evaluatortype","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-1","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-1","content":" Description​  The EvaluatorType enumeration class delineates the various evaluation methodologies available within the evaluator's module. Each type signifies a distinct approach to model assessment.  Values​  INDIVIDUAL: This evaluation type signifies that each model or output is evaluated on its own merit, without considering other variations or examples. COMPARISON: In this evaluation type, one specific example is given, and models or outputs are compared based on how they handle that particular example. ALL: This type indicates a comprehensive evaluation approach, where all available examples are considered. Each model or output is evaluated across the entirety of the provided dataset, giving a holistic view of its performance.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-1","content":" ","version":"Next","tagName":"h3"},{"title":"AlpacaEvalEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#alpacaevalevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-2","content":" The AlpacaEvalEvaluator class is designed to facilitate evaluations using the Alpaca Eval system. This system is employed to rank different model outputs based on human evaluations. The specific implementation interfaces with the OpenAI API to carry out these evaluations, ensuring a blend of automation with human-like discernment.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-2","content":" Description​  Tailored for the AlpacaEvalEvaluator, this configuration class sets the guidelines and parameters for the evaluation process.  Attributes​  alpaca_annotator_name(str): Specifies the Alpaca Eval annotator to be used. The default value is "alpaca_eval_gpt4". matching_technique(MatchingTechnique): Determines the technique used for matching during the evaluation. The default value is MatchingTechnique.MATCH.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-2","content":" Evaluating Sample Data with Alpaca Evaluator​  # Creating a configuration for AlpacaEvalEvaluator evaluator_config = AlpacaEvalEvaluatorConfig( name="alpaca_eval_evaluator", alpaca_annotator_name="alpaca_eval_gpt4", evaluator_type=EvaluatorType.COMPARISON ) # Initializing the evaluator with the given configuration evaluator = AlpacaEvalEvaluator(evaluator_config) # Sample data for assessment sample_group_data = [...] # A list of ExperimentResult objects # Evaluating the sample data evaluator.evaluate_comparison(sample_group_data) # Printing the evaluation results for experiment in sample_group_data: print(experiment.evaluator_outputs)   Use Alpaca Evaluator in the YiVal config​  evaluators: - evaluator_type: comparison alpaca_annotator_name: alpaca_eval_gpt4 metric_calculators: - method: AVERAGE name: alpaca_eval_evaluator   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-1","content":" ","version":"Next","tagName":"h3"},{"title":"BertScoreEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#bertscoreevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-3","content":" BERTScore is an evaluation metric for language models based on the BERT language model. Instead of relying solely on token matching, BERTScore leverages the contextual embeddings from BERT to match words in the candidate and reference sentences using cosine similarity. This makes it robust against paraphrasing, allowing it to effectively gauge the semantic similarity between two sentences. Its strong correlation with human judgment, both on sentence-level and system-level evaluations, makes it an attractive metric for various NLP tasks.  ","version":"Next","tagName":"h3"},{"title":"BertScoreEvaluatorConfig​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#bertscoreevaluatorconfig","content":" Description​  Configuration class for the BertScoreEvaluator. It specifies various parameters that dictate how the evaluation using BERTScore will be performed.  Attributes​  evaluator_type: Specifies the type of evaluation. In the context of the BertScoreEvaluator, it's set to EvaluatorType.INDIVIDUAL, meaning each model output is evaluated individually against its corresponding expected result.The default value is EvaluatorType.INDIVIDUAL. description: A descriptive text providing additional context or details about the evaluator.The default value is "This is the description of the evaluator". lan: Language of the text being evaluated. BERTScore uses this to select the appropriate pre-trained BERT model. For instance, 'zh' corresponds to Chinese.The default value is 'zh'. indicator: Specifies which score to return out of precision (p), recall (r), or F1 score (f).The default value is precision or 'p'. name: Name of the evaluator. display_name: Display name for the metric. lan: Language of the text being evaluated. BERTScore uses this to select the appropriate pre-trained BERT model. metric_calculators: List of additional metric calculators to be applied.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-3","content":" In the usage example below, the evaluator is set up to compute the F1 BERTScore for a model's translation against the expected translation.  evaluator_config = BertScoreEvaluatorConfig( name="bertscore_evaluator", display_name="bertscore", lan="en", indicator="f", metric_calculators=[] ) input_data_example = InputData( content={ "instruction": "Translate the sentence to English.", }, expected_result="Have a great day!" ) experiment_result_example = ExperimentResult( input_data=input_data_example, combination={ "wrapper1": "var1", "wrapper2": "var2" }, raw_output=MultimodalOutput(text_output="Have a nice day!"), latency=30.0, token_usage=20 ) evaluator = BertScoreEvaluator(evaluator_config) result = evaluator.evaluate(experiment_result_example) print(result)   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-2","content":" ","version":"Next","tagName":"h3"},{"title":"OpenAIEloEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#openaieloevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-4","content":" This module provides an implementation of the ELO-based evaluation system in the OpenAIEloEvaluator class. The ELO system ranks different model outputs based on human evaluations. This specific implementation interfaces with the OpenAI API to conduct those evaluations.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-3","content":" Description​  Configuration class for the OpenAIEloEvaluator. It specifies various parameters that dictate how the evaluation using the ELO system interfacing with OpenAI will be performed.  Attributes​  openai_model_name: Specifies which OpenAI model to use for the evaluation.The default is set to use the "gpt-4" model. input_description: Provides a description of the input data that will be evaluated. This can be utilized when presenting data to the OpenAI model for ranking.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-4","content":" Creating the OpenAIEloEvaluatorConfig File​   evaluator = OpenAIEloEvaluator( OpenAIEloEvaluatorConfig( name="openai_elo_evaluator", input_description="Translate the given English sentence to French", evaluator_type=EvaluatorType.ALL, ) ) experiment = create_test_data_v2() evaluator.evaluate_based_on_all_results([experiment]) print(experiment)   Use BertScoreEvaluator in YiVal Config​  evaluators: - evaluator_type: all input_description: Given an tech startup business, generate one corresponding landing page headline metric_calculators: [] name: openai_elo_evaluator model_name: gpt-4   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-3","content":" ","version":"Next","tagName":"h3"},{"title":"PythonValidationEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#pythonvalidationevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-5","content":" Evaluates the raw output of an experiment by attempting to execute it as Python code. If the code executes without any errors, a positive result is returned. Otherwise, a negative result is returned.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-4","content":" Description​  Configuration class for the PythonValidationEvaluator. It specifies the matching technique to be used for the evaluation.  Attributes​  matching_technique: The default value is MatchingTechnique.MATCH.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-5","content":" evaluators: - evaluator_type: individual matching_technique: includes metric_calculators: - method: AVERAGE name: python_validation_evaluator   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-4","content":" ","version":"Next","tagName":"h3"},{"title":"RougeEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#rougeevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-6","content":" The Rouge Evaluator is an advanced tool tailored for assessing the quality of dialogue model outputs. Leveraging the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) metric, this evaluator emphasizes the relevance, coherence, and fluency of the generated content. Designed specifically for dialogue systems, the Rouge Evaluator offers developers and researchers a refined measure to gauge their model's efficiency, ensuring continuous improvement.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-5","content":" Description​  Configuration class for the RougeEvaluator. It specifies the type of ROUGE metric to be used for the evaluation.  Attributes​  evaluator_type: Specifies that the evaluator assesses each experiment individually.The default value is EvaluatorType.INDIVIDUAL. description: A brief description of the evaluator.The default value is "This is the description of the evaluator". rough_type: Specifies which type of ROUGE metric to use (e.g., "rouge-1", "rouge-2", "rouge-L").The default value is "rouge-1".  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-6","content":" import RougeEvaluator, RougeEvaluatorConfig import ExperimentResult experiment_result = ExperimentResult( # ... your experiment result details here ... ) config = RougeEvaluatorConfig() evaluator = RougeEvaluator(config) output = evaluator.evaluate(experiment_result) print(output)   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-5","content":" ","version":"Next","tagName":"h3"},{"title":"StringExpectedResultEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#stringexpectedresultevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-7","content":" The String Expected Result Evaluator introduces a specialized class designed for evaluating expected results that are strings. This class extends the foundational evaluator and is equipped to compare actual and expected outputs using various matching techniques. The primary goal is to determine the accuracy of the generated strings in comparison to the expected outputs.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#class-definition-6","content":" Description​  This class extends the BaseEvaluator and provides specific implementation for evaluating string expected results using different matching techniques.  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#example-7","content":" UseStringExpectedResult in YiVal Config​  evaluators: - evaluator_type: individual matching_technique: includes metric_calculators: - method: AVERAGE name: string_expected_result   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#source-code-6","content":" ","version":"Next","tagName":"h3"},{"title":"Custom Evaluator Guide: SimpleEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#custom-evaluator-guide-simpleevaluator","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#introduction-8","content":" Evaluators are central components in the experimental framework that interpret experiment results and offer either quantitative or qualitative feedback. This guide will walk you through the steps of creating a custom evaluator, named SimpleEvaluator, which returns a value of 1 if the result is 1, and 0 otherwise.  ","version":"Next","tagName":"h3"},{"title":"Creating the SimpleEvaluator Configuration​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#creating-the-simpleevaluator-configuration","content":" Before creating the evaluator, we define the configuration for our SimpleEvaluator. The configuration helps in defining how the evaluator should behave and what parameters it may require.  from dataclasses import dataclass, field from typing import Any, Dict, List from yival.schemas.evaluator_config import ( BaseEvaluatorConfig, EvaluatorType, MetricCalculatorConfig, ) @dataclass class SimpleEvaluatorConfig(BaseEvaluatorConfig): """ Configuration for SimpleEvaluator. """ metric_calculators: List[MetricCalculatorConfig] = field( default_factory=list ) evaluator_type = EvaluatorType.INDIVIDUAL def asdict(self) -> Dict[str, Any]: base_dict = super().asdict() base_dict["metric_calculators"] = [ mc.asdict() for mc in self.metric_calculators ] return base_dict   ","version":"Next","tagName":"h3"},{"title":"Implementing the SimpleEvaluator​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#implementing-the-simpleevaluator","content":" Now, let's create the SimpleEvaluator that utilizes the above configuration:  from simple_evaluator import SimpleEvaluatorConfig from yival.evaluators.base_evaluator import BaseEvaluator from yival.schemas.evaluator_config import EvaluatorOutput @BaseEvaluator.register("simple_evaluator") class SimpleEvaluator(BaseEvaluator): """ A basic evaluator that returns a value of 1 if the result is 1, and 0 otherwise. """ def __init__(self, config: SimpleEvaluatorConfig): super().__init__(config) def evaluate(self, experiment_result) -> EvaluatorOutput: """ Evaluate the experiment result and produce an evaluator output. Args: experiment_result: The result of an experiment to be evaluated. Returns: EvaluatorOutput: The result of the evaluation. """ result = 1 if experiment_result == 1 else 0 return EvaluatorOutput(name="Simple Evaluation", result=result)   ","version":"Next","tagName":"h3"},{"title":"Config​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#config","content":" Next you can config the evaluator  custom_evaluators: simple_evaluator: class: /path/to/simple_evaluator.SimpleEvaluator config_cls: /path/to/simple_evaluator_config.SimpleEvaluatorConfig   And use it  evaluators: - name: simple_evaluator simple_evaluator: metric_calculators: []   ","version":"Next","tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Evaluation","url":"/YiValApi/docs/API-Reference/evaluation#conclusion","content":" By following this guide, you've successfully developed, configured, and registered a custom evaluator named SimpleEvaluator within the experimental framework. Custom evaluators, like the one you've created, enable a tailored approach to interpreting and analyzing experiment results, ensuring the specific needs of an experiment are met. ","version":"Next","tagName":"h3"},{"title":"Getting Started","type":0,"sectionRef":"#","url":"/YiValApi/docs/Getting-Started","content":"","keywords":"","version":"Next"},{"title":"Prerequisites & Installation​","type":1,"pageTitle":"Getting Started","url":"/YiValApi/docs/Getting-Started#prerequisites--installation","content":" After making sure we have all the prerequisites to install YiVal, we begin installing YiVal using any of the methods mentioned in Section Installation Methods.  ","version":"Next","tagName":"h2"},{"title":"YiVal's Role in Retrieval​","type":1,"pageTitle":"Getting Started","url":"/YiValApi/docs/Getting-Started#yivals-role-in-retrieval","content":" In this task, we use FAISS to store prompts to be able to quickly search and retrieve the prompts. In addition, FAISS allows us to turn prompts into vectors, which facilitates finding the ideal prompt. YiVal simplifies experimentation with AI applications and tasks while being integrated with FAISS, in two main ways:  Find Matching Prompts: Based on a given situation, Yival searches the FAISS database to find similar prompts.Refine with GPT: Sometimes, the initially found prompts might not be perfect. So, we use GPT to rerank them, ensuring that we pick the most suitable one.    ","version":"Next","tagName":"h2"},{"title":"Generating Prompts​","type":1,"pageTitle":"Getting Started","url":"/YiValApi/docs/Getting-Started#generating-prompts","content":" Putting this all together alongside using ChatGPT 3.5 would look like:  # Create a chat message sequence messages = [{ "role": "user", "content": str(StringWrapper("", name="prompt")) + f'\\n{input}' }] # Get a response from ChatGPT response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=messages )   Notice that there was no need to provide a predefined prompt using this approach.  ","version":"Next","tagName":"h2"},{"title":"YiVal Configuration File​","type":1,"pageTitle":"Getting Started","url":"/YiValApi/docs/Getting-Started#yival-configuration-file","content":" Now, we start working on creating a YiVal configuration file. Here is an example of YiVal Configuration file for travel guide use case:  custom_function: demo.auto_prompt_bot.reply dataset: source_type: user_input description: Basic config for interactive mode evaluators: [] custom_variation_generators: retrivel_variation_generator: class: demo.prompts_retrivel.retrivel_variation_generator.RetrivelVariationGenerator config_cls: demo.prompts_retrivel.retrivel_variation_generator_config.RetrivelVariationGeneratorConfig variations: - name: prompt generator_config: use_case: "travel guide" generator_name: retrivel_variation_generator   Here we have specified the dataset to be from user input and used demo.auto_prompt_bot.reply as a custom function and used retrivel_variation_generator to generate a prompt related to travel guide use case.  ","version":"Next","tagName":"h2"},{"title":"Real-Time Results​","type":1,"pageTitle":"Getting Started","url":"/YiValApi/docs/Getting-Started#real-time-results","content":" You can see the results achieved in real-time below. Please check this notebook to learn about this process in further detail.   ","version":"Next","tagName":"h2"},{"title":"Introduction","type":0,"sectionRef":"#","url":"/YiValApi/docs/Introduction","content":"Introduction YiVal is a versatile platform and framework designed to streamline the evaluation and enhancement of your Generative AI applications. Its core mission is to comprehensively simplify and optimize the development process for Generative AI applications. YiVal achieves this by taking a generative AI task and utilizing it to generate data tailored for a specific objective. It then explores all potential combinations, subsequently evaluating and selecting the optimal parameters for enhancement. YiVal serves as a versatile tool for various purposes: Generative AI Model Enhancement: Streamline the process of improving the performance of Generative AI models by fine-tuning prompts, parameters, and retrieval mechanisms. Latency Reduction: Optimize your AI applications to deliver faster responses and reduce latency, enhancing the user experience. Cost Efficiency: Lower the inference costs associated with your AI applications while maintaining or improving their quality and performance. Customized Testing: Tailor your test data and evaluation methods to suit the specific requirements of your Generative AI applications. Comprehensive Evaluation: Evaluate various aspects of your AI models and applications comprehensively, including prompts, model metadata, and retrieval configurations. Multi-Modal Support: Enable your AI applications to seamlessly process and generate various data modalities, including text, images, audio, and video. Unified Platform: Simplify your workflow by accessing all these capabilities within a single, user-friendly interface, making enhancement and evaluation more efficient. YiVal offers an open-source framework that caters to a wide range of users, from individuals and beginners to enterprises. For beginners, the platform provides an automated agent mode, offering a seamless one-click experience for creating applications. Enterprise users, on the other hand, can leverage both experiment and production modes to optimize and enhance complex models, providing flexibility and scalability to suit different needs. To address your specific needs tailored just for you, please reach out to us at contactus@yival.io","keywords":"","version":"Next"},{"title":"Logger","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/logger","content":"","keywords":"","version":"Next"},{"title":"TokenLogger​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#tokenlogger","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#introduction","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#class-definition","content":" Description​  The TokenLogger class is a singleton class designed to keep track of token usage. As a singleton, it ensures that there's only one instance of the logger throughout the runtime of the application. The class provides methods to log token usage, retrieve the current usage, and reset the counter.  Methods​  ","version":"Next","tagName":"h3"},{"title":"log​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#log","content":" Logs or adds the number of tokens used to the current usage counter.Parameters: tokens(int): The number of tokens to add to the current usage counter.  ","version":"Next","tagName":"h3"},{"title":"get_current_usage​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#get_current_usage","content":" Retrieves the current token usage.Returns: int: The token usage of the current operation.  ","version":"Next","tagName":"h3"},{"title":"reset​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#reset","content":" Resets the current token usage counter to zero.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Logger","url":"/YiValApi/docs/API-Reference/logger#example","content":" To utilize the TokenLogger:  Initialization: Since it's a singleton, you don't need to worry about multiple instances.   logger = TokenLogger()   Logging Tokens: You can add tokens to the counter.   logger.log(150)   Retrieving Usage: Get the current token usage.   current_usage = logger.get_current_usage() print(current_usage) # This will print 150   Resetting the Counter: If you want to reset the token usage:   logger.reset()  ","version":"Next","tagName":"h3"},{"title":"Architecture","type":0,"sectionRef":"#","url":"/YiValApi/docs/Architecture","content":"","keywords":"","version":"Next"},{"title":"Data Generation​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#data-generation","content":" The process starts with the generation of a dataset which can come from multiple sources:  Specific Data Generator: A defined method or algorithm that automatically churns out data.Data Reader: A component that reads data from external places.Manual Input: As straightforward as it sounds, data can be added manually.  ","version":"Next","tagName":"h2"},{"title":"Combination Creation​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#combination-creation","content":" Once we have our dataset, we form combinations that are pivotal for the subsequent analysis:  Formed using specific combination creators.Defined manually.  ","version":"Next","tagName":"h2"},{"title":"Analysis​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#analysis","content":" This is the heart of the Yival framework. A custom function provided by the user takes in the dataset and combination list to produce valuable insights.  ","version":"Next","tagName":"h2"},{"title":"Evaluation​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#evaluation","content":" After analysis, the results are subjected to evaluation. Several methodologies can be applied to grasp and gauge the data's behavior deeply.  ","version":"Next","tagName":"h2"},{"title":"Selection​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#selection","content":" From the evaluations, the most promising results are selected. This process ensures only the most vital insights are pushed forward.  ","version":"Next","tagName":"h2"},{"title":"Enhancer​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#enhancer","content":" The selected results are then fine-tuned in this phase. An "Enhancer" is applied to enhance these results. This stage can loop back to the analysis stage, indicating an ongoing, iterative process of refinement.  ","version":"Next","tagName":"h2"},{"title":"Trainer​","type":1,"pageTitle":"Architecture","url":"/YiValApi/docs/Architecture#trainer","content":" Yival also supports finetuning models, and we offer two methods:  Use various dataset generators built into Yival (including Huggingface, etc.) for data upload or generation, and then finetune the model.Provide a custom_func, use advanced models like GPT-4 for data generation, and customize the selection criteria. The model is then fine-tuned based on the selected data.  Yival supports all base models supported by Huggingface and provides advanced fine-tuning methods such as LoRA, and 8bit/4bit quantization.   ","version":"Next","tagName":"h2"},{"title":"Installation","type":0,"sectionRef":"#","url":"/YiValApi/docs/Installation","content":"","keywords":"","version":"Next"},{"title":"Prerequisites​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#prerequisites","content":" Python Version: Ensure that Python 3.10 or later is installed. We suggest using a Python version manager such as PyEnv, Virtualenv, and Anaconda, if needed.OpenAI API Key: Obtain an API key from OpenAI. Once you have the key, set it as an environment variable named OPENAI_API_KEY.  ","version":"Next","tagName":"h2"},{"title":"Installation Methods​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#installation-methods","content":" ","version":"Next","tagName":"h2"},{"title":"Using pip (Recommended)​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#using-pip-recommended","content":" You can install the yival package directly using pip:  pip install yival   ","version":"Next","tagName":"h3"},{"title":"Using Poetry​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#using-poetry","content":" If you're interested in contributing or configuring a development environment, Poetry is the recommended choice. Below are the step-by-step instructions to help you get started:  Install Poetry: If you haven't already, install Poetry.Clone the Repository:  git clone https://github.com/YiVal/YiVal.git cd YiVal   Setup with Poetry: Initialize the Python virtual environment and install dependencies using Poetry:  poetry install --sync   ","version":"Next","tagName":"h3"},{"title":"Create Your First YiVal Program​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#create-your-first-yival-program","content":" Once the setup is complete, you can swiftly begin your journey with YiVal by creating datasets containing randomly generated tech startup business names following the instructions below. You can find a step-by-step guide below to begin creating your first YiVal program. Here we provide 2 ways to start with:  ","version":"Next","tagName":"h2"},{"title":"Run With Poetry​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#run-with-poetry","content":" Set OpenAI API Key: Replace $YOUR_OPENAI_API_KEY with your actual OpenAI API key.  export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY   Execute YiVal: Run the following command from within the /YiVal directory:  poetry run yival [your args]   If you want to run generation task, you can replace your args with gen. Such as:  poetry run yival gen   If you want to run with your custom config, you can replace your args with your config path. Such as:  poetry run yival demo\\configs\\alpaca_eval.yml   ","version":"Next","tagName":"h3"},{"title":"Run With YiVal​","type":1,"pageTitle":"Installation","url":"/YiValApi/docs/Installation#run-with-yival","content":" Navigate to the yival Directory:  cd /YiVal/src/yival   Set OpenAI API Key: Replace $YOUR_OPENAI_API_KEY with your actual OpenAI API key.  export OPENAI_API_KEY=$YOUR_OPENAI_API_KEY   Define YiVal Configuration: Create a configuration file named config_data_generation.yml for automated test dataset generation with the following content:  description: Generate test data dataset: data_generators: openai_prompt_data_generator: chunk_size: 100000 diversify: true model_name: gpt-4 input_function: description: # Description of the function Given a tech startup business, generate a corresponding landing page headline name: headline_generation_for_business parameters: tech_startup_business: str # Parameter name and type number_of_examples: 3 output_csv_path: generated_examples.csv source_type: machine_generated   Execute YiVal: Run the following command from within the /YiVal/src/yival directory:  yival run config_data_generation.yml   Check the Generated Dataset: The generated test dataset will be stored in generated_examples.csv. ","version":"Next","tagName":"h3"},{"title":"Combination Improver","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/combination-improver","content":"","keywords":"","version":"Next"},{"title":"BaseCombinationImprover​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#basecombinationimprover","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#introduction","content":" This module defines the base class for combination improvers. Combination improvers are responsible for improving the combination of experiments based on their experiment results.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#example","content":" ","version":"Next","tagName":"h3"},{"title":"OpenAIPromptBasedCombinationImprover​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#openaipromptbasedcombinationimprover","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#introduction-1","content":" This module provides an implementation of a combination improver using OpenAI's model to suggest improvements. It leverages the capabilities of OpenAI's language model to analyze the results of an experiment and provide suggestions on how to improve the combination of parameters. The module offers a prompt-based mechanism where the language model is prompted with structured information about the experiment and its results. The model then responds with potential improvements.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#class-definition-1","content":" Description​  The configuration object is specifically tailored for the OpenAIPromptBasedCombinationImprover class. It provides settings to control the iterative improvement process using OpenAI's model for suggesting combination improvements.  Attributes​  openai_model_name(str)**:** Specifies the OpenAI model to be used.The default value is "gpt-4". max_iterations(int)**:** The maximum number of iterations for the improvement process.The default value is set to 3. stop_conditions(Optional[Dict[str, float]])**:** Conditions to halt the iterative improvement process based on evaluator scores.The default value is None. average_score(Optional[float]) A threshold for the average score to stop the iterative process.The default value is None. selection_strategy(Optional[SelectionOutput]) Strategy for selecting the best combination.The default value is None.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#example-1","content":" ","version":"Next","tagName":"h3"},{"title":"OptimizeByPromptImprover​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#optimizebypromptimprover","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#introduction-2","content":" This module offers an implementation of Optimization by PROmpting (OPRO), a method inspired by the paper Optimization by PROmpting). OPRO leverages the capabilities of large language models (LLMs) like GPT-4 to optimize tasks by iteratively refining prompts.  The optimization process is driven by a structured prompt that consists of the following sections:  HEAD_META_INSTRUCTION SOLUTION_SCORE_PAIRS OPTIMATION_TASK_FORMAT (optional) END_META_INSTRUCTION As the optimization process progresses through iterations, new evaluator scores and prompts are appended to the SOLUTION_SCORE_PAIRS section, enhancing the overall prompt for the next iteration. For practical demonstrations of this concept, refer to the file demo/configs/headline_generation_improve.yml and the appendix of the aforementioned paper.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#class-definition-2","content":" OptimizeByPromptImproverConfig(BaseCombinationImproverConfig)​  Description​  The configuration object tailored for theOptimizeByPromptImprover class, controlling how the optimization by prompting is executed.  Attributes​  improve_var(List[str]): List of variables that need improvement. head_meta_instruction(str): The initial instructional part of the prompt. end_meta_instruction(str): The ending part of the prompt. optimation_task_format(Optional[str]): An optional task format for optimization. model_name(str): Specifies the LLM model to be used.The default value is "gpt-4". max_iterations(int): The maximum number of optimization iterations.The default value is 3.  Example​  Use OPRO Improver to Enhace Experiment Results​  # Create an OPRO configuration improver_config = OptimizeByPromptImproverConfig( improve_var=["task"], head_meta_instruction="Start of the prompt...", end_meta_instruction="End of the prompt...", optimation_task_format="Optional format...", model_name="gpt-4", max_iterations=5 ) # Initialize the OPRO improver with the configuration improver = OptimizeByPromptImprover(improver_config) # Use the improver to enhance an experiment's prompts improved_output = improver.improve(experiment, config, evaluator, token_logger)   Using the OpenAIPromptBasedCombinationImprover in YiVal config​  improver: name: "optimize_by_prompt_improver" model_name: "gpt-4" max_iterations: 2 improve_var: ["task"] head_meta_instruction: | Now you will help me generate a prompt which is used to generate a corresponding story according to the species of an animal which is [animal_species] and its character [animal_character]. I already have some prompt and its evaluation results : end_meta_instruction: | Give me a new prompt that is different from all pairs above, and has a evaluation value higher than any of above.   ","version":"Next","tagName":"h3"},{"title":"Custom Combination Improver Guide: improve​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#custom-combination-improver-guide-improve","content":" This module defines the base class for combination improvers. Combination improvers are responsible for improving the combination of experiments based on their experiment results.  ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#introduction-3","content":" Combination improvers play a pivotal role in the experimental framework by optimizing the combination of experiments based on their outcomes. By leveraging combination improvers, experiments can be fine-tuned to achieve better results. This guide will outline the process of creating a custom combination improver.  ","version":"Next","tagName":"h3"},{"title":"Overview of Base Combination Improver​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#overview-of-base-combination-improver","content":" The BaseCombinationImprover class provides the foundational structure for all combination improvers. It offers methods to:  Register new combination improvers. Fetch registered combination improvers. Retrieve their default configurations. The main responsibility of a combination improver is to improve the setup of experiments based on their results.  ","version":"Next","tagName":"h3"},{"title":"Implementing a Custom Combination Improver​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#implementing-a-custom-combination-improver","content":" To create a custom combination improver, one should inherit from the BaseCombinationImprover class and implement the improve abstract method:  class CustomCombinationImprover(BaseCombinationImprover): """ Custom combination improver to optimize the setup of experiments. """ def improve(self, experiment, config, evaluator, token_logger): """ Custom logic to improve the experiment based on its results. Args: experiment (Experiment): The experiment with its results. config (ExperimentConfig): The original experiment configuration. evaluator (Evaluator): A utility class to evaluate the ExperimentResult. token_logger (TokenLogger): Logs the token usage. Returns: ImproverOutput: The result of the improvement. """ # Custom logic for improvement goes here pass   ","version":"Next","tagName":"h3"},{"title":"Config​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#config","content":" custom_improvers: class: /path/to/custom_improver.CustomImprover config_cls: /path/to/custom_improver_config.CustomImproverConfig   To use it  improver: name: custom_improver   ","version":"Next","tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Combination Improver","url":"/YiValApi/docs/API-Reference/combination-improver#conclusion","content":" By following this guide, you have successfully created and registered a custom combination improver named CustomCombinationImprover within the experimental framework. This custom improver will allow you to optimize experiment combinations based on specific logic and criteria you define. As experiments evolve and grow in complexity, custom combination improvers like the one you've developed will become instrumental in achieving more refined and better results. ","version":"Next","tagName":"h3"},{"title":"Schema","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/schema","content":"","keywords":"","version":"Next"},{"title":"ExperimentConfig​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#experimentconfig","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#introduction","content":" The ExperimentConfig class offers a thorough configuration schema for defining and executing experiments. It encompasses a wide range of experiment components, such as dataset configurations, custom functions, evaluators, trainers, and more.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#class-definition","content":" Description​  The ExperimentConfig class outlines the configuration for an experiment, capturing both mandatory and optional parameters.  Attributes​  description(str): A brief description of the experiment. dataset(DatasetConfig): The configuration related to the dataset used in the experiment. custom_function(Optional[str]): A custom function for the experiment (if any). variations(Optional[List[WrapperConfig]]): A list of configurations for different variations or wrappers. selection_strategy(Optional[Dict[str, BaseConfig]]): Strategy for selecting specific configurations or variations. wrapper_configs(Optional[Dict[str, BaseWrapperConfig]]): Configuration for specific wrappers. combinations_to_run(Optional[List[Tuple[str, Any]]]): Specific combinations to execute during the experiment. evaluators(Optional[List[Union[EvaluatorConfig, ComparisonEvaluatorConfig, GlobalEvaluatorConfig]])]: Configuration for evaluators that assess the experiment's results. improver(Optional[BaseCombinationImproverConfig]): Configuration for the improver to enhance the combinations. trainer(Optional[BaseTrainerConfig]): Configuration for training models. output(Optional[OutputConfig]): Configuration detailing the experiment's output format and destination. human_rating_configs(Optional[List[HumanRatingConfig]]): Configuration for human raters evaluating the experiment. existing_experiment_path(Optional[str]): Path to a pre-existing experiment (if any). version(Optional[str]): The version of the experiment. output_parser(Optional[str]): Parser for the experiment's output. metadata(Optional[Dict[str, Any]]): Additional metadata related to the experiment. custom_reader, custom_combination_improver, custom_data_generators, custom_wrappers, custom_evaluators, custom_variation_generators, custom_selection_strategies, custom_improvers(all Optional[Dict[str, Dict[str, Any]]]): Custom configurations for various components of the experiment. Each custom attribute allows users to define specific configurations tailored to their requirements.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#example","content":" custom_function: yival.demo.qa.qa dataset: file_path: demo/data/yival_expected_results.csv reader: csv_reader source_type: dataset reader_config: expected_result_column: expected_result description: Configuration fo question answering with expected results. evaluators: - evaluator_type: individual matching_technique: includes metric_calculators: - method: AVERAGE name: string_expected_result variations: - name: qa variations: - instantiated_value: "" value: "" value_type: str variation_id: null - instantiated_value: "Think first, then make a decision. Some random thoughts:" value: "Think first, then make a decision. Some random thoughts:" value_type: str variation_id: null   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"InputData​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#inputdata","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#introduction-1","content":" The InputData class provides a structured representation of individual data samples used in an experiment. It captures essential attributes like the unique identifier, the actual content (input parameters), and the expected result.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#class-definition-1","content":" Description​  The InputData class represents data for a single example in an experiment. It organizes the data into a structured format, making it easier to process and evaluate within the experiment.  Attributes​  content(Dict[str, Any]): A dictionary that encapsulates all the necessary input parameters for a custom function or experiment. This could include features, parameters, or any other required data points. example_id(Optional[str]): A unique identifier for the individual data sample or example. This can be useful for tracking, referencing, or debugging purposes. The default value is None. expected_result(Optional[Any]): Represents the expected outcome or result corresponding to the given input. This can be useful for evaluation, comparison, or validation tasks. The default value is None.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#example-1","content":" # Sample data representation using InputData sample_data = InputData( content={ "feature_1": 5.7, "feature_2": 3.2, "feature_3": 4.1 }, example_id="sample_001", expected_result="Class_A" ) # In this example, `sample_data` represents an individual data point with three features. The expected result for this data sample is "Class_A", and it is uniquely identified by the ID "sample_001".   This documentation offers a comprehensive guide to the InputData class. Ensure you adapt the specified content, IDs, results, and other parameters to synchronize with your project's structure and requirements.  ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Schema","url":"/YiValApi/docs/API-Reference/schema#source-code-1","content":"","version":"Next","tagName":"h3"},{"title":"Reader","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/reader","content":"","keywords":"","version":"Next"},{"title":"BaseReader​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#basereader","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#introduction","content":" This module provides an abstract foundation for data readers. Data readers are responsible for reading data from various sources, and this module offers a base class to define and register new readers, retrieve existing ones, and fetch their configurations. The design encourages efficient parallel processing by reading data in chunks.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#example","content":" ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#source-code","content":" ","version":"Next","tagName":"h3"},{"title":"CSV Reader​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#csv-reader","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#introduction-1","content":" The CSVReader class offers a streamlined interface for reading datasets from CSV files. Built atop the BaseReader class, it provides extensive configuration options, ensures error handling, and facilitates reading data in chunks.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#class-definition-1","content":" CSVReader(BaseReader)​  Description​  ​ The CSVReader class, derived from BaseReader, is crafted to read datasets from CSV files.  Attributes​  config (CSVReaderConfig): A configuration object detailing the reader's parameters. default_config (CSVReaderConfig): A default configuration for the reader.  Methods​  __init__(self, config: CSVReaderConfig): Initializes the CSVReader with the provided configuration.Parameters: config (CSVReaderConfig): The configuration object designated for the CSVReader. read(self, path: str) -> Iterator[List[InputData]]: Reads the CSV file and yields chunks of InputData.Parameters: path (str): Path pointing to the CSV file. Returns: An iterator that successively yields lists of InputData.  Notes​  The read method inspects headers in the CSV file and issues an error if absent.Rows that lack data values are bypassed, but a log warning is recorded.If a column is earmarked for expected results, the method extracts those values.Data rows are consumed in chunks. When a chunk meets its specified size, it's yielded. The chunk size is determined by the chunk_size attribute in the CSVReaderConfig.The class enables registration with the BaseReader through the register_reader method.  Source Code​  CSVReaderConfig(BaseReaderConfig)​  Description​  ​ The configuration object is tailored specifically for the CSVReader class.  Attributes​  use_first_column_as_id (bool): A flag to determine if the first column should be used as an ID. The default value is False.expected_result_column (Optional[str]): Specifies the column name that contains expected results, if any. The default value is None.  Source Code​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#example-1","content":" Here is a step-by-step guide on how to use CSV Readergiven a CSV data  Sample CSV Data (my_dataset.csv)​  Suppose we have a dataset concerning sales data for different products:  ProductID,ProductName,Sales,ExpectedOutput 1001,WidgetA,500,High 1002,WidgetB,150,Low 1003,WidgetC,300,Medium 1004,WidgetD,,Low 1005,WidgetE,450,   In this CSV:  ProductID is a unique identifier for products.ProductName is the name of the product.Sales represent the number of units sold.ExpectedOutput is a categorical value indicating the sales volume (High, Medium, Low).  Using the CSVReader​  Given the configuration below:  config = CSVReaderConfig(use_first_column_as_id=True, expected_result_column="ExpectedOutput") csv_reader = CSVReader(config) csv_file_path = "./data/my_dataset.csv"   When we use the read method:  for chunk in csv_reader.read(csv_file_path): for data in chunk: print(data.content)   The output would look like:  {'ProductID': '1001', 'ProductName': 'WidgetA', 'Sales': '500'} {'ProductID': '1002', 'ProductName': 'WidgetB', 'Sales': '150'} {'ProductID': '1003', 'ProductName': 'WidgetC', 'Sales': '300'}   Note:  The row with ProductID 1004 is skipped because it has missing data in the Sales column.The row with ProductID 1005 is skipped because it lacks an ExpectedOutput.The ExpectedOutput column is not present in the content as it's marked for extraction.  Results & Handling​  The data extracted by the CSVReader will be in the form of InputData objects. Each object will have:  example_id: The unique identifier (from the ProductID column, as specified by the use_first_column_as_id flag). content: The actual content of the row (excluding the ExpectedOutput column). expected_result: The extracted expected result from the ExpectedOutput column. For the row with ProductID 1001, the InputData object will look like:  InputData( example_id='1001', content={'ProductID': '1001', 'ProductName': 'WidgetA', 'Sales': '500'}, expected_result='High' )   Using the CSVReader in YiVal config​   dataset: file_path: demo/data/yival_expected_results.csv reader: csv_reader source_type: dataset reader_config: expected_result_column: expected_result   ","version":"Next","tagName":"h3"},{"title":"HuggingFaceDatasetReader​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#huggingfacedatasetreader","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#introduction-2","content":" The HuggingFaceDatasetReader class provides an interface to read datasets directly from the HuggingFace Datasets server. It allows for fetching data, transforming its structure, and filtering based on inclusion and exclusion patterns.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#class-definition-2","content":" HuggingFaceDatasetReader(BaseReader)​  Description​  ​ The HuggingFaceDatasetReader class, derived from BaseReader, is designed to read datasets from HuggingFace's Datasets server.  Attributes​  config (HuggingFaceDatasetReaderConfig): Configuration object specifying the reader's parameters. default_config (HuggingFaceDatasetReaderConfig): Default configuration for the reader.  Methods​  __init__(self, config: HuggingFaceDatasetReaderConfig): Initializes the HuggingFaceDatasetReader with the provided configuration.Parameters: config (HuggingFaceDatasetReaderConfig): The configuration object for the reader. read(self, path: str) -> Iterator[List[InputData]]: Reads the dataset from the specified HuggingFace Datasets server's URL and yields lists of InputData.Parameters: path (str): URL pointing to the HuggingFace Datasets server. Returns: An iterator that produces lists of InputData.  HuggingFaceDatasetReaderConfig(BaseReaderConfig)​  Description​  ​ The configuration object is specific to the HuggingFaceDatasetReader class.  Attributes​  example_limit (int): The maximum number of examples to fetch from the dataset.The default value is 1. output_mapping (Dict[str, str]): A mapping to transform the keys in the dataset. The Dict key is the original dataset key, and the corresponding value is the new key.The default value is an empty dictionary or {}. include (List[str]): List of regex patterns. Only items matching these patterns will be included.The default value is an empty list or[]. exclude (List[str]): List of regex patterns. Items matching these patterns will be excluded.The default value is an empty list or [].  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#example-2","content":" Filtering Out Hard Leetcode Problems in HugginFace Dataset​  In the example below, the reader fetches data from the given HuggingFace Datasets server's URL, transforms the key "question" to "leetcode_problem", and filters out any entry labeled as "Hard".  # Assuming necessary imports are in place # Define the reader configuration config = HuggingFaceDatasetReaderConfig( chunk_size=1000, example_limit=100, output_mapping={'question': 'leetcode_problem'}, include=['^(?!.*# Hard).*$'] ) # Create an instance of HuggingFaceDatasetReader with the specified configuration reader = HuggingFaceDatasetReader(config) # Define the URL pointing to the HuggingFace Datasets server url = "https://datasets-server.huggingface.co/rows?dataset=BoyuanJackchen%2Fleetcode_free_questions_text&config=default&split=train" # Read and process data for data_chunk in reader.read(url): for data in data_chunk: print(data.content)   Using the HuggingFaceDatasetReader in YiVal config​   dataset: file_path: https://datasets-server.huggingface.co/rows?dataset=griffin%2Fchain_of_density&config=annotated&split=test reader: huggingface_dataset_reader source_type: dataset reader_config: example_limit: 2 output_mapping: article: article   ","version":"Next","tagName":"h3"},{"title":"Source Code​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#source-code-3","content":" ","version":"Next","tagName":"h3"},{"title":"Custom Reader Guide: TXTReader​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#custom-reader-guide-txtreader","content":" This guide provides the steps to create custom data readers by subclassing the provided BaseReader class. The example demonstrates creating a TXTReader to read .txt files.  ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#introduction-3","content":" Data readers are responsible for reading data from various sources. By subclassing the BaseReader, you can create custom readers tailored to your specific data format needs.  ","version":"Next","tagName":"h3"},{"title":"BaseReader Overview​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#basereader-overview","content":" The BaseReader class offers a blueprint for designing data readers. It has methods for:  Register new readers. Retrieve registered readers and their configurations. Read data in chunks. The class provides an abstract method read that you must override in your custom reader. The method is designed to read data in chunks for efficient parallel processing.  ","version":"Next","tagName":"h3"},{"title":"Creating a Custom Reader (TXTReader)​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#creating-a-custom-reader-txtreader","content":" Design the TXTReaderConfig Class​  Before creating the reader, design a configuration class specific to the TXTReader. This class will inherit from the base BaseReaderConfig:  from dataclasses import asdict, dataclass from yival.data.base_reader import BaseReaderConfig @dataclass class TXTReaderConfig(BaseReaderConfig): """ Configuration specific to the TXT reader. """ delimiter: str = "\\n" # Default delimiter for txt files. def asdict(self): return asdict(self)   Implement the TXTReader Class​  Now, create the TXTReader class, subclassing the BaseReader:  from typing import Iterator, List from txt_reader_config import TXTReaderConfig from yival.data.base_reader import BaseReader from yival.schemas.common_structures import InputData class TXTReader(BaseReader): """ TXTReader is a class derived from BaseReader to read datasets from TXT files. Attributes: config (TXTReaderConfig): Configuration object specifying reader parameters. Methods: __init__(self, config: TXTReaderConfig): Initializes the TXTReader with a given configuration. read(self, path: str) -> Iterator[List[InputData]]: Reads the TXT file and yields chunks of InputData. """ config: TXTReaderConfig default_config = TXTReaderConfig() def __init__(self, config: TXTReaderConfig): super().__init__(config) self.config = config def read(self, path: str) -> Iterator[List[InputData]]: chunk = [] chunk_size = self.config.chunk_size with open(path, mode="r", encoding="utf-8") as file: for line in file: line_content = line.strip().split(self.config.delimiter) # Each line in the TXT file is treated as a separate data point. example_id = self.generate_example_id({"content": line_content}, path) input_data_instance = InputData( example_id=example_id, content=line_content ) chunk.append(input_data_instance) if len(chunk) >= chunk_size: yield chunk chunk = [] if chunk: yield chunk   Config​  After defining the config and reader subclass, we can define the yml config file:  custom_reader: txt_reader: class: /path/to/text_reader.TXTReader config_cls: /path/to/txt_reader_config.TXTReaderConfig   dataset: source_type: dataset reader: txt_reader file_path: "/Users/taofeng/YiVal/data/headline_generation.txt" reader_config: delimiter: "\\n"   ","version":"Next","tagName":"h3"},{"title":"Conclusion​","type":1,"pageTitle":"Reader","url":"/YiValApi/docs/API-Reference/reader#conclusion","content":" Creating custom data readers with the provided framework is straightforward. You can design readers tailored to various data formats by simply subclassing the BaseReader and overriding its read method. With this capability, you can efficiently read data in chunks, making it suitable for parallel processing and large datasets. ","version":"Next","tagName":"h3"},{"title":"Finetune","type":0,"sectionRef":"#","url":"/YiValApi/docs/API-Reference/finetune","content":"","keywords":"","version":"Next"},{"title":"OpenAIFineTuneUtils​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#openaifinetuneutils","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#introduction","content":" This module is designed to fine-tune OpenAI's GPT model using a specified dataset and conditions. It provides a utility for extracting relevant results from an experiment, transforming this data into the desired format, and then leveraging OpenAI's API to perform fine-tuning.  It takes ExperimentResult dump (pkl), and filtered conditions to extract the result.  ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#class-definition","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#example","content":" def main(): finetune( 'test_demo_results.pkl', "name == openai_prompt_based_evaluator AND result >= 0 AND display_name == clarity", )   ","version":"Next","tagName":"h3"},{"title":"SFTTrainer​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#sfttrainer","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#introduction-1","content":" The SFTTrainer class is designed to finetune a pre-trainer model on a specific task. It's a part of the yival framework that allows for easy and efficient fine-tuning of models. The class is built on top of the Hugging Face Transformers library and provides a high-level, easy-to-use API for fine-tuning.  The SFTTrainer class supports fine-tuning with different configurations, including enabling bits and bytes for quantization and LoRA for low-rank adaptation. It also supports different training arguments such as batchsize、learning_rate and number of epochs.  We recommend two ways to use Yival's SFTTrainer.  Use various dataset generators built into Yival (including huggingface, openai_generator, etc.) for data upload or generation, and then finetune the modelProvide a custom_func, use advanced models like GPT-4 for data generation, and customize the selection criteria. The model is then finetuned based on the selected data.  ","version":"Next","tagName":"h3"},{"title":"DataSetConfig​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#datasetconfig","content":" Description​  Configuration class for the training arguments. It specifies various parameters including per device train batch size, gradient accumulation steps, gradient checkpointing, max grad norm, num train epochs, learning rate, bf16, save total limit, logging steps, optim, lr scheduler type, warmup ratio, and log level.  Attributes​  per_device_train_batch_size(int): specifies the batch size for training per device. gradient_accumulation_steps(int): specifies the number of steps to accumulate gradients before updating. gradient_checkpointing(bool): specifies whether to use gradient checkpointing to save memory. max_grad_norm(float): specifies the maximum norm of the gradients. num_train_epochs(int): specifies the number of training epochs. learning_rate(float): specifies the learning rate. bf16(bool): specifies whether to use bf16 precision for training. save_total_limit(int): specifies the total number of checkpoints to save. logging_steps(int): specifies the number of steps between logging. optim(str): specifies the optimizer to use for training. lr_scheduler_type(str): specifies the type of learning rate scheduler to use. warmup_ratio(float): specifies the warmup ratio for the learning rate scheduler. log_level(str): specifies the log level.  ","version":"Next","tagName":"h3"},{"title":"TrainArguments​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#trainarguments","content":" Description​  Configuration class for the dataset. It specifies various parameters including prompt key, completion key, formatting prompts format, and condition.  Attributes​  ","version":"Next","tagName":"h3"},{"title":"BnbConfig​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#bnbconfig","content":" Description​  Configuration class for the bits and bytes. It specifies whether to load in 4 bit or 8 bit.  Attributes​  load_in_4_bit(bool): specifies whether to load in 4 bit. load_in_8_bit(bool): specifies whether to load in 8 bit.  ","version":"Next","tagName":"h3"},{"title":"LoRAConfig​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#loraconfig","content":" Description​  Configuration class for the LoRA. It specifies various parameters including r, lora alpha, bias, task type, lora dropout, and inference mode.  Attributes​  r(int): specifies the rank for the LoRA. lora_alpha(int): specifies the alpha for the LoRA. bias(str): specifies the bias for the LoRA. task_type(str):specifies the task type for the LoRA.lora_dropout(float): specifies the dropout for the LoRA. inference_mode(bool): specifies whether to use inference mode for the LoRA.  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#example-1","content":" Dataset load and finetune model in yival config​  description: Generated experiment config dataset: data_generators: openai_prompt_data_generator: chunk_size: 100000 diversify: true # model_name specify the llm model , e.g. a16z-infra/llama-2-13b-chat:9dff94b1bed5af738655d4a7cbcdcde2bd503aa85c94334fe1f42af7f3dd5ee3 model_name: gpt-4 prompt: "Please provide a concrete and realistic test case as a dictionary for function invocation using the ** operator. Only include parameters, excluding description and name. Ensure it's succinct and well-structured. **Only provide the dictionary.**" input_function: description: The current function is to evaluate the English to Chinese translation ability of the large language model. You will play the role of a teacher, so please provide a coherent English sentence (teacher_quiz), and give the corresponding Chinese translation (teachaer_answer). name: translation_english_to_chinese parameters: teacher_quiz: str teacher_answer: str expected_param_name: teacher_answer number_of_examples: 5 output_path: english2chinese1.pkl source_type: machine_generated trainer: name: sft_trainer model_name: PY007/TinyLlama-1.1B-Chat-v0.3 output_path: output dataset_config: prompt_key: teacher_quiz enable_lora: False enable_bits_and_bytes: False   Custom_func then customize the selection criteria. The model is then finetuned based on the selected data in yival config  custom_function: demo.headline_generation_detail.headline_generation description: Generated experiment config dataset: data_generators: openai_prompt_data_generator: chunk_size: 100000 diversify: true # model_name specify the llm model , e.g. a16z-infra/llama-2-13b-chat:9dff94b1bed5af738655d4a7cbcdcde2bd503aa85c94334fe1f42af7f3dd5ee3 model_name: gpt-4 prompt: "Please provide a concrete and realistic test case as a dictionary for function invocation using the ** operator. Only include parameters, excluding description and name. Ensure it's succinct and well-structured. **Only provide the dictionary.**" input_function: description: "Given a tech startup business named [tech_startup_business], specializing in [business], and target_peopleing [target_people], generate a corresponding landing page headline." name: headline_generation_for_business parameters: tech_startup_business: str business: str target_people: str number_of_examples: 3 output_path: null source_type: machine_generated variations: - name: task variations: - instantiated_value: Generate landing page headline for {tech_startup_business}, company business is {business}, target_people is {target_people} value: Generate landing page headline for {tech_startup_business}, company business is {business}, target_people is {target_people} value_type: str variation_id: null evaluators: - evaluator_type: individual metric_calculators: - method: AVERAGE name: openai_prompt_based_evaluator display_name: clear prompt: |- You are assessing a submitted answer on a given task based on a criterion. Here is the data: - Task: Given an tech startup business, generate one corresponding landing page headline - Does the headline clearly communicate what the startup does or what problem it solves? It should be immediately clear to anyone who reads the headline what the startup's purpose is. A lack of clarity can lead to confusion and may discourage potential users or investors. [Input]: {tech_startup_business} [Result]: {raw_output} Answer the question by selecting one of the following options: A It fails to meet the criterion at all. B It somewhat meets the criterion, but there is significant room for improvement. C It meets the criterion to a satisfactory degree. D It meets the criterion very well. E It meets the criterion exceptionally well, with little to no room for improvement. choices: ["A", "B", "C", "D", "E"] # model_name specify the llm model , e.g. a16z-infra/llama-2-13b-chat:9dff94b1bed5af738655d4a7cbcdcde2bd503aa85c94334fe1f42af7f3dd5ee3 model_name: gpt-4 description: "evaluate the quality of the landing page headline" scale_description: "0-4" choice_scores: A: 0 B: 1 C: 2 D: 3 E: 4 selection_strategy: ahp_selection: criteria: - "openai_prompt_based_evaluator: clear" - average_token_usage - average_latency criteria_maximization: "openai_prompt_based_evaluator: clear": true average_latency: false average_token_usage: false criteria_weights: "openai_prompt_based_evaluator: clear": 0.6 average_latency: 0.2 average_token_usage: 0.2 normalize_func: "z-score" trainer: name: sft_trainer model_name: PY007/TinyLlama-1.1B-Chat-v0.3 output_path: output dataset_config: prompt_key: teacher_quiz condition: "name == openai_prompt_based_evaluator AND result >= 0 AND display_name == clear" enable_lora: False enable_bits_and_bytes: False   ","version":"Next","tagName":"h3"},{"title":"BackUpSFTTrainer​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#backupsfttrainer","content":" ","version":"Next","tagName":"h2"},{"title":"Introduction​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#introduction-2","content":" ","version":"Next","tagName":"h3"},{"title":"Class Definition​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#class-definition-1","content":" Description​  Attributes​  ","version":"Next","tagName":"h3"},{"title":"Example​","type":1,"pageTitle":"Finetune","url":"/YiValApi/docs/API-Reference/finetune#example-2","content":"","version":"Next","tagName":"h3"}],"options":{"id":"default"}}