TigerTune/notebooks/finetune_text_generation_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "551753b7-6cd2-4f81-aec0-da119e4705ad",
   "metadata": {},
   "source": [
    "# Finetune LLMs\n",
    "\n",
    "In this notebook, we show users how to finetune their own large language models on Google colab.\n",
    "\n",
    "We go through three main sections:\n",
    "1. Install the library\n",
    "2. Finetuning the model (using our `TextGenerationTransformersFinetuneEngine`)\n",
    "2. Run inference on the fine tuned model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5988a147",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "1. Clone the repository:\n",
    "    ```\n",
    "    git clone https://github.com/tigerrag/tiger.git\n",
    "    ``` \n",
    "    \n",
    "2. Upload it to Google Drive. Make sure tiger repository is in the root directory in Google Drive.\n",
    "\n",
    "3. You'll need to select CUDA GPU like \"A100\",\"T4\", etc on Google Colab. Based on our experiment, although it only use ~14G GPU RAM in A100, \"T4 GPU\" with 15GB GPU RAM option will not be sufficient."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62368cb8-a303-48b1-8429-5e3655abcc3b",
   "metadata": {},
   "source": [
    "## Run Finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe3eeb5",
   "metadata": {},
   "source": [
    "Fine tuned model is uploaded to Google Drive automatically, and stored in the directory called \"exp_finetune\". You can customize it by setting the model_output_path parameter.\n",
    "\n",
    "With the toy dataset, the entire finetuning will take ~2mins on A100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fcdb3f4",
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mFailed to start Jupyter in the environment 'Python 3.11.5 64-bit (/opt/homebrew/bin/python3)'. \n",
      "ImportError: cannot import name 'notebookapp' from 'notebook' (/opt/homebrew/lib/python3.11/site-packages/notebook/__init__.py) \n",
      "View Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "# Mount Google Drive.\n",
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "932f6e24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Point default path to TigerTune in Google Drive\n",
    "import sys\n",
    "sys.path.append('/content/drive/MyDrive/tiger/TigerTune')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c23ce52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "!pip install -q accelerate peft bitsandbytes transformers trl tensorflow pyyaml h5py scikit-plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d08066-5f00-48f1-b12a-e80bc193d4c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import TigerTune\n",
    "from tigertune.finetuning import TextGenerationTransformersFinetuneEngine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26ca3566",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify Training dataset. We use the toy_data by default. You can also use your own data.\n",
    "training_dataset = \"/content/drive/MyDrive/tiger/TigerTune/datasets/generation/toy_data_train.jsonl\"\n",
    "eval_dataset = \"/content/drive/MyDrive/tiger/TigerTune/datasets/generation/toy_data_evaluation.jsonl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "26625ab5-ddc9-4dbd-9936-39b69c6a7cdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an instance of TextGenerationTransformersFinetuneEngine. \n",
    "finetune_engine = TextGenerationTransformersFinetuneEngine(\n",
    "    training_dataset,\n",
    "    base_model_id=\"daryl149/llama-2-7b-chat-hf\",\n",
    "    eval_dataset=training_dataset,\n",
    "    model_output_path=\"/content/drive/MyDrive/exp_finetune\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c330778f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start fine tuning. \n",
    "finetune_engine.finetune()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "828dd6fe-9a8a-419b-8663-56d81ce73774",
   "metadata": {},
   "source": [
    "## Evaluate Finetuned Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a66b83-4cbb-4374-a632-0f1bb2b785ab",
   "metadata": {},
   "source": [
    "In this section, we evaluate the fine tuned model.\n",
    "\n",
    "We show that finetuning on instruction-based dataset significantly improve upon an opensource text generation model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a743160a",
   "metadata": {},
   "outputs": [],
   "source": [
    "finetune_engine.inference(\n",
    "        prompt=\"What is RAG?\")"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
  },
  "kernelspec": {
   "display_name": "llama_index_v2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}