usage/create_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5fe86434-1860-43f6-8be5-04fc1a89d3e5",
   "metadata": {},
   "source": [
    "# Create a pipeline\n",
    "\n",
    "This notebook demonstrates how to use the SDK to create pipelines.\n",
    "\n",
    "We begin by creating a client with our Personal Access Token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-02T02:31:14.025232Z",
     "start_time": "2024-10-02T02:31:14.020850Z"
    }
   },
   "outputs": [],
   "source": [
    "from glassflow import GlassFlowClient\n",
    "\n",
    "# Please edit this variable with your own personal access token from https://app.glassflow.dev/profile\n",
    "personal_access_token = \"< your-personal-access-token >\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "17837ba9391bbb22",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-02T02:31:42.279492Z",
     "start_time": "2024-10-02T02:31:42.276575Z"
    }
   },
   "outputs": [],
   "source": [
    "client = GlassFlowClient(\n",
    "    personal_access_token=personal_access_token\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f924d64-bc85-4139-b151-ec2b10c238d7",
   "metadata": {},
   "source": [
    "The next step is to create a transformation function to run on the pipeline. Let's create a simple transformation that prints the event data to the logs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cb6e02de-e3d3-4ea3-9883-6e60ecac2949",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing transformation function ...\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "echo \"Writing transformation function ...\"\n",
    "cat <<EOT > transformation.py \n",
    "def handler(data, log):\n",
    "    log.info(data)\n",
    "    return data\n",
    "EOT\n",
    "echo \"Done!\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3929267-2b7a-4d88-a43b-ef244c4ba57c",
   "metadata": {},
   "source": [
    "In GlassFlow, pipelines need to be created inside a space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0555b8a5-4a6a-427f-8165-29fcd018b43d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Space ID: d9b1c8df-b95a-4ddf-9731-e27f34c9a0f7\n"
     ]
    }
   ],
   "source": [
    "space = client.create_space(name=\"tutorials\")\n",
    "print(\"Space ID:\", space.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04161e0e-774a-44d8-b806-6b5832c37a0c",
   "metadata": {},
   "source": [
    "Now, we have everything we need to create a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e47dbfd-7b31-4f2a-af4d-428da4db5f2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline ID: bf90e3b8-a0d4-4a8b-819f-710357b5dc1e\n"
     ]
    }
   ],
   "source": [
    "pipeline = client.create_pipeline(\n",
    "    name=\"example pipeline\",\n",
    "    space_id=space.id,\n",
    "    transformation_file=\"transformation.py\",\n",
    ")\n",
    "print(\"Pipeline ID:\", pipeline.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fa208a0f-4e60-4816-aa56-d2cf7163d7e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "def handler(data, log):\n",
       "    log.info(data)\n",
       "    return data\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Markdown as md\n",
    "\n",
    "md(f\"```python\\n{pipeline.transformation_code}\\n```\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b4db9ac-7249-4978-9805-d40852719b8a",
   "metadata": {},
   "source": [
    "If we look at the pipeline logs, we can see that the pipeline was created successfully! 🚀\n",
    "\n",
    "(Logs might take a few seconds to appear, so you may need to run this cell a few times for the logs to show up.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2b56b716-790a-4def-a508-58b5132f11ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"level\": \"INFO\", \"severity_code\": 200, \"timestamp\": \"2024-10-02T05:42:12.252989163Z\", \"payload\": {\"message\": \"Function is uploaded\"}}\n",
      "{\"level\": \"INFO\", \"severity_code\": 200, \"timestamp\": \"2024-10-02T05:42:12.161987721Z\", \"payload\": {\"message\": \"Pipeline is created\"}}\n"
     ]
    }
   ],
   "source": [
    "logs_res = pipeline.get_logs()\n",
    "for log in logs_res.logs:\n",
    "    print(log.to_json())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a8e93c4-9117-40c8-a4d9-fe5913537ca4",
   "metadata": {},
   "source": [
    "### Transformation Requirements\n",
    "\n",
    "The previous example demonstrates a basic pipeline. In some cases, we might need to include additional dependencies for our transformation code. We can pass a requirements.txt file to the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "536f1205-cad5-4ae7-86fb-d490b6c26c57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline ID: c439ede3-c259-446d-8ad7-0c81b7a29b5c\n"
     ]
    }
   ],
   "source": [
    "pipeline_with_requirements = client.create_pipeline(\n",
    "    name=\"example pipeline with requirements\",\n",
    "    transformation_file=\"transformation.py\",\n",
    "    space_id=space.id,\n",
    "    requirements=\"requests==2.32.3\"\n",
    ")\n",
    "print(\"Pipeline ID:\", pipeline_with_requirements.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c22e5a6-93fe-4e6a-85c6-91a1cfca5a86",
   "metadata": {},
   "source": [
    "### Environment Variables\n",
    "\n",
    "Pipelines support environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "01b743a2-5dd7-4ef1-a467-4980c283a0cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline ID: 02b93a9f-e769-4114-8f95-61c8cfcd0d42\n"
     ]
    }
   ],
   "source": [
    "pipeline_with_env_vars = client.create_pipeline(\n",
    "    name=\"example pipeline with requirements\",\n",
    "    transformation_file=\"transformation.py\",\n",
    "    space_id=space.id,\n",
    "    env_vars=[\n",
    "        {\n",
    "            \"name\": \"MY_VAR\",\n",
    "            \"value\": \"example value\"\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "print(\"Pipeline ID:\", pipeline_with_env_vars.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dadba156-44af-40ea-9ff5-531d877bc5e2",
   "metadata": {},
   "source": [
    "## Clean-up\n",
    "\n",
    "Delete the pipelines and spaces we created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f91a05cd-fba7-44dd-a49e-711a8513b726",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.delete()\n",
    "pipeline_with_env_vars.delete()\n",
    "pipeline_with_requirements.delete()\n",
    "space.delete()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}