cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ef518f1b-d68a-43ae-a812-5fd6c0c19bcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e587a75-929e-43f2-81c7-59655e10fb0b",
   "metadata": {},
   "source": [
    "# Data Cleaning\n",
    "\n",
    "Start by removing unneeded variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e360f88b-5eff-4661-a8df-c1247b2f241e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#load data and select columns\n",
    "df = pd.read_csv(\"./data/all_rawis.csv\").loc[\n",
    "    :, ['scholar_indx', 'name', 'grade', 'area_of_interest',\n",
    "        'tags', 'students_inds', 'teachers_inds']].rename(\n",
    "    columns={'grade':'generation'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68cf6a06-3b17-41a8-b86f-d6b736b43243",
   "metadata": {},
   "source": [
    "Some records have duplicated data under different scholar indices. Let's remove the duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b9d9ff8f-5b59-43f3-b805-c9e0b4598ba8",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop_duplicates(subset=['name','generation','area_of_interest','tags','students_inds','teachers_inds'],\n",
    "                  keep='first',ignore_index=True,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6903069f-a70e-40a2-bed0-74e224b3f44b",
   "metadata": {},
   "source": [
    "There are now 24247 records in the data.\n",
    "\n",
    "## Remove Records Without Teachers\n",
    "\n",
    "Because of the nature of hadith, every narrator should have at least one teacher except the Prophet, who has no teachers. Let's remove all records without teacher indices that also do not appear in any other scholars' student indices, except the Prophet. This will also have the effect of ensuring there are no isolated nodes. There are some records that have teacher names but not indices, these are few enough that we can ignore them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "970b00e2-3d9f-4393-825b-f30d912f7b73",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#doing this inside a function to keep variables local\n",
    "#return dataset after removing all records without teacher indices, except the Prophet\n",
    "def rm_noteachers(df):\n",
    "    prophet = df.loc[0,:].to_dict()#the prophet\n",
    "    \n",
    "    has_teacher=set()#all scholar_indx which are listed as students somewhere\n",
    "    for inds in df['students_inds']:\n",
    "        if isinstance(inds, list):\n",
    "            for ind in inds:\n",
    "                has_teacher.add(ind)\n",
    "\n",
    "    no_teachers_inds = list(df.loc[1:,'teachers_inds'].isna())#whether teachers_inds is NA by position\n",
    "    no_teachers = set()#all scholar_indx verified to have no teachers in the dataset\n",
    "    for i in range(len(no_teachers_inds)):\n",
    "        if no_teachers_inds[i]:\n",
    "            if df['scholar_indx'][i+1] in has_teacher:\n",
    "                pass\n",
    "            else:\n",
    "                no_teachers.add(df['scholar_indx'][1+i])\n",
    "\n",
    "    narrators = df.loc[1:,:].query('scholar_indx not in '+str(list(no_teachers)))   \n",
    "    \n",
    "    return narrators.append(\n",
    "        prophet,ignore_index=True).sort_values(\n",
    "        by='scholar_indx',axis='index').set_index('scholar_indx',drop=False)\n",
    "\n",
    "#modify the data\n",
    "df = rm_noteachers(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ae4d2e4-0771-46e3-af7d-f9148796c019",
   "metadata": {},
   "source": [
    "There are now 13061 records remaining.\n",
    "\n",
    "## Remove Untrustworthy Narrators\n",
    "\n",
    "The `area_of_interest` column contains some data on trustworthiness for ~5500 records, so let's use this to remove all records with a reputation of less than 'sahih'.\n",
    "\n",
    "First, let's extract all of the grades and add them to the dataset as a new column `grade`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1a93b557-ae0f-49d8-8e03-7730e6d4fe5d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#extract grades\n",
    "grades = []\n",
    "for item in df['area_of_interest'].str.findall(r\"\\[Grade:[^\\]]+\\]\").values:\n",
    "    if isinstance(item, list):\n",
    "        if len(item) > 0:\n",
    "            grade = item[0].lower().strip('[]').lstrip('grade').lstrip(':')\n",
    "            grades.append(grade)\n",
    "            continue\n",
    "    grades.append('undefined')\n",
    "\n",
    "#assign grade column to the df and remove area_of_interest column\n",
    "df = df.assign(grade=grades)\n",
    "del df['area_of_interest']\n",
    "del grades #remove unneeded variable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ad9545-09a2-48b3-9dbf-7f76828e1f96",
   "metadata": {},
   "source": [
    "Take a look at the unique grade values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "eec105a7-a1d1-4512-9f93-b8387b9afe50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['undefined', 'no doubt', 'thiqah thiqah', 'thiqah', 'maqbool',\n",
       "       'sadooq', 'weak', 'sadooq/delusion', 'not thiqah',\n",
       "       'unknown-majhool', 'abandoned', 'liar', 'accused liar'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['grade'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33b6f103-d9d0-4156-8a55-7a04610e00ca",
   "metadata": {},
   "source": [
    "See the data dictionary at muslimscholars.info for explanations of each grade.\n",
    "\n",
    "We are including:\n",
    "no doubt (Companions), thiqah thiqah (Awthaqun Nas), thiqah (Thiqat), sadooq (Saduq), sadooq/delusion (Saduq Yahim), maqbool (Maqbool/Layyin), not thiqah (Majhool al-haal/Mastur), and undefined\n",
    "\n",
    "We are excluding:\n",
    "abandoned (Matruk), accused liar (Muttaham bi'l kadhib), liar (Kadhdhaab, waddaa'), weak (Da'eef), and unknown-majhool (Majhool)\n",
    "\n",
    "Now let's remove all untrustworthy narrators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "771bff71-64c2-482d-8967-2988ff89c0ee",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#again working inside a function to keep variables local\n",
    "#return dataset after removing all untrustworthy narrators\n",
    "def rm_untrustworthy(df):\n",
    "    untrustworthy = [] #this will hold indices to drop\n",
    "    drop_grades = ['abandoned','liar','accused liar','weak','unknown-majhool'] #grades we are excluding\n",
    "    \n",
    "    #build list of indices of untrustworthy narrators\n",
    "    for indx, grade in df['grade'].items():\n",
    "        if (grade in drop_grades):\n",
    "            untrustworthy.append(indx)\n",
    "                \n",
    "    return df.drop(index=untrustworthy)\n",
    "    \n",
    "#modify the data\n",
    "df = rm_untrustworthy(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b51af8f6-22fc-451b-8b8e-3f5dea4f0bbd",
   "metadata": {},
   "source": [
    "There are now 12,429 records remaining."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2edea4e-24a0-4241-bc6f-c46d49b5a4aa",
   "metadata": {},
   "source": [
    "## Create Gender Variable\n",
    "\n",
    "Now let's clean up the `tags` column. We are only interested in the `Female` tag, so let's create a new column `gender` where records with a `Female` tag are encoded as `\"f\"` and records without a `Female` tag are encoded as `\"m\"`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "66e93f77-c2c9-40ff-b0c0-6212c2a3dc58",
   "metadata": {},
   "outputs": [],
   "source": [
    "#tags are stored as a single string\n",
    "#return \"f\" if \"female\" found in tags, otherwise \"m\"\n",
    "def get_gender(tag):\n",
    "    if pd.isna(tag):\n",
    "        return \"m\"\n",
    "    elif tag.lower().find('female') == -1:\n",
    "        return \"m\"\n",
    "    else:\n",
    "        return \"f\"\n",
    "\n",
    "#assign gender column to the df and remove tags column\n",
    "df = df.assign(gender=df['tags'].transform(get_gender))\n",
    "del df['tags']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bafbbca-62ea-476a-bd6b-5d2d61b39f96",
   "metadata": {},
   "source": [
    "It's possible that some female narrators were missing tags and have been misclassified. Since by convention women's names contain \"bint\" (i.e. \"daughter of\") while men's names contain \"bin\" (i.e. \"son of\") we can check to see if any scholars with \"bint\" in their names have been classified as men. Since names may contain multiple levels of ancestors, the presence of \"bint\" is necessary but not sufficient to conclude a record has been misclassified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ffb16957-f482-4b89-bf20-358167686403",
   "metadata": {},
   "outputs": [],
   "source": [
    "#detect names with 'bint' that are classified 'm'\n",
    "#generate array of indices with potentially misclassified gender\n",
    "flagged = []\n",
    "for indx, data in df.loc[:,['name','gender']].iterrows():\n",
    "    if (data[0].lower().find('bint') >= 0) & (data[1] == 'm'):\n",
    "        flagged.append(indx)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700b108a-f660-459a-94d2-238456edc2ff",
   "metadata": {},
   "source": [
    "Now let's look at the records flagged as misclassified. There are only 15, so we can manually double-check all of them. Some will not appear here because we have since altered them in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2e828176-e6ef-499c-839d-429a10dcb974",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scholar_indx</th>\n",
       "      <th>name</th>\n",
       "      <th>generation</th>\n",
       "      <th>students_inds</th>\n",
       "      <th>teachers_inds</th>\n",
       "      <th>grade</th>\n",
       "      <th>gender</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>scholar_indx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>186</th>\n",
       "      <td>186</td>\n",
       "      <td>Umm Kulthum bint 'Amr (Jarwal) ( أم كلثوم بنت ...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6222</th>\n",
       "      <td>6222</td>\n",
       "      <td>al-Swda'a bint 'Asim Lha Shbh ( السوداء ( رضي ...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6415</th>\n",
       "      <td>6415</td>\n",
       "      <td>al-Fry'h bint Wahub al-Zhryh Rf'ha ( الفريعة ب...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6607</th>\n",
       "      <td>6607</td>\n",
       "      <td>'Abdullah al-Bakry Rwt bint'h Bhyh ( عبد الله ...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6864</th>\n",
       "      <td>6864</td>\n",
       "      <td>Hqh bint 'Amr Slt al-Qbltyn ( حقة بنت عمرو صلت...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7729</th>\n",
       "      <td>7729</td>\n",
       "      <td>al-Tfyl bin Akhy Jwyryh bint al-Harith ( الطفي...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8367</th>\n",
       "      <td>8367</td>\n",
       "      <td>Khbab Mwla Fitimah bint 'Utba bin Rabi'y ( خبا...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8708</th>\n",
       "      <td>8708</td>\n",
       "      <td>Shrhbyl bin Habib Zwj al-Shfa'a bint ( شرحبيل ...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9179</th>\n",
       "      <td>9179</td>\n",
       "      <td>'Isa al-Msyh bin Maryam al-Sdyqh bint ( عيسى ا...</td>\n",
       "      <td>Comp.(RA)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18885</th>\n",
       "      <td>18885</td>\n",
       "      <td>Kinanah Client of Safiyya bint Huyayy كنانة مو...</td>\n",
       "      <td>Follower(Tabi') [3rd Generation]</td>\n",
       "      <td>20604, 24467, 20323, 20321, 28095</td>\n",
       "      <td>60, 4, 13, 11041</td>\n",
       "      <td>maqbool</td>\n",
       "      <td>m</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              scholar_indx                                               name  \\\n",
       "scholar_indx                                                                    \n",
       "186                    186  Umm Kulthum bint 'Amr (Jarwal) ( أم كلثوم بنت ...   \n",
       "6222                  6222  al-Swda'a bint 'Asim Lha Shbh ( السوداء ( رضي ...   \n",
       "6415                  6415  al-Fry'h bint Wahub al-Zhryh Rf'ha ( الفريعة ب...   \n",
       "6607                  6607  'Abdullah al-Bakry Rwt bint'h Bhyh ( عبد الله ...   \n",
       "6864                  6864  Hqh bint 'Amr Slt al-Qbltyn ( حقة بنت عمرو صلت...   \n",
       "7729                  7729  al-Tfyl bin Akhy Jwyryh bint al-Harith ( الطفي...   \n",
       "8367                  8367  Khbab Mwla Fitimah bint 'Utba bin Rabi'y ( خبا...   \n",
       "8708                  8708  Shrhbyl bin Habib Zwj al-Shfa'a bint ( شرحبيل ...   \n",
       "9179                  9179  'Isa al-Msyh bin Maryam al-Sdyqh bint ( عيسى ا...   \n",
       "18885                18885  Kinanah Client of Safiyya bint Huyayy كنانة مو...   \n",
       "\n",
       "                                    generation  \\\n",
       "scholar_indx                                     \n",
       "186                                  Comp.(RA)   \n",
       "6222                                 Comp.(RA)   \n",
       "6415                                 Comp.(RA)   \n",
       "6607                                 Comp.(RA)   \n",
       "6864                                 Comp.(RA)   \n",
       "7729                                 Comp.(RA)   \n",
       "8367                                 Comp.(RA)   \n",
       "8708                                 Comp.(RA)   \n",
       "9179                                 Comp.(RA)   \n",
       "18885         Follower(Tabi') [3rd Generation]   \n",
       "\n",
       "                                  students_inds     teachers_inds      grade  \\\n",
       "scholar_indx                                                                   \n",
       "186                                         NaN                 1  undefined   \n",
       "6222                                        NaN                 1  undefined   \n",
       "6415                                        NaN                 1  undefined   \n",
       "6607                                        NaN                 1  undefined   \n",
       "6864                                        NaN                 1  undefined   \n",
       "7729                                        NaN                 1  undefined   \n",
       "8367                                        NaN                 1  undefined   \n",
       "8708                                        NaN                 1  undefined   \n",
       "9179                                        NaN                 1  undefined   \n",
       "18885         20604, 24467, 20323, 20321, 28095  60, 4, 13, 11041    maqbool   \n",
       "\n",
       "             gender  \n",
       "scholar_indx         \n",
       "186               m  \n",
       "6222              m  \n",
       "6415              m  \n",
       "6607              m  \n",
       "6864              m  \n",
       "7729              m  \n",
       "8367              m  \n",
       "8708              m  \n",
       "9179              m  \n",
       "18885             m  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.filter(items=flagged,axis='index')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "681b9a22-9840-4dc7-b723-bff6f763eedb",
   "metadata": {},
   "source": [
    "Notes: \n",
    "\n",
    "- 641's name starts with a \"bint\" but their spouse is listed as 'Iqrab bint Mua'dh on muslimscholars.info. Lesbian companion to the prophet???\n",
    "    - No, 641 is a male; this is a translation error. The correct translation of the Arabic name should be \"Qays bin al-Khutaym al-Ansari\".\n",
    "\n",
    "- The latter half of 7729's name does correspond to one of the prophet's wives but I cant tell what's up with the first half.\n",
    "    - The correct translation of the Arabic name would be \"al-Tufayl son of the brother of Juwayriyyah bint al-Harith\". So he was male. \n",
    "\n",
    "- 9179 seems to be Jesus but why would Jesus be listed as a companion?\n",
    "    - That's so weird??? Checking his profile on https://muslimscholars.info/manage.php?submit=scholar&ID=9179, yes it is Jesus - even the Arabic biography on the page matches up. I have no idea why he's listed as a companion, nor why it says his \"place of stay\" is the Hijaz? This entry should be deleted.\n",
    "\n",
    "- On muslimscholars.info 18885 is identified as \"Kinanah (bin Nabiyya), Client of Safiyya bint Huyayy\" so I'm assuming male classification is correct.\n",
    "    - Yes, I agree - correct male classification. \n",
    "\n",
    "Typos:\n",
    "- 641: Qays bint al-Khutaym should be translated as Qays bin al-Khutaym. In fact, every time his name is mentioned in English it is spelled with the \"bint\" typo - this includes the instances where his children's names are listed. So I corrected all the instances of this in the all_rawis.csv file.\n",
    "- 3046: It shouldn't be \"Qutila bint Qays.\" Qutila bint Qays was indeed a female, but #3046 is a male and his name, correctly translated from Arabic, should be Qays bin al-Makshuh al-Muradi. I corrected this in the csv file.\n",
    "-3117: Should be \"Mu'awiya bin Suwayd al-Mazni\" (not bint). Corrected in file.\n",
    "\n",
    "Manually checking the flagged records identified the following records as misclassified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5a93a1b0-d630-489b-b9f0-42a70b00c20f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#indices of misclassified records - individuals who were listed as male but are in fact female. \n",
    "misclassified = [186, 6222, 6415, 6864]\n",
    "\n",
    "#generate corrected gender column\n",
    "corrected_gender = []\n",
    "for indx, gender in df['gender'].items():\n",
    "    if indx in misclassified:\n",
    "        corrected_gender.append('f')\n",
    "    else:\n",
    "        corrected_gender.append(gender)\n",
    "\n",
    "#remove old gender column and assign new gender column to the df\n",
    "del df['gender']\n",
    "df = df.assign(gender=corrected_gender)\n",
    "del flagged #remove unneeded variables\n",
    "del misclassified\n",
    "del corrected_gender"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b0a936-6145-4aec-a0d8-c23ad4fa5cd7",
   "metadata": {},
   "source": [
    "Now that we're comfortable with our gender classifications, let's check the gender ratio in the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d1223ad0-157f-4205-b0ae-0e350fb2ca3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "m    11225\n",
       "f     1204\n",
       "Name: gender, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['gender'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23f80f00-e0c1-443b-8442-9b615f8c694a",
   "metadata": {},
   "source": [
    "## Clean Up Edgelists\n",
    "\n",
    "Connections for each scholar are stored as strings containing lists of their students' and teachers' numeric indices. Let's turn these into lists of numeric indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "da557143-e7e2-4032-a424-dc77c4d16765",
   "metadata": {},
   "outputs": [],
   "source": [
    "#build corrected columns for students_inds and teachers_inds\n",
    "#strings of lists of numbers ---> lists of numeric indices\n",
    "students_inds_corrected = []\n",
    "teachers_inds_corrected = []\n",
    "for indx, data in df.loc[:,['students_inds','teachers_inds']].iterrows():\n",
    "    \n",
    "    students = data[0]\n",
    "    if pd.isna(students):\n",
    "        students_inds_corrected.append(students)\n",
    "    elif isinstance(students, str):\n",
    "        students_temp = []\n",
    "        for item in students.split(','):\n",
    "            if item.strip().isdigit():\n",
    "                students_temp.append(int(item.strip()))\n",
    "            else:\n",
    "                print(\"non-numeric in students_inds at id=\"+str(indx)+\", value: \"+item.strip())\n",
    "        students_inds_corrected.append(students_temp)\n",
    "    else:\n",
    "        raise TypeError(\"students_inds value at indx \"+str(indx)+\" is neither str nor NaN\")\n",
    "    \n",
    "    teachers = data[1]\n",
    "    if pd.isna(teachers):\n",
    "        teachers_inds_corrected.append(teachers)\n",
    "    elif isinstance(teachers, str):\n",
    "        teachers_temp = []\n",
    "        for item in teachers.split(','):\n",
    "            if item.strip().isdigit():\n",
    "                teachers_temp.append(int(item.strip()))\n",
    "            else:\n",
    "                print(\"non-numeric in teachers_inds at id=\"+str(indx)+\", value: \"+item.strip())\n",
    "        teachers_inds_corrected.append(teachers_temp)\n",
    "    else:\n",
    "        raise TypeError(\"teachers_inds value at indx \"+str(indx)+\" is neither str nor NaN\")\n",
    "\n",
    "#remove old columns and assign corrected columns to the dataset\n",
    "del df['students_inds']\n",
    "del df['teachers_inds']\n",
    "df = df.assign(students_inds=students_inds_corrected, teachers_inds=teachers_inds_corrected)\n",
    "del students_inds_corrected #remove unneeded variables\n",
    "del teachers_inds_corrected"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5395bfb-5e76-4e96-a07b-84efa13dddef",
   "metadata": {},
   "source": [
    "## Export Data\n",
    "\n",
    "Let's look at the cleaned dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "798dcf02-c8cf-4ac2-a14c-b50de0b8d9f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scholar_indx</th>\n",
       "      <th>name</th>\n",
       "      <th>generation</th>\n",
       "      <th>grade</th>\n",
       "      <th>gender</th>\n",
       "      <th>students_inds</th>\n",
       "      <th>teachers_inds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>scholar_indx</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Prophet Muhammad(saw) ( محمّد صلّی اللہ علیہ و...</td>\n",
       "      <td>Rasool Allah</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "      <td>[2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19, 5...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Abu Bakr As-Siddique ( أبو بكر الصديق ( رضي ال...</td>\n",
       "      <td>Comp.(RA) [1st Generation]</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "      <td>[3, 4, 5, 8, 49, 53, 107, 168, 17, 106, 18, 29...</td>\n",
       "      <td>[1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>'Umar ibn al-Khattab ( عمر بن الخطاب بن نفيل (...</td>\n",
       "      <td>Comp.(RA) [1st Generation]</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "      <td>[54, 18, 563, 4, 5, 6, 9, 8, 39, 16, 27, 28, 4...</td>\n",
       "      <td>[1, 2]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>'Uthman ibn 'Affaan ( عثمان بن عفان ( رضي الله...</td>\n",
       "      <td>Comp.(RA) [1st Generation]</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "      <td>[10582, 10587, 16, 49, 123, 391, 13, 19, 16, 1...</td>\n",
       "      <td>[1, 2, 3]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>Ali ibn Abi Talib ( علي بن أبي طالب بن عبد الم...</td>\n",
       "      <td>Comp.(RA) [1st Generation]</td>\n",
       "      <td>undefined</td>\n",
       "      <td>m</td>\n",
       "      <td>[30, 31, 16, 400, 13, 38, 182, 438, 17, 18, 10...</td>\n",
       "      <td>[1, 2, 3, 63, 163]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38948</th>\n",
       "      <td>38948</td>\n",
       "      <td>Yazid bin S'aid al-Sabahi يزيد بن سعيد الصباحي</td>\n",
       "      <td>3rd Century AH</td>\n",
       "      <td>thiqah</td>\n",
       "      <td>m</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[20001, 20185]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38992</th>\n",
       "      <td>38992</td>\n",
       "      <td>'Abdul Qahir bin Rashid bin Sa'd عبد القاهر بن...</td>\n",
       "      <td>3rd Century AH</td>\n",
       "      <td>thiqah</td>\n",
       "      <td>m</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[26012]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38996</th>\n",
       "      <td>38996</td>\n",
       "      <td>Muslim bin Yazid bin Madhkur مسلم بن يزيد بن م...</td>\n",
       "      <td>3rd Century AH</td>\n",
       "      <td>thiqah</td>\n",
       "      <td>m</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[17256]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39104</th>\n",
       "      <td>39104</td>\n",
       "      <td>Ibrahim bin Musa al-Maktab إبراهيم بن موسى المكتب</td>\n",
       "      <td>3rd Century AH</td>\n",
       "      <td>thiqah</td>\n",
       "      <td>m</td>\n",
       "      <td>[38719]</td>\n",
       "      <td>[28578]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39144</th>\n",
       "      <td>39144</td>\n",
       "      <td>Isma'il bin Ishaq bin Isma'il إسماعيل بن إسحاق</td>\n",
       "      <td>3rd Century AH</td>\n",
       "      <td>thiqah</td>\n",
       "      <td>m</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[20164, 30007]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12429 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              scholar_indx                                               name  \\\n",
       "scholar_indx                                                                    \n",
       "1                        1  Prophet Muhammad(saw) ( محمّد صلّی اللہ علیہ و...   \n",
       "2                        2  Abu Bakr As-Siddique ( أبو بكر الصديق ( رضي ال...   \n",
       "3                        3  'Umar ibn al-Khattab ( عمر بن الخطاب بن نفيل (...   \n",
       "4                        4  'Uthman ibn 'Affaan ( عثمان بن عفان ( رضي الله...   \n",
       "5                        5  Ali ibn Abi Talib ( علي بن أبي طالب بن عبد الم...   \n",
       "...                    ...                                                ...   \n",
       "38948                38948     Yazid bin S'aid al-Sabahi يزيد بن سعيد الصباحي   \n",
       "38992                38992  'Abdul Qahir bin Rashid bin Sa'd عبد القاهر بن...   \n",
       "38996                38996  Muslim bin Yazid bin Madhkur مسلم بن يزيد بن م...   \n",
       "39104                39104  Ibrahim bin Musa al-Maktab إبراهيم بن موسى المكتب   \n",
       "39144                39144     Isma'il bin Ishaq bin Isma'il إسماعيل بن إسحاق   \n",
       "\n",
       "                              generation      grade gender  \\\n",
       "scholar_indx                                                 \n",
       "1                           Rasool Allah  undefined      m   \n",
       "2             Comp.(RA) [1st Generation]  undefined      m   \n",
       "3             Comp.(RA) [1st Generation]  undefined      m   \n",
       "4             Comp.(RA) [1st Generation]  undefined      m   \n",
       "5             Comp.(RA) [1st Generation]  undefined      m   \n",
       "...                                  ...        ...    ...   \n",
       "38948                     3rd Century AH     thiqah      m   \n",
       "38992                     3rd Century AH     thiqah      m   \n",
       "38996                     3rd Century AH     thiqah      m   \n",
       "39104                     3rd Century AH     thiqah      m   \n",
       "39144                     3rd Century AH     thiqah      m   \n",
       "\n",
       "                                                  students_inds  \\\n",
       "scholar_indx                                                      \n",
       "1             [2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19, 5...   \n",
       "2             [3, 4, 5, 8, 49, 53, 107, 168, 17, 106, 18, 29...   \n",
       "3             [54, 18, 563, 4, 5, 6, 9, 8, 39, 16, 27, 28, 4...   \n",
       "4             [10582, 10587, 16, 49, 123, 391, 13, 19, 16, 1...   \n",
       "5             [30, 31, 16, 400, 13, 38, 182, 438, 17, 18, 10...   \n",
       "...                                                         ...   \n",
       "38948                                                       NaN   \n",
       "38992                                                       NaN   \n",
       "38996                                                       NaN   \n",
       "39104                                                   [38719]   \n",
       "39144                                                       NaN   \n",
       "\n",
       "                   teachers_inds  \n",
       "scholar_indx                      \n",
       "1                            NaN  \n",
       "2                            [1]  \n",
       "3                         [1, 2]  \n",
       "4                      [1, 2, 3]  \n",
       "5             [1, 2, 3, 63, 163]  \n",
       "...                          ...  \n",
       "38948             [20001, 20185]  \n",
       "38992                    [26012]  \n",
       "38996                    [17256]  \n",
       "39104                    [28578]  \n",
       "39144             [20164, 30007]  \n",
       "\n",
       "[12429 rows x 7 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e38d423-db1f-48bc-a28f-28dccd5caf6f",
   "metadata": {},
   "source": [
    "Save dataset to pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f8ff0bd1-d0a6-42b4-94fe-704bc3305a7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle('./data/cleaned_rawis.pkl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5e0a359",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "76bf5271334c370c1bfa97a17722864aaf327a0c81c7c096d2561442ed8fdfd4"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc-autonumbering": true,
  "toc-showmarkdowntxt": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}