Unstructured data to structured data conversion via EXTRACT_COLUMN

[hershd23]

Added custom function for extracting columns from unstructured data
new file: ../evadb/functions/extract_columns.py

[hershd23]

@xzdandy I created a python notebook as well but it gets gitignored while the rest of the tutorial notebooks don't any idea?

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[hershd23]

Solves #1235

[hershd23]

@xzdandy @pchunduri6 moved to a "one-column-at-a-time" implementation as you recommended.

The notebook has the implementation

[hershd23]

For one column at a time I think this PR is ready for review @xzdandy @pchunduri6.

For the other changes discussed with either of you, I think it makes sense to take that up in a separate PR else this will bloat. Let me know what you think

[xzdandy]

Can we also add a long integration test for the function under https://github.com/georgia-tech-db/evadb/tree/staging/test/integration_tests/long/functions? We can skip the test in circle ci due to openai key, but I think it is good to have one.

It can either be end-to-end (i.e., SQL queries) or directly test the function class.

[hershd23]

Yes @xzdandy on it

[hershd23]

Also this is failing the linter check for a Colab Notebook. Can you point me towards information on how to add that

[xzdandy]

We need to skip the notebook test at

evadb/script/test/test.sh

Line 97 in c2457b2

PYTHONPATH=./ python -m pytest --durations=5 --nbmake --overwrite "./tutorials" --capture=sys --tb=short -v --log-level=WARNING --nbmake-timeout=3000 --ignore="tutorials/08-chatgpt.ipynb" --ignore="tutorials/14-food-review-tone-analysis-and-response.ipynb" --ignore="tutorials/15-AI-powered-join.ipynb" --ignore="tutorials/16-homesale-forecasting.ipynb" --ignore="tutorials/17-home-rental-prediction.ipynb" --ignore="tutorials/18-stable-diffusion.ipynb" --ignore="tutorials/19-employee-classification-prediction.ipynb"

due to open ai key

[xzdandy]

Also this is failing the linter check for a Colab Notebook. Can you point me towards information on how to add that

Remove the last empty cell.

[hershd23]

12-01-2023 17:31:12 [check_notebook_format:295] ERROR: ERROR: Notebook /Users/hershdhillon23/projects/evadb/script/formatting/../../tutorials/20-structured-data.ipynb does not contain correct Colab link -- update the link.

Do not have a collar link right now

[xzdandy]

12-01-2023 17:31:12 [check_notebook_format:295] ERROR: ERROR: Notebook /Users/hershdhillon23/projects/evadb/script/formatting/../../tutorials/20-structured-data.ipynb does not contain correct Colab link -- update the link.

Do not have a collar link right now

The current notebook actually does not work on the colab. I was trying to make it work yesterday and I think it needs several modifications. One fix can help is that can you add the EXTRACT_COLUMN to bootstrap functions in https://github.com/georgia-tech-db/evadb/blob/staging/evadb/functions/function_bootstrap_queries.py

[pchunduri6]

Should we perform this operation using ChatGPT directly or use something like pandasAI to write a function using LLM and then extract the column we need? Writing a function is much cheaper token cost-wise, but less robust.
@hershd23 @xzdandy Any thoughts?

[xzdandy]

Should we perform this operation using ChatGPT directly or use something like pandasAI to write a function using LLM and then extract the column we need? Writing a function is much cheaper token cost-wise, but less robust.
@hershd23 @xzdandy Any thoughts?

Hi @pchunduri6, I think it depends on the task. If the extract column is based on patterns, I think we can generate regex for saving the cost and improve efficiency. On the other hand, if the task is semantic based, we need to rely on the LLM to extract the information.