Update /generate to not split classes & functions across cells

[srdas]

Fixes #1111

/generate will sometimes split a class or a function across multiple code cells. After tracing the code in generate.py the error arises from unanticipated LLM behavior in the curation of the resultant Jupyter notebook in the creation of the outline object. This is best handled by post-processing the notebook to merge "hanging" code cells with the preceding code cell.

The error looks like this:

After the fix, the error is no longer present. See this example:

Here is another example of the rectified notebook generation:

if necessary to check the log, add print statements in lines 236, 240, 245 to the code as shown below.

The before and after versions of the logged notebook can be compared for lines with ***** CELL 1 ***** as shown here:

[dlqqq]

@srdas Awesome work! Love the creativity applied here. Few minor comments below.

[dlqqq]

@srdas One other question: with this change, are you able to run every notebook generated via /generate? It would be helpful to identify & document what issues still exist with /generate so that they can be fixed in the future.

[srdas]

During testing across several different /generate use cases, one more issue seemed to occur (though somewhat rarely) -- there are a few cells that contains markdown text but the cell_type is "code" instead. Therefore, added another sweep through the cells to detect these cases and flip the cell_type to markdown.

Additional code to handle this is a new function:

And a second post-processing pass through the notebook to fix cells that should be markdown:

By adding print statements, this was checked as shown below, the cell_type before and after the rectification is shown:

See also the corresponding fix in the notebook:

The last example:

Notebooks usually run from end to end in most cases. The two cases when they do not are unrelated to the /generate code. These are:

1. A missing package
2. The function in the package being used is deprecated and the generated code is still using the old version. This is simply the fact that the LLM is not up to date.

[dlqqq]

@srdas Awesome work, this is fantastic! I used this to generate 3 notebooks and they all ran well. By contrast, the existing implementation only generates a runnable notebook <50% of the time.

The only issue I noticed was that /generate doesn't add a %pip install ... command for 3P libraries, so sometimes you need to add a cell to install that.

Left some minor points of feedback below.

[dlqqq]

@srdas Awesome work, thank you!! ❤️

[dlqqq]

@meeseeksdev please backport to v3-dev