v0.2.4

• Fixed Builder object is not callable error (#67). In this release, we have made an enhancement to the Backends class in the databricks/labs/lsql/backends.py file. The DatabricksSession.builder() method call in the __init__ method has been changed to DatabricksSession.builder. This update uses the builder attribute to create a new instance of DatabricksSession without calling it like a function. The sdk_config method is then used to configure the instance with the required settings. Finally, the getOrCreate method is utilized to obtain a SparkSession object, which is then passed as a parameter to the parent class constructor. This modification simplifies the code and eliminates the error caused by treating the builder attribute as a callable object. Software engineers may benefit from this change by having a more streamlined and error-free codebase when working with the open-source library.
• Prevent silencing of pylint (#65). In this release, we have introduced a new job, "no-lint-disabled", to the GitHub Actions workflow for the repository. This job runs on the latest Ubuntu version and checks out the codebase with a full history. It verifies that no new instances of code suppressing pylint checks have been added, by filtering the differences between the current branch and the main branch for new lines of code, and then checking if any of those new lines contain a pylint disable comment. If any such lines are found, the job will fail and print a message indicating the offending lines of code, thereby ensuring that the codebase maintains a consistent level of quality by not allowing linting checks to be bypassed.
• Updated _SparkBackend.fetch() to return iterator instead of list (#62). In this release, the fetch() method of the _SparkBackend class has been updated to return an iterator instead of a list, which can result in reduced memory usage and improved performance, as the results of the SQL query can now be processed one element at a time. A new exception has been introduced to wrap any exceptions that occur during query execution, providing better debugging and error handling capabilities. The test_runtime_backend_fetch() unit test has been updated to reflect this change, and users of the fetch() method should be aware that it now returns an iterator and must be consumed to obtain the desired data. Thorough testing is recommended to ensure that the updated method still meets the needs of the application.

Contributors: @nfx, @qziyuan, @bishwajit-db