Proposal: Introduce metrics reporting for lance

[yanghua]

As a table format, Lance plays a crucial role in data storage and management for AI data. A comprehensive metrics system is essential in monitoring table formats, diagnosing health status, and performance-oriented governance. This is fully reflected in Apache Iceberg.

Apache Iceberg introduced metrics reporting1. It exposes some metadata and statistical information of core table operations (such as Scan/Commit) in the form of metrics. This ensures that these core pieces of information are not hidden beneath the iceberg.

In terms of its uses, with the external exposure of these metrics, we can do many things.

• Monitoring: Includes some indicators related to load and performance. Third parties can provide some insightful dashboards based on these indicators 24;
• Diagnosis: Based on monitoring, we can provide some incremental value. One of them is the diagnosis of some performance issues, which will assist in optimizing layout or performance;
• Optimization: Based on the provided indicators, we can optimize performance at both the read and write ends.

[yanghua]

@eddyxu @westonpace @wjones127 Would like to listen to your thoughts and feedback.

[westonpace]

I think we're very much in favor of adding additional metrics. I had an initial start at this here; #3075 but I'm not sure if I'll get back to it soon. I was planning on adding some storage statistics soon as well.

I guess this issue itself is a little broad though. Is there a particular standard you are looking for? Is this something you are hoping to work on?

[yanghua]

Hi @westonpace Thanks for your feedback.

I guess this issue itself is a little broad though.

Yes, this proposal is on "idea" status currently. If the community gives positive feedback, I'd like to refine it.

Is there a particular standard you are looking for?

Actually, I am just looking for some thoughts and feedback or the status(if the community has kicked off).

I will take a look about #3075

Is this something you are hoping to work on?

Yes, I would like to give some help at least on two parts:

• introduce the framework for metrics reporting;
• enrich the statistics and metrics in some core actions on the Lance dataset, e.g. scan/take/commit..