chore(llmobs): use span store instead of temporary tags

[Yun-Kim]

This PR performs some cleanup refactors on the LLM Obs SDK and associated integrations. Specifically regarding the data stored, which includes LLMObs span metadata/metrics/tags/IO:

• Stop storing these as temporary span tags and instead use the span store field, which allows arbitrary key value pairs but is not submitted to Datadog. This removes the potential for temporary tags to be not extracted and still submitted as a APM span tag.
• Stop attempting safe_json() (i.e. json.dumps()) to store the above data, which is an expensive operation that adds up with the number of separate calls, and instead just store the raw values of the stored objects in the store field, and only call safe_json() "once" at payload encoding time.

Things to look out for:

• Previously we were calling safe_json() every time to store data as string span tags. One danger includes errors during span processing due to wrong types (expect string, likely receive a dictionary/object from the span store field)
• By avoiding any jsonify processing before encode time, a small edge case appeared from the LLMObs SDK decorator function which auto-annotates non-LLM spans with input function argument maps. In Python 3.8, the bind_partial().arguments call used to extract the function arguments returns an OrderedDict (otherwise returns a regular Dict() in Python >= 3.9, which broke some tests as we were simply casting to a string when storing the input/output value). I added a fix to cast the bind_partial().arguments object to a dict to avoid this issue coming up.

Next Steps

This is a great first step, but there are still tons of performance improvements we can make to our encoding/writing. The most notable is that we call json.dumps() on span events more than once (to calculate the payload size before adding to the buffer).

Checklist

• PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[github-actions]

CODEOWNERS have been resolved as:

tests/llmobs/test_llmobs_span_encoder.py                                @DataDog/ml-observability
ddtrace/llmobs/_integrations/anthropic.py                               @DataDog/ml-observability
ddtrace/llmobs/_integrations/bedrock.py                                 @DataDog/ml-observability
ddtrace/llmobs/_integrations/gemini.py                                  @DataDog/ml-observability
ddtrace/llmobs/_integrations/langchain.py                               @DataDog/ml-observability
ddtrace/llmobs/_integrations/openai.py                                  @DataDog/ml-observability
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_trace_processor.py                                      @DataDog/ml-observability
ddtrace/llmobs/_utils.py                                                @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
ddtrace/llmobs/decorators.py                                            @DataDog/ml-observability
tests/contrib/anthropic/test_anthropic_llmobs.py                        @DataDog/ml-observability
tests/contrib/langchain/test_langchain_llmobs.py                        @DataDog/ml-observability
tests/contrib/openai/test_openai_llmobs.py                              @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/test_llmobs_decorators.py                                  @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability
tests/llmobs/test_llmobs_span_agentless_writer.py                       @DataDog/ml-observability
tests/llmobs/test_llmobs_trace_processor.py                             @DataDog/ml-observability

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-11-27 22:45:12

Comparing candidate commit 8cf2317 in PR branch yunkim/llmobs-use-span-store with baseline commit b9573be in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 328 metrics, 2 unstable metrics.