Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Symbolize heap profiles #9964

Closed
erikgrinaker opened this issue Dec 2, 2024 · 0 comments · Fixed by #10153
Closed

Symbolize heap profiles #9964

erikgrinaker opened this issue Dec 2, 2024 · 0 comments · Fixed by #10153
Assignees
@erikgrinaker
Labels
a/observability Area: related to observability c/storage Component: storage

Comments

@erikgrinaker
Copy link
Contributor

erikgrinaker commented Dec 2, 2024 •
edited
Loading

In #9778, we added heap profiles based on Jemalloc and rust-jemalloc-pprof. However, these profiles are not symbolized -- this requires users to download the binary and symbolize the profiles themselves to do anything useful with it. It's also likely that Grafana Cloud Profiles requires symbolized profiles.

We should symbolize the pprof profile on the server-side before sending it to the client. This can be done using e.g. the Rust addr2line crate, and then (re-)serialize the pprof Protobuf profile. We should consider upstreaming this to rust-jemalloc-pprof, if they want it, which might avoid an extra de/serialization roundtrip.
@erikgrinaker erikgrinaker added a/observability Area: related to observability c/storage Component: storage labels Dec 2, 2024
@erikgrinaker erikgrinaker self-assigned this Dec 2, 2024
This was referenced Dec 2, 2024
Merged
Closed
@erikgrinaker erikgrinaker mentioned this issue Dec 15, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Dec 17, 2024
@erikgrinaker
utils: symbolize heap profiles (#10153)
a55853f 
## Problem

Jemalloc heap profiles aren't symbolized. This is inconvenient, and
doesn't work with Grafana Cloud Profiles.

Resolves #9964.

## Summary of changes

Symbolize the heap profiles in-process, and strip unnecessary cruft.

This uses about 100 MB additional memory to cache the DWARF information,
but I believe this is already the case with CPU profiles, which use the
same library for symbolization. With cached DWARF information, the
symbolization CPU overhead is negligible.

Example profiles:

*
[pageserver.pb.gz](https://github.com/user-attachments/files/18141395/pageserver.pb.gz)
*
[safekeeper.pb.gz](https://github.com/user-attachments/files/18141396/safekeeper.pb.gz)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@erikgrinaker erikgrinaker

Labels
a/observability Area: related to observability c/storage Component: storage
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

utils: symbolize heap profiles neondatabase/neon
1 participant
@erikgrinaker