Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] Abnormal memory usage (possibly http3/quic leak) #679

Open
3 tasks done
Jay-716 opened this issue Oct 18, 2024 · 3 comments
Open
3 tasks done

[Bug Report] Abnormal memory usage (possibly http3/quic leak) #679

Jay-716 opened this issue Oct 18, 2024 · 3 comments

Comments

@Jay-716
Copy link

Jay-716 commented Oct 18, 2024

Checks

  • I have searched the existing issues
  • I have read the documentation
  • Is it your first time sumbitting an issue

Current Behavior

dae 0.7.0-0.8.0 running 3 hours can take up to 2GiB or even more memory.

Expected Behavior

Memory usage should ideally be within ~1G.

Steps to Reproduce

  1. minimal reproducible config(still a little bit long, sorry): config.dae
  2. run dae for a while
  3. memory consumption increases (especially when checking nodes latency), and never drop down

Environment

  • Dae version: 0.8.0, v0.8.0, v0.7.1, v0.7.0 (those prefixed with 'v' is self-build tests on tagged versions)
  • OS: Arch Linux
  • Kernel: arch 6.6.56-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 10 Oct 2024 12:04:53 +0000 x86_64 GNU/Linux
  • Others: nil

Anything else?

I switch between 0.8.0 release, 0.7.0, 0.7.1 tag, not seeing prominent difference.
I know little about go and its debugging, but I manage to enable ppref and collect some samples.
I perf on heap and goroutine, and generate pngs with commands like go tool pprof -png http://localhost:{port}/debug/pprof/{goroutine or heap} > out.png. Here are the results:

Memory Usage Heap Goroutine
~500MiB heap-2024-10-18_09:56:03 goroutine-2024-10-18_09:55:53
~1.5GiB heap-2024-10-18_12:22:34 goroutine-2024-10-18_12:22:43

It is notable that the http3/quic client called by dns checker takes the most goroutines and memory.
I hope these images help. I will provide more related information ASAP if needed. Your work is highly appreciated!

@dae-prow
Copy link
Contributor

dae-prow bot commented Oct 18, 2024

Thanks for opening this issue!

@Jay-716
Copy link
Author

Jay-716 commented Oct 18, 2024

Right after I open this issue I immediately realize that I forget to add dip(223.5.5.5) -> direct routing rule. Sorry ;(
I am not sure if that is the key. Testing again...
If it is true, I will close this.

@Jay-716
Copy link
Author

Jay-716 commented Oct 18, 2024

Right after I open this issue I immediately realize that I forget to add dip(223.5.5.5) -> direct routing rule. Sorry ;( I am not sure if that is the key. Testing again... If it is true, I will close this.

No luck. Tried both dip(223.5.5.5) -> direct and dip(223.5.5.5) -> must_direct and got similar pperf result:

http3/quic client called by dns checker takes the most goroutines and memory


EDIT: Removing all hysteria2 nodes works. So this is likely a hysteria2 specific problem. I took a quick glance at relevant code, and found nothing but some snips like context.TODO(). Anyway, I have no idea where the problem is since I am not familiar with go. :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant