Erlang tracing as a source of OT traces

[javiermtorres]

Hi everyone! I was wondering if Erlang traces could be used as a source for OT traces. Both functionalities seem to overlap somewhat, but would also complement each other nicely. What would be your opinion? Thanks in advance.

[tsloughter]

There have been experiments with this. I'd like to see it as something you could enable at runtime to get additional trace data.

But they don't exactly overlap, like we can't use Erlang tracing to handle Otel traces completely because that would remove the ability of the user to use Erlang tracing for debugging purposes at runtime because there can be only one tracer.

We've had discussions with the OTP team on some ways to improve this and to do cross-process propagation automatically.

[javiermtorres]

Tracing management is somewhat fuzzy at this point, I'd agree on that :)

What did the OTP team think and/or propose? Thanks!!!!!

[tsloughter]

They have a number of tracing related items they'd like to get to, it is just a matter of prioritizing.

[bullno1]

I've been looking at this too and it would be great if tracing can just be flipped on automatically for interprocess spans.
Then one would only have to manually add intraprocess spans for better visibility.
Below is my research so far:

seq_trace looks good at first thanks to its ability to "infect" every process that has message sent to.
But it's not clear how to cleanup this token once the trace is done.
There is only seq_trace:reset_trace/0 which resets everything, including some potentially concurrent traces.
A seq_trace:reset_trace(Label) would be great.

erlang:trace is highly flexible with matchspec and dbg:fun2ms makes it a lot easier to write matchspecs.
I haven't tested it but it seems that matchspec can be used to simulate the "infectious" property of seq_trace with the enable_trace action.
For cleaning up, the matchspec could be setup so that it has specific "exit" trigger.
For example, returning from handle_call in a gen_server.
But the global nature of trace_pattern makes it annoying to extend and share.

"Userspace"/OTP level tracing seems to be the most promising and easy to implement so far.
Your application should mostly consist of special processes spawned with proc_lib and responds to sys.
sys:install can be used to attach a tracer.
There are 2 trace methods that I can think of:

• Attach as part of gen_server:start_link (server-side attachment).
  This makes the entire gen_server traceable by default.
  However, I'm not sure what's the overhead of calling erlang:process_info repeatedly to propagate context.

• Attach as part of a call (client-side attachment).
  So instead of using gen_server:call, you could so something like: otel_gen_server:call which is a wrapper.
  The wrapper would:

  1. Call sys:install to add a temporary tracer.
  2. Call gen_server:call.

  This is clean and the sys:install call can be omitted when recording is not enabled.
  The trade-off is that now we have an overhead of a gen:call in the recording case.
  It would be nice if there is a sys:install_async.
  AFAIK, all system messages have call semantic.

OTP tracing is manual and not magical but it seems clean.
I'll be implementing this in the next few days.

[bullno1]

That would be the cleanest.
But it would also require participation from the server.

I am trying to find a way to only require participation from the client.
It would allow me to trace some 3rd party applications as well.

[bryannaegele]

Some work was done by Vince several years ago.

There are a few notes in our WG minutes https://docs.google.com/document/d/18JVh6ICLyRCJBpRVIXwcR1fb-K4qRX3WHYgtM7mFx2U/edit#heading=h.150bie40noot. Search seq_trace

[bryannaegele]

erlef/observability-wg#2

[bryannaegele]

If I recall, the use case was to propagate context transparently, not auto-instrumentation. Otel isn't intended to produce auto-instrumentation stacktraces, so there's still a balance there, I believe. Trying to juggle traces within gen processes is also a place where dragons lie.

The biggest downside to seq_trace, I believe, was that you can only have one, so otel would be hijacking it. There was also something about needing OTP changes to get support for maps for some thing. I can't be sure. It was over 3 years ago.

[tsloughter]

@bullno1 it would only require participation from otel_gen_server.

[javiermtorres]

Hi everyone, has this aspect progressed in some way? Any contribution from the OTP people? Were there any insights derived from acd6811?
Thanks a lot in advance :)