Manual tracing for auto-instrumentation-enabled Lambda fuctions

[nicognaW]

I enabled the auto-instrumentation support by following the instructions on ADOT Lambda Support For JavaScript in my TypeScript Lambda functions. The auto-instrumentation is enabled successfully as follow:

However, when I manually create spans following instructions from here, the information seems not reported to the CloudWatch X-Ray.

My operations exactly

1. I added this layer to my Lambda function: arn:aws:lambda:us-west-2:901920570463:layer:aws-otel-nodejs-amd64-ver-1-15-0:2
2. I enabled the active tracing in my AWS Lambda function
3. I added some manual instrumentations, code:

   const sdk = new NodeSDK({
     resource: new Resource({
       [SemanticResourceAttributes.SERVICE_NAME]: 'echo-handler',
       [SemanticResourceAttributes.SERVICE_VERSION]: '1.0',
     }),
     traceExporter: new OTLPTraceExporter(),
     }),
   });
   
   sdk.start();
   const tracer = api.trace.getTracer('echo-handler-tracer');
   
   module.exports.handler = async (
     event: APIGatewayProxyEvent,
     context: Context,
     callback: Callback<APIGatewayProxyResult>,
   ): Promise<APIGatewayProxyResult> => {
     return tracer.startActiveSpan('echo API handler', async (span) => {
       ... some logics
       span.end();
       return {
         statusCode: 200,
         body: JSON.stringify({
           message: requestBody.message,
           tables: tables,
         }),
       };
     });
   };

Question

I'm unable to find any information about this, is manually instrumenting not supported when the auto-instrumentation is enabled?

[nicognaW]

This is the code I tried later, still doesn't work:

const _traceExporter = new OTLPTraceExporter();
const _spanProcessor = new BatchSpanProcessor(_traceExporter);
const sdk = new NodeSDK({
  textMapPropagator: new AWSXRayPropagator(),
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'echo-handler',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0',
  }),
  instrumentations: [
    new AwsLambdaInstrumentation({
      requestHook: (span, { event, context }) => {
        span.setAttribute('faas.name', context.functionName);
        span.setAttribute('event', event);
      },
      responseHook: (span, { err, res }) => {
        if (err instanceof Error) span.setAttribute('faas.error', err.message);
        if (res) span.setAttribute('faas.res', res);
      },
    }),
    new HttpInstrumentation(),
    new AwsInstrumentation({
      suppressInternalInstrumentation: true,
    }),
  ],
  traceExporter: _traceExporter,
  spanProcessor: _spanProcessor,
  idGenerator: new AWSXRayIdGenerator(),
});

sdk.start();

const tracer = api.trace.getTracer('echo-handler-tracer');

module.exports.handler = async (
  event: APIGatewayProxyEvent,
  context: Context,
  callback: Callback<APIGatewayProxyResult>,
): Promise<APIGatewayProxyResult> => {
  return tracer.startActiveSpan('echo API handler', async (span) => {
    ... some logics
    span.end();
    return {
      statusCode: 200,
      body: JSON.stringify({
        message: requestBody.message,
        tables: tables,
      }),
    };
  });
};

When I turn the activate trace option on, only traces provided by the layer (from my perspective) is exported to the X-Ray, and the traces from the code is not.

[nicognaW]

I took a log to check if the default URL is correct:

import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const _traceExporter = new OTLPTraceExporter({});
console.log(`using default OTLP TraceExporter URL: ${_traceExporter.getDefaultUrl({})}`);

The log:

From ADOT document

Also note that, the OTEL_EXPORTER_OTLP_ENDPOINT environment variable does not need to be set. The default value is http://localhost:4318, as the ADOT Node.JS lambda layer only supports OTLP over HTTP.

This seems to be right.

[ferdy-roz]

were you able to get this working? I am running into the same issue

[nicognaW]

were you able to get this working? I am running into the same issue

Are you also using the layer from the ADOT document?

[ferdy-roz]

were you able to get this working? I am running into the same issue

Are you also using the layer from the ADOT document?

Yes -- I was able to get manual instrumentation working for other backends like New Relic, so been working with that. Could not get xray to pick up any of my spans/attributes

[nicognaW]

I turned to use PowerTools instead of the ADOT layer FYI @ferdy-roz

[gshpychka]

Could it be because you're not awaiting your async callback?