From 215d5f84d3e4b564014e9609ca8c8c7757db1398 Mon Sep 17 00:00:00 2001 From: kruskall <99559985+kruskall@users.noreply.github.com> Date: Wed, 12 Jun 2024 21:04:38 +0200 Subject: [PATCH] fix: replace ndjson.org with ndjson spec (#3997) the original website expired and it's currently serving malicious content. (cherry picked from commit 698d4e000c2d3d0f0f409ed931125e3aad2eb27a) # Conflicts: # docs/en/serverless/apm/apm-server-api/api-events.mdx --- docs/en/apm-server/api-events.asciidoc | 2 +- .../apm/apm-server-api/api-events.mdx | 138 ++++++++++++++++++ 2 files changed, 139 insertions(+), 1 deletion(-) create mode 100644 docs/en/serverless/apm/apm-server-api/api-events.mdx diff --git a/docs/en/apm-server/api-events.asciidoc b/docs/en/apm-server/api-events.asciidoc index 8ebd478e89..1eeb032024 100644 --- a/docs/en/apm-server/api-events.asciidoc +++ b/docs/en/apm-server/api-events.asciidoc @@ -13,7 +13,7 @@ Events can be: * Metrics Each event is sent as its own line in the HTTP request body. -This is known as http://ndjson.org[newline delimited JSON (NDJSON)]. +This is known as https://github.com/ndjson/ndjson-spec[newline delimited JSON (NDJSON)]. With NDJSON, agents can open an HTTP POST request and use chunked encoding to stream events to the APM Server as soon as they are recorded in the agent. diff --git a/docs/en/serverless/apm/apm-server-api/api-events.mdx b/docs/en/serverless/apm/apm-server-api/api-events.mdx new file mode 100644 index 0000000000..2bc1a53d92 --- /dev/null +++ b/docs/en/serverless/apm/apm-server-api/api-events.mdx @@ -0,0 +1,138 @@ + + +
+ + +Most users do not need to interact directly with the events intake API. + + +The events intake API is what we call the internal protocol that APM agents use to talk to the managed intake service. +Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. +Events can be: + +* Transactions +* Spans +* Errors +* Metrics + +Each event is sent as its own line in the HTTP request body. +This is known as [newline delimited JSON (NDJSON)](https://github.com/ndjson/ndjson-spec). + +With NDJSON, agents can open an HTTP POST request and use chunked encoding to stream events to the managed intake service +as soon as they are recorded in the agent. +This makes it simple for agents to serialize each event to a stream of newline delimited JSON. +The managed intake service also treats the HTTP body as a compressed stream and thus reads and handles each event independently. + +Refer to to learn more about the different types of events. + +
+ +### Endpoints + +The managed intake service exposes the following endpoints for Elastic APM agent data intake: + +| Name | Endpoint | +|---|---| +| APM agent event intake | `/intake/v2/events` | + +{/* | RUM event intake (v2) | `/intake/v2/rum/events` | +| RUM event intake (v3) | `/intake/v3/rum/events` | */} + +
+ +### Request + +Send an `HTTP POST` request to the managed intake service `intake/v2/events` endpoint: + +```bash +https://{hostname}:{port}/intake/v2/events +``` + +The managed intake service supports asynchronous processing of batches. +To request asynchronous processing the `async` query parameter can be set in the POST request +to the `intake/v2/events` endpoint: + +```bash +https://{hostname}:{port}/intake/v2/events?async=true +``` + + +Since asynchronous processing defers some of the event processing to the +background and takes place after the client has closed the request, some errors +can't be communicated back to the client and are logged by the managed intake service. +Furthermore, asynchronous processing requests will only be scheduled if the managed intake service can +service the incoming request, requests that cannot be serviced will receive an internal error +`503` "queue is full" error. + + +{/* For RUM send an `HTTP POST` request to the managed intake service `intake/v3/rum/events` endpoint instead: + +```bash +http(s)://{hostname}:{port}/intake/v3/rum/events +``` */} + +
+ +### Response + +On success, the server will respond with a 202 Accepted status code and no body. + +Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202. + +
+ +### API Errors + +There are two types of errors that the managed intake service may return to an agent: + +* Event related errors (typically validation errors) +* Non-event related errors + +The managed intake service processes events one after the other. +If an error is encountered while processing an event, +the error encountered as well as the document causing the error are added to an internal array. +The managed intake service will only save 5 event related errors. +If it encounters more than 5 event related errors, +the additional errors will not be returned to agent. +Once all events have been processed, +the error response is sent. + +Some errors, not relating to specific events, +may terminate the request immediately. +For example: IP rate limit reached, wrong metadata, etc. +If at any point one of these errors is encountered, +it is added to the internal array and immediately returned. + +An example error response might look something like this: + +```json +{ + "errors": [ + { + "message": "", [^1] + "document": "" [^2] + },{ + "message": "", + "document": "" + },{ + "message": "", + "document": "" + },{ + "message": "too many requests" [^3] + }, + ], + "accepted": 2320 [^4] +} +``` +[^1]: An event related error +[^2]: The document causing the error +[^3]: An immediately returning non-event related error +[^4]: The number of accepted events + +If you're developing an agent, these errors can be useful for debugging. + +
+ +### Event API Schemas + +The managed intake service uses a collection of JSON Schemas for validating requests to the intake API.