docs: Tracing doc

Add `tracing.md` to document how to use the instrumentation to trace the
Firecracker process.

Signed-off-by: Jonathan Woollett-Light <[email protected]>

docs/tracing.md

@@ -0,0 +1,73 @@
+# Tracing
+
+## Introduction
+
+Instrumentation based tracing is implemented with
+[`log`](https://github.com/rust-lang/log) and
+[`log_instrument`](https://github.com/JonathanWoollett-Light/log-instrument),
+outputting a `Trace` level log when entering and exiting every function.
+
+It is disabled by default at compile-time to remove all impact it would
+have on the release builds.
+
+To enable it you need to remove `"max_level_debug"` from the features
+for `log` under [`../src/vmm/Cargo.toml`](../src/vmm/Cargo.toml):
+
+```diff
+- log = { version = "0.4.17", features = ["std", "serde", "max_level_debug"] }
++ log = { version = "0.4.17", features = ["std", "serde"] }
+```
+
+This will result in a signficant increase in the binary size and a
+signficant regression in performance.
+
+## Filtering
+
+### Run-time
+
+You can filter by module and file this can be done like:
+
+```bash
+curl -X PUT --unix-socket "${API_SOCKET}" \
+    --data "{
+        \"level\": \"Trace\",
+        \"file\": \"src/firecracker/src/api_server_adapter.rs\",
+    }" \
+    "http://localhost/logger"
+```
+
+- Instrumentation logs are `Trace` level, at runtime the level must be
+  set to `Trace` to see them.
+- This will only output logs from `src/firecracker/src/api_server_adapter.rs`.
+
+This can mitigate most of the performance regression.
+
+### Compile-time
+
+It can be valuable to produce a binary with targetted tracing that does not need to be filtered at
+run-time.
+
+To reproduce the same filtering as run-time at compile-time, you can use
+[`clippy-tracing`](https://github.com/JonathanWoollett-Light/clippy-tracing)
+at compile-time like:
+
+```bash
+# Remove all instrumentation.
+clippy-tracing --action strip --path ./src
+# Adds instrumentation to the specific file/s
+clippy-tracing --action fix --path ./src/firecracker/src/api_server_adapter.rs
+```
+
+Then at run-time:
+
+```bash
+curl -X PUT --unix-socket "${API_SOCKET}" \
+    --data "{
+        \"level\": \"Trace\",
+    }" \
+    "http://localhost/logger"
+```
+
+- Instrumentation logs are `Trace` level, at runtime the level must be
+  set to `Trace` to see them.
+- There is now no need to add a run-time filter.