Observability
The RIC exposes its state three ways: a Prometheus metrics endpoint, structured logs, and a bundled Grafana dashboard. xApps report their own telemetry separately through the SDK's sinks.
Metrics endpoint
When telemetry.port is non-zero the RIC serves Prometheus metrics at
GET /metrics, driven directly on the worker loops (no extra threads):
The endpoint answers even before any E2 node connects. The
telemetry.mode knob (Configuration)
selects production (aggregate counters + registry gauges, constant-size
body) or profile (adds per-(deployment, node) EPI latency histograms).
Live gauges
| Metric | Meaning |
|---|---|
airpuls_sm_plugins_loaded |
SM plugins registered at startup |
airpuls_subscriptions_active |
Active subscriptions across all nodes |
airpuls_sm_subscriptions_active |
Active subscriptions, labelled by sm |
airpuls_xapp_subscriptions |
Subscriptions owned per xApp |
The live E2 node registry is also exposed as gauges, labelled with the BCD-decoded PLMN.
Procedure counters
| Metric | Meaning |
|---|---|
airpuls_subscription_requests_total / _failures_total |
RIC Subscription outcomes |
airpuls_indications_total / airpuls_insert_indications_total |
RIC Indications handled (REPORT / INSERT) |
airpuls_control_requests_total / _failures_total / _timeout_total |
RIC Control outcomes |
airpuls_query_requests_total / _responses_total / _failures_total{reason} / _timeout_total |
RIC Query outcomes. reason="info-unavailable" is a node correctly reporting it holds none of the requested information and is expected in normal operation, not a fault |
airpuls_reset_requests_received_total / _sent_total |
E2 Reset in both directions |
airpuls_connection_updates_sent_total / _acked_total / _failed_total |
E2 Connection Update |
airpuls_messages_handled_total |
All dispatched E2AP messages |
airpuls_decode_failures_total / airpuls_validation_rejected_total |
Malformed / rejected inbound frames |
Per-SM data-volume counters
Byte-volume counters labelled by service model let you see exactly how much each SM moves, in each direction:
| Metric | Interface · direction |
|---|---|
airpuls_sm_indications_bytes_in_total{sm="…"} |
Southbound ingress (gNB → RIC) |
airpuls_sm_indications_bytes_out_total{sm="…"} |
Northbound egress (RIC → xApp) |
airpuls_sm_control_bytes_in_total{sm="…"} / _out_total |
Control path, both directions |
airpuls_sm_indications_total{sm="…"} / airpuls_sm_control_requests_total{sm="…"} |
Per-SM event counts |
A rate() over a _bytes_ counter gives bytes/sec. A flat line for an
SM means no traffic for it is reaching the RIC — the first thing to check
when an xApp sees no indications.
Transport & runtime health
| Metric | Meaning |
|---|---|
airpuls_sb_bytes_in_total / _out_total |
Aggregate southbound SCTP bytes |
airpuls_nb_bytes_in_total / _out_total |
Aggregate northbound IPC bytes |
airpuls_sctp_send_eagain_total |
SCTP send back-pressure events |
airpuls_cross_thread_post_overflows_total |
SPSC cross-thread queue overflows (SB→NB back-pressure) |
EPI latency histograms (profile mode)
With telemetry.mode: profile, the E2SM-EPI probe
driver populates full-trip RTT histograms, decomposed by leg:
| Metric | Leg |
|---|---|
airpuls_epi_full_rtt_seconds |
End-to-end xApp → agent → xApp |
airpuls_epi_sb_rtt_seconds |
Southbound (RIC ↔ agent) |
airpuls_epi_nb_rtt_seconds |
Northbound (RIC ↔ xApp) |
airpuls_epi_agent_service_seconds |
Time spent inside the agent |
airpuls_epi_ric_routing_seconds |
Time spent routing inside the RIC |
Logging
Structured logging with five levels — FATAL, ERROR, WARNING,
INFO, VERBOSE — each line carrying a timestamp, level, module tag,
and source location:
The level is resolved CLI flag (-v / -q) → logging.level in
nrtric.yml → default info. A WARNING is emitted at startup when a
unix:// xApp endpoint clamps the northbound worker pool to one, so
notable non-default configuration is visible in operator logs.
Grafana dashboard
A Grafana dashboard ships under deploy/ in the source tree. It plots
the per-SM data-volume counters on mirrored panels — "Southbound Data
Volume per SM (RIC ↔ E2 Node)" and "Northbound Data Volume per SM
(RIC ↔ xApp)" — with direction encoded by sign, alongside subscription
gauges, procedure rates, and (in profile mode) the EPI latency
breakdown. Point Grafana at the Prometheus that scrapes the RIC's
/metrics endpoint and import the dashboard JSON.
Confirming data flow
When you switch a node from the emulator to a live gNB, the per-SM
byte panels are where you confirm traffic is actually flowing: watch
rate(airpuls_sm_indications_bytes_in_total{sm="KPM"}[1m]) climb as
subscriptions produce indications.