Express courier. Direct.
Also overnight.
x25pronto Switzerland takes goods direct to the destination. Small loads go by car. Larger loads go by van or Van XL. Direct, express and overnight trips run Monday to Sunday when a slot is free. Free times are shown in the booking calendar. This keeps the plan clear.
Observability for webhooks: Dashboards, tracing and alerting
How to monitor webhook integrations using Prometheus metrics, OpenTelemetry traces and targeted alerts to detect and fix failures faster.
Quick answer
In short: combine metrics, distributed tracing and focused alerts. Record latency, success rates and payload errors with Prometheus. Instrument code with OpenTelemetry to trace requests end‑to‑end. Visualise data in Grafana. Create alerts for rising error rates, latency and connection failures. This reveals root causes quickly.
Why observability matters for webhooks
Webhook endpoints connect you to external systems. Failures or slow responses affect integrations immediately. Observability lets you see causes, not just symptoms: is latency inside your service, at the target API or in the network? Metrics, traces and logs help you find the problem faster.
Which metrics to collect (Prometheus)
Use Prometheus for time‑series metrics. Key metrics: 1) request count by endpoint and method. 2) success/failure ratio (2xx vs 4xx/5xx). 3) latency histograms (p50, p90, p99). 4) retry and dead‑letter counters. 5) connection, DNS and TLS errors. Labels: endpoint, tenant, region, http_status, retry_attempt. These enable trend analysis and reliable alerts.
OpenTelemetry tracing: what and how to instrument
Instrument the webhook flow end‑to‑end with OpenTelemetry. Create spans for receive, validation, processing, outbound call and response. Propagate context via W3C Trace Context headers. Important attributes: http.method, http.url, http.status_code, retry_count, handler_name. Use sampling wisely: lower sampling under heavy load, keep higher sampling for SLO‑critical paths.
Dashboards: overview, drilldown and SLA checks
Build three Grafana dashboard layers: 1) overview (traffic, error rate, p90/p99 latency). 2) endpoint details (top error causes, distributions). 3) trace exploration (links to Jaeger/Tempo). Add latency heatmaps and a table for frequent 4xx/5xx errors. Show SLAs or SLOs visually for stakeholders.
Alerting: useful thresholds and actions
Alerts should be timely and actionable. Example thresholds: - Error rate > 5% for 5 min on a critical endpoint. - p95 latency > 1.5s for 10 min. - Retries per minute > 50. - Dead‑letter messages up > 20% vs 24h median. Pair metric alerts with trace links and sample payloads. Route alerts to the right team and add escalation steps.
Incident workflow: metrics → traces → logs
Start with metrics to narrow scope, open relevant traces for failed requests and add logs for payload and validation context. A clear incident playbook and runbook lowers time‑to‑resolution and prevents repeated mistakes.
Frequently asked questions
How to combine Prometheus and OpenTelemetry effectively?
Prometheus is great for aggregated time‑series metrics. OpenTelemetry provides distributed traces and rich span context. Export metrics via Prometheus clients and traces via OpenTelemetry SDK. Link metric alerts to example traces for quick root‑cause work.
What sampling rate works under high load?
Start with 1–5% global sampling at high throughput. Increase sampling to 20–100% for critical endpoints or during incidents. The goal is useful visibility without unsustainable storage costs.
How do I set alert thresholds without causing noise?
Use relative and absolute thresholds, apply time windows (5–10 min) and compare to historical baselines. Silence alerts during known deployments and tune in staging first.
Should I provide separate dashboards for partners and operations?
Yes. A partner‑facing dashboard shows SLOs and basic status. Internal dashboards include traces, raw metrics and tools for engineers.