What are the key considerations for observability in a Go production service?
Quick Answer
The three pillars are logging (structured, via log/slog, covered earlier), metrics (counters, gauges, and histograms exposed for a monitoring system to scrape, commonly via the prometheus/client_golang library's /metrics HTTP endpoint), and tracing (following a single request across multiple services, commonly via OpenTelemetry's Go SDK, which propagates a trace context through context.Context alongside the request). A production Go service typically instruments all three: structured logs for detailed, searchable event records; metrics for aggregate health signals (request rate, error rate, latency percentiles) that drive dashboards and alerts; and distributed tracing for understanding where time is actually spent across a multi-service request, especially valuable once a system has more than a couple of services calling each other.
Detailed Answer
The three observability pillars, logs, metrics, and traces, each answer a different operational question, and a mature Go service typically has all three.
Metrics with Prometheus's client library
var (
requestCount = prometheus.NewCounterVec(
prometheus.CounterOpts{Name: "http_requests_total"},
[]string{"method", "path", "status"},
)
requestDuration = prometheus.NewHistogramVec(
prometheus.HistogramOpts{Name: "http_request_duration_seconds"},
[]string{"method", "path"},
)
)
func init() {
prometheus.MustRegister(requestCount, requestDuration)
}
func instrumentedHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
next.ServeHTTP(w, r)
requestDuration.WithLabelValues(r.Method, r.URL.Path).Observe(time.Since(start).Seconds())
requestCount.WithLabelValues(r.Method, r.URL.Path, "200").Inc()
})
}
http.Handle("/metrics", promhttp.Handler()) // Prometheus scrapes this endpoint
This gives you aggregate signals, request rate, error rate, and latency distribution, that drive dashboards and alerting rules, answering "is the service healthy right now, in general."
Distributed tracing with OpenTelemetry
tracer := otel.Tracer("myapp")
func handler(w http.ResponseWriter, r *http.Request) {
ctx, span := tracer.Start(r.Context(), "handle-request")
defer span.End()
result, err := fetchFromDatabase(ctx) // trace context propagates through ctx
if err != nil {
span.RecordError(err)
}
// ...
}
Tracing answers a different question than metrics: "for this specific slow request, where exactly was the time spent, across however many services it touched." Propagating the trace context through context.Context is what lets spans from different services stitch together into one coherent trace.
Why all three matter together
Logs: "what exactly happened, in detail, for this specific event"
Metrics: "is the system healthy in aggregate, right now"
Traces: "for this one specific slow/failed request, where did the time actually go"
Relying on only one pillar leaves real blind spots. Metrics alone tell you that p99 latency spiked, but not why. A trace for one of the slow requests during that spike often reveals the actual bottleneck, like a slow downstream dependency, that aggregate metrics alone couldn't pinpoint.