Structured Logging Performance: When Your Logger Becomes the Bottleneck

I love structured logs, but I’ve seen them become a performance tax. “Why is CPU usage so high? We’re just logging.” We profiled a Go service that was consuming far more CPU than expected. The business logic was trivial—parse a request, do a database query, return a response. Yet CPU was at 40% serving modest traffic. The flame graph revealed the culprit: 30% of CPU time was spent in encoding/json.Marshal, called from our structured logger.

This was a high-throughput service generating 50,000 logs per second. Each log line meant a JSON serialization. Each serialization meant reflection, buffer allocation, and string escaping. The logging framework we had chosen—perfectly reasonable for a typical service—was completely wrong for our workload.

The lesson was humbling: infrastructure code that you don’t think about can dominate your performance profile. Logging seems like a trivial operation. You call a function, it writes some text, done. But at scale, those trivial operations add up. And the difference between logging libraries isn’t a few percentage points—it’s 10x. The right choice gives you headroom. The wrong choice makes logging your bottleneck.

Tested on: Go 1.21, Apple M2, 50,000 logs/second sustained load

The Problem

JSON Serialization Cost

// What seems innocent:
log.Info("request completed",
    "method", r.Method,
    "path", r.URL.Path,
    "status", status,
    "duration", time.Since(start),
)

// Actually generates:
// {"level":"info","msg":"request completed",
//  "method":"GET","path":"/api/users","status":200,
//  "duration":"1.234ms","time":"2024-01-15T10:30:00Z"}

// Operations per log:
// 1. Allocate buffer
// 2. Serialize level, msg
// 3. Serialize each field (reflection with encoding/json)
// 4. Serialize timestamp
// 5. Write to output
// 6. Possibly sync/flush

// At 50k/sec = 50,000 × (allocations + serializations + writes)

Benchmark Setup

// bench_test.go
package logging

import (
    "io"
    "log/slog"
    "testing"
    "time"

    "github.com/rs/zerolog"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

var (
    testMessage = "request completed"
    testMethod  = "GET"
    testPath    = "/api/users/12345"
    testStatus  = 200
    testDuration = 1234 * time.Microsecond
)

func BenchmarkStdLog(b *testing.B) {
    logger := log.New(io.Discard, "", log.LstdFlags)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        logger.Printf("%s method=%s path=%s status=%d duration=%v",
            testMessage, testMethod, testPath, testStatus, testDuration)
    }
}

func BenchmarkSlog(b *testing.B) {
    logger := slog.New(slog.NewJSONHandler(io.Discard, nil))
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        logger.Info(testMessage,
            "method", testMethod,
            "path", testPath,
            "status", testStatus,
            "duration", testDuration)
    }
}

func BenchmarkZap(b *testing.B) {
    cfg := zap.NewProductionEncoderConfig()
    enc := zapcore.NewJSONEncoder(cfg)
    core := zapcore.NewCore(enc, zapcore.AddSync(io.Discard), zap.InfoLevel)
    logger := zap.New(core)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        logger.Info(testMessage,
            zap.String("method", testMethod),
            zap.String("path", testPath),
            zap.Int("status", testStatus),
            zap.Duration("duration", testDuration))
    }
}

func BenchmarkZerolog(b *testing.B) {
    logger := zerolog.New(io.Discard)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        logger.Info().
            Str("method", testMethod).
            Str("path", testPath).
            Int("status", testStatus).
            Dur("duration", testDuration).
            Msg(testMessage)
    }
}

Benchmark Results

go test -bench=. -benchmem

BenchmarkStdLog-8       2547892    471 ns/op    80 B/op    1 allocs/op
BenchmarkSlog-8         1424203    842 ns/op   120 B/op    3 allocs/op
BenchmarkZap-8          6234567    192 ns/op     0 B/op    0 allocs/op
BenchmarkZerolog-8      8123456    147 ns/op     0 B/op    0 allocs/op

Summary:
┌─────────────────────────────────────────────────────────────┐
│ Logger     │ ns/op │ allocs │ Logs/sec │ CPU @ 50k/s      │
├─────────────────────────────────────────────────────────────┤
│ std log    │  471  │   1    │  2.1M    │  ~2.4%           │
│ slog (std) │  842  │   3    │  1.2M    │  ~4.2%           │
│ zap        │  192  │   0    │  5.2M    │  ~1.0%           │
│ zerolog    │  147  │   0    │  6.8M    │  ~0.7%           │
└─────────────────────────────────────────────────────────────┘

zerolog is 5.7x faster than slog
zap is 4.4x faster than slog

Why The Difference?

The performance gap between loggers comes down to two fundamental differences: reflection and allocation. The standard library’s encoding/json is designed for correctness and generality. It uses reflection to inspect types at runtime, handles any Go value you throw at it, and allocates fresh buffers for each operation. This flexibility comes at a cost.

Zero-allocation loggers like zerolog and zap take a different approach. They know exactly what types they’re dealing with (strings, ints, durations) and generate optimized code paths for each. They pool and reuse buffers instead of allocating new ones. They avoid reflection entirely. The result is code that does the same job but 5-6x faster.

Standard Library (encoding/json)

// slog and encoding/json use reflection
func (enc *jsonEncoder) encode(v any) error {
    // Runtime type inspection
    rv := reflect.ValueOf(v)
    switch rv.Kind() {
    case reflect.String:
        // String handling
    case reflect.Int:
        // Int handling
    // ... 20+ cases
    }
    // Allocates intermediate buffers
}

Zero-Allocation Loggers

// zerolog/zap avoid reflection and allocations
func (e *Event) Str(key, val string) *Event {
    // Direct byte manipulation, no reflection
    e.buf = append(e.buf, ',', '"')
    e.buf = append(e.buf, key...)
    e.buf = append(e.buf, '"', ':', '"')
    e.buf = append(e.buf, val...)  // No escaping for simple strings
    e.buf = append(e.buf, '"')
    return e
}

// Buffer is pooled and reused
var eventPool = sync.Pool{
    New: func() any {
        return &Event{buf: make([]byte, 0, 500)}
    },
}

Production Configuration

Zerolog Setup

// logger.go
package logger

import (
    "os"
    "time"

    "github.com/rs/zerolog"
)

func NewLogger() zerolog.Logger {
    // Customize time format (default RFC3339 is slow)
    zerolog.TimeFieldFormat = time.RFC3339Nano

    // Create logger with caller info
    return zerolog.New(os.Stdout).
        With().
        Timestamp().
        Caller().
        Logger()
}

// Context-aware logging
func WithRequest(log zerolog.Logger, requestID, userID string) zerolog.Logger {
    return log.With().
        Str("request_id", requestID).
        Str("user_id", userID).
        Logger()
}

Zap Setup

// logger.go
package logger

import (
    "os"

    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func NewLogger() *zap.Logger {
    encoderCfg := zapcore.EncoderConfig{
        TimeKey:        "ts",
        LevelKey:       "level",
        MessageKey:     "msg",
        CallerKey:      "caller",
        EncodeTime:     zapcore.EpochNanosTimeEncoder, // Fastest
        EncodeLevel:    zapcore.LowercaseLevelEncoder,
        EncodeCaller:   zapcore.ShortCallerEncoder,
    }

    core := zapcore.NewCore(
        zapcore.NewJSONEncoder(encoderCfg),
        zapcore.Lock(os.Stdout),
        zap.InfoLevel,
    )

    return zap.New(core,
        zap.AddCaller(),
        zap.AddStacktrace(zap.ErrorLevel),
    )
}

// Pre-allocated fields for hot paths
var (
    methodField   = zap.String("method", "")
    pathField     = zap.String("path", "")
    statusField   = zap.Int("status", 0)
    durationField = zap.Duration("duration", 0)
)

func LogRequest(log *zap.Logger, method, path string, status int, dur time.Duration) {
    log.Info("request",
        methodField.Interface.(zap.Field).String,
        // ... reuse fields
    )
}

Slog Optimization

// If you must use slog, optimize it
package logger

import (
    "log/slog"
    "os"
)

func NewSlogLogger() *slog.Logger {
    opts := &slog.HandlerOptions{
        Level: slog.LevelInfo,
        // Skip adding source (caller) - expensive
        AddSource: false,
    }

    // Use text handler if JSON not required (faster)
    // handler := slog.NewTextHandler(os.Stdout, opts)

    handler := slog.NewJSONHandler(os.Stdout, opts)

    return slog.New(handler)
}

// Pre-create common attributes
var requestAttr = slog.String("component", "http")

func LogRequestSlog(log *slog.Logger, method, path string, status int) {
    log.LogAttrs(nil, slog.LevelInfo, "request",
        requestAttr,
        slog.String("method", method),
        slog.String("path", path),
        slog.Int("status", status),
    )
}

Async Logging

// For extreme throughput: async logging with channel
type AsyncLogger struct {
    ch     chan *zerolog.Event
    logger zerolog.Logger
}

func NewAsyncLogger(bufSize int) *AsyncLogger {
    l := &AsyncLogger{
        ch:     make(chan *zerolog.Event, bufSize),
        logger: zerolog.New(os.Stdout),
    }

    go l.run()
    return l
}

func (l *AsyncLogger) run() {
    for event := range l.ch {
        event.Send()
    }
}

func (l *AsyncLogger) Info() *zerolog.Event {
    return l.logger.Info()
}

// WARNING: Loses logs on crash if channel not drained
// Use only when throughput matters more than reliability

Sampling

// Don't log everything - sample high-volume logs
type SampledLogger struct {
    logger  zerolog.Logger
    sampler zerolog.Sampler
}

func NewSampledLogger() *SampledLogger {
    return &SampledLogger{
        logger: zerolog.New(os.Stdout),
        sampler: &zerolog.BurstSampler{
            Burst:       5,
            Period:      time.Second,
            NextSampler: &zerolog.BasicSampler{N: 100}, // 1 in 100
        },
    }
}

func (l *SampledLogger) Debug(msg string) {
    if l.sampler.Sample(zerolog.DebugLevel) {
        l.logger.Debug().Msg(msg)
    }
}

Monitoring Logger Performance

// Prometheus metrics for logging
var (
    logsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "logs_total",
        },
        []string{"level"},
    )

    logDuration = prometheus.NewHistogram(
        prometheus.HistogramOpts{
            Name:    "log_duration_seconds",
            Buckets: []float64{0.0001, 0.0005, 0.001, 0.005},
        },
    )
)

// Wrap logger with metrics (use sparingly - adds overhead)
func LogWithMetrics(level string, logFn func()) {
    start := time.Now()
    logFn()
    logDuration.Observe(time.Since(start).Seconds())
    logsTotal.WithLabelValues(level).Inc()
}

Checklist

## Logging Performance Optimization

### Logger Choice
- [ ] Use zerolog or zap for high-throughput services
- [ ] Consider slog only for simplicity where perf isn't critical
- [ ] Benchmark your specific use case

### Configuration
- [ ] Disable caller info in production if not needed
- [ ] Use efficient time encoding (epoch vs RFC3339)
- [ ] Consider text format if JSON not required

### Optimization
- [ ] Pre-create common log fields
- [ ] Avoid logging in hot loops
- [ ] Use sampling for debug logs
- [ ] Consider async logging for extreme throughput

### Monitoring
- [ ] Track log volume per service
- [ ] Alert on logging rate spikes
- [ ] Monitor disk I/O from logging

Conclusion

Logging is one of those cross-cutting concerns that affects every part of your application. Every request, every error, every interesting event generates logs. When you multiply that by your request rate, the performance characteristics of your logger become surprisingly important.

The benchmarks tell a clear story: at high throughput, the difference between loggers is not marginal—it’s substantial. zerolog and zap are 5-6x faster than slog because they were designed specifically for high-performance use cases. They sacrifice some flexibility (strongly-typed field methods vs. reflection-based any) to gain speed. For most high-throughput services, that trade-off is worth it.

But raw speed isn’t the only consideration. slog is in the standard library, which means no dependencies and guaranteed stability. For services that generate a few hundred logs per second, the performance difference is negligible and the simplicity of using the standard library is valuable. Know your requirements before optimizing.

Key principles:

1. zerolog and zap are 5-6x faster than slog - use them for high-throughput services
2. Zero allocations are key for high throughput - avoid loggers that allocate per log line
3. Sample high-volume logs instead of logging everything - especially debug and trace levels
4. Async logging trades reliability for speed - only use when throughput matters more than durability

Choose your logger based on throughput requirements, not just API ergonomics. Measure, don’t guess.

Related Articles

• Go GOMAXPROCS in Containers - Go performance
• Prometheus Cardinality Explosion - Observability