Monitoring & Observability for Solo Developers: Go + GCP

I run Amida-san (amida-san.com), a web service with 10,000+ users, as a solo developer. The backend is written in Go and deployed on GCP.

In the previous article, I covered the CI/CD pipeline. This article focuses on monitoring, logging, alerts, and admin tooling—the systems that keep the service running reliably.

A service being "up" and a service being "stable" are two different things. Detecting issues early, identifying root causes quickly, and responding effectively—that's what this monitoring stack is built for.

Architecture Overview

Structured Logging with slog

Why Structured Logs?

Text-based logs make it difficult to trace what happened after the fact. Structured logs enable field-level filtering and aggregation in Cloud Logging.

I adopted log/slog, added to Go's standard library in Go 1.21.

Unified Log Functions

// Consistent log output across all layers
shared.LogInfo(ctx, shared.ComponentInteractor, shared.OpCreateEvent, "Event created",
    shared.LogKeyEventId, event.ID,
    shared.LogKeyUserId, userID)

shared.LogError(ctx, shared.ComponentRepository, shared.OpDBQuery, "Query failed", err,
    shared.LogKeyEventId, eventID)

All log output goes through four functions: LogInfo / LogWarn / LogError / LogDebug. Each automatically attaches:

• Component name (Router, Interactor, Repository, etc.)
• Operation name (CreateEvent, DBQuery, etc.)
• HTTP context (request ID, endpoint, method, remote address, user agent)
• Source information (file name, line number)

Log Level Strategy

Each layer uses appropriate log levels:

To avoid duplicate logs, the rule is: log at the Repository layer, skip duplicate logging at the Interactor layer.

Request Logging Middleware

func RequestLoggingMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        ctx := c.Request.Context()

        shared.LogInfo(ctx, shared.ComponentRouter, shared.OpRequestStart,
            "Request started")

        c.Next()

        duration := time.Since(start)
        statusCode := c.Writer.Status()

        if statusCode >= 500 {
            shared.LogWarn(ctx, shared.ComponentRouter, shared.OpRequestCompleted,
                "Request completed with server error",
                shared.LogKeyStatusCode, statusCode,
                "duration_ms", duration.Milliseconds())
        } else {
            shared.LogInfo(ctx, shared.ComponentRouter, shared.OpRequestCompleted,
                "Request completed successfully",
                shared.LogKeyStatusCode, statusCode,
                "duration_ms", duration.Milliseconds())
        }
    }
}

All requests are logged at start and completion, with log levels switching based on status code. 5xx responses use Warn (triggering Discord notifications), while others use Info. The middleware uses Warn rather than Error because the actual error source (Interactor or Repository layer) already logs at Error level—avoiding duplication while enabling per-request anomaly detection.

Request ID Tracking

func HTTPContextMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        requestID := c.GetHeader("X-Request-ID")
        if requestID == "" {
            requestID = generateRequestID() // Random 16-char hex
        }

        ctx := context.WithValue(c.Request.Context(), keys.RequestID, requestID)
        c.Header("X-Request-ID", requestID)
        c.Request = c.Request.WithContext(ctx)
        c.Next()
    }
}

Request IDs are automatically attached to all logs and returned in response headers. This enables cross-cutting traceability for any single request.

Discord Error Alerts

Cloud Logging requires you to actively check for issues. As a solo developer, you need a system that pushes notifications to you when something goes wrong.

Error vs. Warn Notification Strategy

// LogError → Immediate Discord notification (@everyone)
func LogError(ctx context.Context, component LogComponent, operation LogOperation,
    message string, err error, keyValues ...interface{}) {
    logWithLevel(ctx, slog.LevelError, component, operation, message, err, keyValues...)

    if globalDiscordConfig != nil && globalDiscordConfig.Enabled {
        SendDiscordErrorAlert(ctx, globalDiscordConfig.WebhookURL,
            component, operation, message, err, keyValues...)
    }
}

// LogWarn → Buffer and batch send at intervals
func LogWarn(ctx context.Context, component LogComponent, operation LogOperation,
    message string, keyValues ...interface{}) {
    logWithLevel(ctx, slog.LevelWarn, component, operation, message, nil, keyValues...)

    if globalDiscordConfig != nil && globalDiscordConfig.Enabled {
        SendDiscordWarnAlert(ctx, globalDiscordConfig.WebhookURL,
            component, operation, message, nil, keyValues...)
    }
}

Warn Buffering

Warnings are buffered up to 10 entries and sent as a single message at regular intervals:

• Avoids hitting Discord Webhook rate limits
• Prevents notification explosion during warning storms
• Aggregated view makes it easier to spot trends

Information Included in Notifications

Discord notifications automatically include:

• Component name and operation name
• Error message
• Source information (filename:line number + function name)
• Request ID, endpoint, method, remote IP
• Business data (user ID, event ID, payment ID, etc.)

With source information, you can identify exactly where in the code the problem occurred just from the notification.

GCP Cloud Monitoring

Terraform-Managed Dashboards and Alerts

All dashboards and alert policies are managed through Terraform. Code-based management ensures reproducibility and change history, rather than manual GCP Console configuration.

Dashboard Layout

A 12-tile dashboard visualizes the following:

Alert Policies

The following alerts are defined in Terraform, triggering email notifications when thresholds are exceeded:

# Example: 5xx error rate alert
resource "google_monitoring_alert_policy" "error_rate" {
  display_name = "High Error Rate"

  conditions {
    display_name = "HTTP 5xx error rate"
    condition_threshold {
      filter     = "... metric.type=\"run.googleapis.com/request_count\" AND metric.label.response_code_class=\"5xx\""
      comparison = "COMPARISON_GT"
      threshold_value = var.error_rate_threshold

      aggregations {
        alignment_period   = "60s"
        per_series_aligner = "ALIGN_DELTA"
      }
    }
  }

  alert_strategy {
    auto_close = "86400s"
  }
}

All alerts have enable_xxx_alerts variables, allowing per-environment enable/disable control.

Log-Based Metrics

ERROR logs from Cloud Logging are converted into custom metrics and used as alert policy targets:

resource "google_logging_metric" "app_errors" {
  name   = "app_error_count"
  filter = "resource.type=\"cloud_run_revision\" AND severity=\"ERROR\""

  metric_descriptor {
    metric_kind  = "DELTA"
    value_type   = "INT64"
    display_name = "Application Error Count"
  }
}

This makes structured log ERROR output directly alertable through GCP.

Error Handler

func ErrorHandler() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.Next()

        lastGinError := c.Errors.Last()
        if lastGinError == nil {
            return
        }

        var appErr *shared.AppError
        if errors.As(lastGinError.Err, &appErr) {
            if gin.Mode() == gin.ReleaseMode {
                // Production: error code and generic message only
                c.JSON(appErr.Status, shared.AppError{
                    Code:    appErr.Code,
                    Message: "An unexpected error occurred.",
                })
            } else {
                // Development: detailed error information
                c.JSON(appErr.Status, appErr)
            }
            return
        }
    }
}

• Production hides internal error details from responses (security)
• Development returns detailed error information (debugging efficiency)
• All errors are recorded in structured logs (traceability)

Admin Web App

A dedicated admin web app (React + TypeScript) provides visibility into service health:

• Payment analytics: revenue trends, success rates, daily breakdowns, event rankings
• System statistics: real-time user count, event count, participant count
• Event management: event status monitoring and operational controls

This surfaces business metrics that are difficult to track through the GCP Console alone.

Conclusion

Here's a summary of the monitoring stack:

Building something that works is just the beginning. Keeping it running reliably requires monitoring infrastructure like this. As a solo developer, building systems that detect problems even when you're not watching is essential.

Originally published at shusukedev.com