Synced from the repo — do not edit here

Canonical source: docs/MONITORING_SETUP.md. This page is generated by docs/scripts/sync-handbook.mjs. Edit the source file in the repo; changes appear here on the next build.

Monitoring & Alerting Setup Guide

This guide covers setting up the complete monitoring stack for Basis Health.

Table of Contents

1. UptimeRobot Custom Domain
2. PagerDuty Integration
3. Google Cloud Monitoring Alerts
4. Slack Integration
5. Sentry Configuration

---

UptimeRobot Custom Domain

Goal: status.basishealth.io → Your UptimeRobot status page

Step 1: Get Your UptimeRobot CNAME

1. Go to UptimeRobot Dashboard
2. Click Status Pages in the left sidebar
3. Select your status page (CB0yTESg4b)
4. Click Settings tab
5. Under Custom Domain, note the CNAME target (e.g., status.uptimerobot.com)

Step 2: Add DNS Record

In your DNS provider (Cloudflare, Route53, etc.):

Type: CNAME
Name: status
Value: status.uptimerobot.com
TTL: Auto/300

Step 3: Configure in UptimeRobot

1. In UptimeRobot Status Page settings
2. Enter custom domain: status.basishealth.io
3. Enable SSL (they provide free SSL)
4. Save

Step 4: Verify

Wait 5-10 minutes, then visit https://status.basishealth.io

---

PagerDuty Integration

Goal: Critical alerts → PagerDuty → Phone/SMS notification

Your PagerDuty Routing Key

c033995a23494708c0b4e4c901a81822

Step 1: Create Alert in Cloud Monitoring

1. Go to Google Cloud Console
2. Navigate to Monitoring → Alerting
3. Click Create Policy

Step 2: Configure Notification Channel

1. In Alerting, click Edit Notification Channels
2. Click Add New under PagerDuty
3. Enter:
   • Display Name: Basis Health Alerts
   • Service Key: c033995a23494708c0b4e4c901a81822
4. Click Save

Step 3: Test PagerDuty Integration

Run this curl command to verify:

curl --request 'POST' \
  --url 'https://events.pagerduty.com/v2/enqueue' \
  --header 'Content-Type: application/json' \
  --data '{
    "payload": {
        "summary": "Test Alert - Please Ignore",
        "severity": "info",
        "source": "Manual Test"
    },
    "routing_key": "c033995a23494708c0b4e4c901a81822",
    "event_action": "trigger"
  }'

You should receive a notification. Resolve it in PagerDuty dashboard.

---

Google Cloud Monitoring Alerts

Critical Alerts (PagerDuty)

Create these alert policies that trigger PagerDuty:

1. Cloud Function Errors

Display Name: "Critical: Cloud Function Error Rate High"
Conditions:
  - Resource Type: Cloud Function
  - Metric: cloudfunctions.googleapis.com/function/execution_count
  - Filter: status != "ok"
  - Threshold: > 10 errors in 5 minutes
Notification: PagerDuty

2. Cloud Function Timeout

Display Name: "Critical: Cloud Function Timeouts"
Conditions:
  - Resource Type: Cloud Function
  - Metric: cloudfunctions.googleapis.com/function/execution_times
  - Filter: execution timeout
  - Threshold: > 5 timeouts in 5 minutes
Notification: PagerDuty

3. Firestore Errors

Display Name: "Critical: Firestore Errors"
Conditions:
  - Resource Type: Cloud Firestore Database
  - Metric: firestore.googleapis.com/api/request_count
  - Filter: response_code >= 500
  - Threshold: > 20 errors in 5 minutes
Notification: PagerDuty

Warning Alerts (Slack Only)

Create these alerts that only go to Slack:

4. High Latency

Display Name: "Warning: High API Latency"
Conditions:
  - Resource Type: Cloud Function
  - Metric: cloudfunctions.googleapis.com/function/execution_times
  - Aggregation: 95th percentile
  - Threshold: > 3000ms for 10 minutes
Notification: Slack

5. Cost Anomaly

Display Name: "Warning: Unexpected Cost Increase"
Conditions:
  - Use Budget Alerts in Billing
  - Threshold: 80% of monthly budget
Notification: Slack

Quick Setup via gcloud CLI

# Create PagerDuty notification channel
gcloud alpha monitoring channels create \
  --display-name="PagerDuty - Basis Health" \
  --type=pagerduty \
  --channel-labels=service_key=c033995a23494708c0b4e4c901a81822

# List channels to get the channel ID
gcloud alpha monitoring channels list

# Create alert policy (replace CHANNEL_ID)
gcloud alpha monitoring policies create \
  --display-name="Critical: Cloud Function Errors" \
  --condition-display-name="Error rate high" \
  --condition-filter='resource.type="cloud_function" AND metric.type="cloudfunctions.googleapis.com/function/execution_count" AND metric.labels.status!="ok"' \
  --condition-threshold-value=10 \
  --condition-threshold-duration=300s \
  --notification-channels=CHANNEL_ID

---

Slack Integration

Step 1: Create Slack Webhook

1. Go to Slack API
2. Create new app or use existing
3. Add Incoming Webhooks feature
4. Create webhook for #basis-alerts channel
5. Copy the webhook URL

Step 2: Add to Cloud Monitoring

1. In Cloud Monitoring → Alerting → Notification Channels
2. Add new Slack channel
3. Paste webhook URL
4. Test the connection

Step 3: Create Slack Channel

Create these channels:

• #basis-alerts - All alerts
• #basis-critical - PagerDuty-level only (optional)
• #basis-deploys - Deployment notifications (optional)

---

Sentry Configuration

New Sentry Project Setup

Since you're creating a new Sentry account:

Step 1: Create Projects

Create these projects in Sentry:

1. basis-hybrid - Flutter iOS/Android app
2. basis-flow - Flutter provider app
3. basis-functions - Python Cloud Functions
4. basis-web - Web applications

Step 2: Get DSN Values

For each project, get the DSN from Project Settings → Client Keys (DSN)

Step 3: Update Flutter Configuration

In hybrid/basishybrid/lib/main.dart:

await SentryFlutter.init(
  (options) {
    options.dsn = 'YOUR_NEW_DSN_HERE';
    options.tracesSampleRate = 0.2; // 20% of transactions
    options.profilesSampleRate = 0.1; // 10% of profiled transactions
    options.environment = kReleaseMode ? 'production' : 'development';
  },
  appRunner: () => runApp(const MyApp()),
);

Step 4: Update Python Configuration

In basis-functions/functions/src/main.py:

import sentry_sdk
from sentry_sdk.integrations.gcp import GcpIntegration

sentry_sdk.init(
    dsn="YOUR_NEW_DSN_HERE",
    integrations=[GcpIntegration()],
    traces_sample_rate=0.2,
    environment="production",
)

Step 5: Configure Alerts in Sentry

1. Go to Sentry → Alerts
2. Create alert rules:
   • High Error Volume: >100 events in 1 hour → Slack
   • New Issue: First occurrence → Slack
   • Regression: Issue reappears → Slack
   • Critical Error: Specific errors → PagerDuty

Step 6: Integrate Sentry with PagerDuty

1. In Sentry → Settings → Integrations
2. Find PagerDuty
3. Connect using your PagerDuty account
4. Map Sentry projects to PagerDuty services

---

Alert Thresholds Summary

Alert	Threshold	Destination	Severity
API Down	100% failure for 2 min	PagerDuty	Critical
Error Rate	>5% for 5 min	PagerDuty	Critical
High Latency	p95 >3s for 10 min	Slack	Warning
Function Timeout	>5 timeouts/5 min	PagerDuty	Critical
Firestore Errors	>20 errors/5 min	PagerDuty	Critical
Cost Anomaly	>80% budget	Slack	Warning
New Sentry Issue	First occurrence	Slack	Info
Sentry Regression	Issue reappears	Slack	Warning

---

Verification Checklist

After setup, verify each component:

• UptimeRobot status page accessible at status.basishealth.io
• Trust Center accessible at trust.basishealth.io
• PagerDuty test alert received
• Slack test message received
• Cloud Monitoring alerts configured
• Sentry new project receiving events
• Dependabot PRs appearing on GitHub

---

Troubleshooting

UptimeRobot Custom Domain Not Working

• Wait up to 24 hours for DNS propagation
• Check CNAME record is correct
• Verify SSL is enabled in UptimeRobot

PagerDuty Not Receiving Alerts

• Verify routing key is correct
• Check PagerDuty service is not disabled
• Test with curl command above

Cloud Monitoring Alerts Not Firing

• Check metric filter syntax
• Verify notification channel is configured
• Check IAM permissions for monitoring

Sentry Not Receiving Events

• Verify DSN is correct
• Check network connectivity to Sentry
• Verify SDK is initialized before errors occur