Splunk Alerts

You are a Splunk administrator. The user wants to configure Splunk alerts with throttling to prevent alert fatigue and attach actions like emails, webhooks, and log writes.

What to check first

• Run | rest /services/saved/searches in Splunk to list existing saved searches that can become alerts
• Verify you have admin or alert management permissions in Splunk
• Check that email settings are configured in Settings > Email Settings if using email actions

Steps

1. Create or open an existing saved search in Splunk Web and click "Save As" > "Alert"
2. Set the alert trigger condition: choose "Per Result", "On Schedule", or "Real-time" based on your use case
3. Configure throttling by setting alert.throttle.period (e.g., 5m, 1h) to suppress duplicate alerts within that window
4. Add a throttling condition using alert.throttle.key field to group alerts — commonly use host or sourcetype
5. Click "Add Actions" and select action type: Email, Webhook, PagerDuty, Splunk Add-on for specific integrations
6. For email: enter recipients, subject (use tokens like $result.field$), and message body with context
7. For webhooks: provide URL and select POST/GET method; use webhook payload editor to format JSON with alert data
8. Set "Send Actions" to trigger on "Alert" (when threshold met) or "Digest" (periodic summary)
9. Click "Save Alert" and test by running the search manually or forcing a trigger

Code

# Step 1: Create a saved search that will become an alert
index=main source=app.log severity=ERROR
| stats count by host, error_code
| where count > 5

# Step 2: Add throttling configuration to prevent alert spam
# In the Alert Settings dialog, set these parameters:
# alert.throttle.period = 5m (suppress same alert for 5 minutes)
# alert.throttle.key = host (group alerts by hostname)

# Step 3: Webhook action payload (JSON format in Splunk UI)
{
  "alert_name": "$search_name$",
  "severity": "high",
  "timestamp": "$timestamp$",
  "result_count": "$result.count$",
  "host": "$result.host$",
  "error_code": "$result.error_code$",
  "search_url": "$search_url$"
}

# Step 4: Email action with rich formatting
Subject: [ALERT] $search_name$ - Error spike on $result.host$

Body:
Alert triggered at $timestamp$

Details:
- Host: $result.host$
- Error Count: $result.count$
- Error Code: $result.error_code$
- Severity: High

Search Link: $search_url$

---

# Step 5: REST API call to create alert via CLI (alternative method)
curl -k -u admin:

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change