mirror of
https://github.com/fabriziosalmi/caddy-waf.git
synced 2025-12-23 22:27:46 -05:00
Update prometheus.md
Updated to match real world scenario :)
This commit is contained in:
fab
@@ -1,247 +1,152 @@
|
||||
## Your WAF log with Prometheus and Grafana
|
||||
# **Caddy WAF, Prometheus and Grafana**
|
||||
|
||||
### **Introduction**
|
||||
Monitor your **caddy-waf** performance and security in real-time with **Prometheus** and **Grafana**. Track key metrics like allowed/blocked requests, rule hits (e.g., "block-scanners", "sql-injection", "xss-attacks", browser integrity checks), and more, to understand your WAF's effectiveness against threats.
|
||||
|
||||
If you're running a **Caddy Web Application Firewall (WAF)**, you likely want to monitor its performance and security metrics in real-time. Metrics like the number of allowed/blocked requests, rule hits, and other key indicators can help you understand how your WAF is performing and identify potential threats.
|
||||
|
||||
To achieve this, you can use **Prometheus** (a powerful time-series database for metrics) and **Grafana** (a visualization tool) to collect, store, and visualize your WAF metrics. However, Prometheus requires metrics to be exposed in a specific format. Since your Caddy WAF already exposes metrics in JSON format via the `/waf_metrics` endpoint, we need a way to convert this JSON data into a format that Prometheus can scrape.
|
||||
|
||||
This is where a **Prometheus exporter** comes in. An exporter acts as a bridge between your application and Prometheus, converting your application's metrics into Prometheus-compatible metrics.
|
||||
|
||||
In this step-by-step guide, I'll walk you through creating and running a **Prometheus exporter** for your Caddy WAF. By the end, you'll have a fully functional exporter that scrapes your `/waf_metrics` endpoint, converts the JSON data into Prometheus metrics, and exposes them for scraping.
|
||||
|
||||
---
|
||||
|
||||
### **What You'll Learn**
|
||||
|
||||
1. How to create a **Prometheus exporter** in Python.
|
||||
2. How to configure **Prometheus** to scrape metrics from the exporter.
|
||||
3. How to set up **Grafana** to visualize your WAF metrics.
|
||||
4. How to run the exporter as a **systemd service** for continuous operation.
|
||||
|
||||
---
|
||||
|
||||
### **Prerequisites**
|
||||
|
||||
Before we begin, ensure you have the following:
|
||||
|
||||
1. **Caddy WAF** running and exposing the `/waf_metrics` endpoint.
|
||||
2. **Python 3.x** installed on your system.
|
||||
3. **Prometheus** installed and running.
|
||||
4. **Grafana** installed and running.
|
||||
5. Basic familiarity with the command line and Python.
|
||||
|
||||
---
|
||||
|
||||
### **Step-by-Step Guide**
|
||||
|
||||
Now that we've set the stage, let’s dive into the step-by-step guide! 🚀
|
||||
|
||||
---
|
||||
This guide helps you create a **Prometheus exporter** to bridge Caddy WAF's JSON metrics (from `/waf_metrics`) to Prometheus's format. You'll then visualize these metrics in Grafana dashboards for actionable insights.
|
||||
|
||||
### **Step 1: Set Up Your Environment**
|
||||
|
||||
1. **Install Python**:
|
||||
- Ensure Python 3.x is installed on your system. You can download it from [python.org](https://www.python.org/).
|
||||
1. **Install Python**: Get Python 3.x from [python.org](https://www.python.org/).
|
||||
|
||||
2. **Install Required Libraries**:
|
||||
- Install the `prometheus-client` and `requests` libraries using pip:
|
||||
```bash
|
||||
pip install prometheus-client requests
|
||||
```
|
||||
2. **Install Libraries**:
|
||||
```bash
|
||||
pip install prometheus-client requests
|
||||
```
|
||||
|
||||
---
|
||||
### **Step 2: Create the Exporter Script (exporter.py)**
|
||||
|
||||
### **Step 2: Create the Exporter Script**
|
||||
```python
|
||||
from prometheus_client import start_http_server, Counter, Gauge
|
||||
import requests
|
||||
import time
|
||||
import json
|
||||
|
||||
1. Create a new file named `exporter.py`:
|
||||
# Define Prometheus metrics
|
||||
TOTAL_REQUESTS = Counter('caddywaf_total_requests', 'Total requests processed')
|
||||
BLOCKED_REQUESTS = Counter('caddywaf_blocked_requests', 'Total requests blocked')
|
||||
ALLOWED_REQUESTS = Counter('caddywaf_allowed_requests', 'Total requests allowed')
|
||||
RULE_HITS = Counter('caddywaf_rule_hits', 'Hits per WAF rule', ['rule_id'])
|
||||
RULE_HITS_BY_PHASE = Counter('caddywaf_rule_hits_by_phase', 'Rule hits by phase', ['phase'])
|
||||
DNS_BLACKLIST_HITS = Counter('caddywaf_dns_blacklist_hits', 'DNS blacklist hits')
|
||||
GEOIP_BLOCKED = Counter('caddywaf_geoip_blocked', 'Blocked by GeoIP')
|
||||
IP_BLACKLIST_HITS = Counter('caddywaf_ip_blacklist_hits', 'IP blacklist hits')
|
||||
RATE_LIMITER_BLOCKED_REQUESTS = Counter('caddywaf_rate_limiter_blocked_requests', 'Rate limiter blocked')
|
||||
RATE_LIMITER_REQUESTS = Counter('caddywaf_rate_limiter_requests', 'Rate limiter requests')
|
||||
WAF_VERSION = Gauge('caddywaf_version', 'WAF version', ['version'])
|
||||
|
||||
```python
|
||||
from prometheus_client import start_http_server, Counter, Gauge
|
||||
import requests
|
||||
import time
|
||||
def fetch_metrics():
|
||||
try:
|
||||
response = requests.get("http://localhost:8080/waf_metrics")
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
# Define Prometheus metrics
|
||||
TOTAL_REQUESTS = Counter('caddywaf_total_requests', 'Total number of requests processed by the WAF')
|
||||
BLOCKED_REQUESTS = Counter('caddywaf_blocked_requests', 'Total number of requests blocked by the WAF')
|
||||
ALLOWED_REQUESTS = Counter('caddywaf_allowed_requests', 'Total number of requests allowed by the WAF')
|
||||
RULE_HITS = Counter('caddywaf_rule_hits', 'Total number of hits per WAF rule', ['rule_id'])
|
||||
TOTAL_REQUESTS.inc(data["total_requests"])
|
||||
BLOCKED_REQUESTS.inc(data["blocked_requests"])
|
||||
ALLOWED_REQUESTS.inc(data["allowed_requests"])
|
||||
DNS_BLACKLIST_HITS.inc(data["dns_blacklist_hits"])
|
||||
GEOIP_BLOCKED.inc(data["geoip_blocked"])
|
||||
IP_BLACKLIST_HITS.inc(data["ip_blacklist_hits"])
|
||||
RATE_LIMITER_BLOCKED_REQUESTS.inc(data["rate_limiter_blocked_requests"])
|
||||
RATE_LIMITER_REQUESTS.inc(data["rate_limiter_requests"])
|
||||
WAF_VERSION.labels(version=data["version"]).set(1)
|
||||
|
||||
def fetch_metrics():
|
||||
# Fetch metrics from the Caddy WAF /waf_metrics endpoint
|
||||
response = requests.get("http://localhost:8080/waf_metrics")
|
||||
data = response.json()
|
||||
for rule_id, hits in data["rule_hits"].items():
|
||||
RULE_HITS.labels(rule_id=rule_id).inc(hits)
|
||||
|
||||
# Update Prometheus metrics
|
||||
TOTAL_REQUESTS.inc(data["total_requests"])
|
||||
BLOCKED_REQUESTS.inc(data["blocked_requests"])
|
||||
ALLOWED_REQUESTS.inc(data["allowed_requests"])
|
||||
if "rule_hits_by_phase" in data:
|
||||
for phase, hits in data["rule_hits_by_phase"].items():
|
||||
RULE_HITS_BY_PHASE.labels(phase=phase).inc(hits)
|
||||
|
||||
for rule_id, hits in data["rule_hits"].items():
|
||||
RULE_HITS.labels(rule_id=rule_id).inc(hits)
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"Error fetching metrics: {e}")
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"JSON Decode Error: {e}")
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Start the Prometheus HTTP server on port 8000
|
||||
start_http_server(8000)
|
||||
print("Exporter started on http://localhost:8000/metrics")
|
||||
|
||||
# Fetch metrics every 10 seconds
|
||||
while True:
|
||||
fetch_metrics()
|
||||
time.sleep(10)
|
||||
```
|
||||
|
||||
---
|
||||
if __name__ == '__main__':
|
||||
start_http_server(8000)
|
||||
print("Exporter started on http://localhost:8000/metrics")
|
||||
while True:
|
||||
fetch_metrics()
|
||||
time.sleep(10)
|
||||
```
|
||||
|
||||
### **Step 3: Run the Exporter**
|
||||
|
||||
1. Start the exporter:
|
||||
```bash
|
||||
python exporter.py
|
||||
```
|
||||
1. Start: `python exporter.py`
|
||||
2. Verify: `http://localhost:8000/metrics` in browser. Check for Prometheus format.
|
||||
|
||||
2. Verify that the exporter is running:
|
||||
- Open your browser and go to `http://localhost:8000/metrics`.
|
||||
- You should see Prometheus-formatted metrics like this:
|
||||
```
|
||||
# HELP caddywaf_total_requests Total number of requests processed by the WAF.
|
||||
# TYPE caddywaf_total_requests counter
|
||||
caddywaf_total_requests 212
|
||||
|
||||
# HELP caddywaf_blocked_requests Total number of requests blocked by the WAF.
|
||||
# TYPE caddywaf_blocked_requests counter
|
||||
caddywaf_blocked_requests 166
|
||||
|
||||
# HELP caddywaf_allowed_requests Total number of requests allowed by the WAF.
|
||||
# TYPE caddywaf_allowed_requests counter
|
||||
caddywaf_allowed_requests 46
|
||||
|
||||
# HELP caddywaf_rule_hits Total number of hits per WAF rule.
|
||||
# TYPE caddywaf_rule_hits counter
|
||||
caddywaf_rule_hits{rule_id="942440"} 1
|
||||
caddywaf_rule_hits{rule_id="block-scanners"} 13
|
||||
```
|
||||
|
||||
---
|
||||
Example output snippet:
|
||||
```
|
||||
# HELP caddywaf_rule_hits_by_phase Rule hits by phase
|
||||
# TYPE caddywaf_rule_hits_by_phase counter
|
||||
caddywaf_rule_hits_by_phase{phase="1"} 1461
|
||||
caddywaf_rule_hits_by_phase{phase="2"} 705
|
||||
```
|
||||
|
||||
### **Step 4: Configure Prometheus**
|
||||
|
||||
1. **Install Prometheus**:
|
||||
- Download Prometheus from [prometheus.io](https://prometheus.io/download/).
|
||||
- Extract the archive and navigate to the Prometheus directory.
|
||||
1. Install: [prometheus.io/download/](https://prometheus.io/download/)
|
||||
2. Edit `prometheus.yml`:
|
||||
|
||||
2. **Edit `prometheus.yml`**:
|
||||
- Open the `prometheus.yml` file in a text editor.
|
||||
- Add a scrape configuration for the exporter:
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: 'caddywaf_exporter'
|
||||
static_configs:
|
||||
- targets: ['localhost:8000']
|
||||
```
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: 'caddywaf_exporter'
|
||||
static_configs:
|
||||
- targets: ['localhost:8000']
|
||||
```
|
||||
|
||||
3. **Start Prometheus**:
|
||||
- Run Prometheus:
|
||||
```bash
|
||||
./prometheus --config.file=prometheus.yml
|
||||
```
|
||||
|
||||
4. **Verify Prometheus**:
|
||||
- Open your browser and go to `http://localhost:9090`.
|
||||
- In the Prometheus UI, go to **Status > Targets** and verify that the exporter is listed as "UP".
|
||||
|
||||
---
|
||||
3. Start: `./prometheus --config.file=prometheus.yml`
|
||||
4. Verify: Prometheus UI (`http://localhost:9090`) > Status > Targets > `caddywaf_exporter` should be UP.
|
||||
|
||||
### **Step 5: Set Up Grafana**
|
||||
|
||||
1. **Install Grafana**:
|
||||
- Download Grafana from [grafana.com](https://grafana.com/grafana/download).
|
||||
- Follow the installation instructions for your operating system.
|
||||
1. Install: [grafana.com/grafana/download](https://grafana.com/grafana/download)
|
||||
2. Start: `http://localhost:3000` (login: `admin/admin`)
|
||||
3. Add Data Source: Configuration > Data Sources > Add data source > Prometheus. URL: `http://localhost:9090`. Save & Test.
|
||||
4. Create Dashboard: Create > Dashboard > Add panel. Example queries:
|
||||
|
||||
2. **Start Grafana**:
|
||||
- Start the Grafana server:
|
||||
```bash
|
||||
systemctl start grafana-server
|
||||
```
|
||||
- Open your browser and go to `http://localhost:3000`.
|
||||
* **Total Requests:** `sum(rate(caddywaf_total_requests[1m]))`
|
||||
* **Blocked Requests:** `sum(rate(caddywaf_blocked_requests[1m]))`
|
||||
* **Top Rule Hits:** `topk(10, sum by (rule_id) (rate(caddywaf_rule_hits[1m])))`
|
||||
* **Rule Hits by Phase:** `sum by (phase) (rate(caddywaf_rule_hits_by_phase[1m]))`
|
||||
* **WAF Version:** `caddywaf_version`
|
||||
|
||||
3. **Add Prometheus as a Data Source**:
|
||||
- Log in to Grafana (default username/password: `admin/admin`).
|
||||
- Go to **Configuration > Data Sources**.
|
||||
- Click **Add data source** and select **Prometheus**.
|
||||
- Set the URL to `http://localhost:9090` and click **Save & Test**.
|
||||
|
||||
4. **Create a Dashboard**:
|
||||
- Go to **Create > Dashboard**.
|
||||
- Add a new panel and use the following queries:
|
||||
- `caddywaf_total_requests`
|
||||
- `caddywaf_blocked_requests`
|
||||
- `caddywaf_allowed_requests`
|
||||
- `caddywaf_rule_hits`
|
||||
- Customize the panels (e.g., use graphs, gauges, or tables).
|
||||
|
||||
---
|
||||
Customize dashboards as needed.
|
||||
|
||||
### **Step 6: Run Everything Together**
|
||||
|
||||
1. **Start Caddy WAF**:
|
||||
- Ensure your Caddy WAF is running and exposing the `/waf_metrics` endpoint.
|
||||
|
||||
2. **Start the Exporter**:
|
||||
- Run the exporter script:
|
||||
```bash
|
||||
python exporter.py
|
||||
```
|
||||
|
||||
3. **Start Prometheus**:
|
||||
- Run Prometheus with the updated configuration.
|
||||
|
||||
4. **Start Grafana**:
|
||||
- Ensure Grafana is running and configured to use Prometheus as a data source.
|
||||
|
||||
5. **Visualize Metrics**:
|
||||
- Use Grafana to create dashboards and visualize your Caddy WAF metrics.
|
||||
1. Start Caddy WAF (`/waf_metrics` accessible).
|
||||
2. Start Exporter: `python exporter.py`
|
||||
3. Start Prometheus (with config).
|
||||
4. Start Grafana (connected to Prometheus).
|
||||
5. Visualize metrics in Grafana.
|
||||
|
||||
---
|
||||
|
||||
### **Optional: Run the Exporter as a Service**
|
||||
### **Optional: Exporter as Service (systemd)**
|
||||
|
||||
To ensure the exporter runs continuously, you can set it up as a systemd service.
|
||||
1. Create: `sudo nano /etc/systemd/system/caddywaf-exporter.service`
|
||||
|
||||
1. Create a service file:
|
||||
```bash
|
||||
sudo nano /etc/systemd/system/caddywaf-exporter.service
|
||||
```
|
||||
2. Content:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Caddy WAF Prometheus Exporter
|
||||
After=network.target
|
||||
|
||||
2. Add the following content:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Caddy WAF Prometheus Exporter
|
||||
After=network.target
|
||||
[Service]
|
||||
User=your_user
|
||||
ExecStart=/usr/bin/python3 /path/to/exporter.py
|
||||
Restart=always
|
||||
|
||||
[Service]
|
||||
User=your_user
|
||||
ExecStart=/usr/bin/python3 /path/to/exporter.py
|
||||
Restart=always
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
3. Run:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start caddywaf-exporter
|
||||
sudo systemctl enable caddywaf-exporter
|
||||
```
|
||||
|
||||
3. Reload systemd and start the service:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl start caddywaf-exporter
|
||||
sudo systemctl enable caddywaf-exporter
|
||||
```
|
||||
|
||||
4. Verify the service is running:
|
||||
```bash
|
||||
sudo systemctl status caddywaf-exporter
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **Conclusion**
|
||||
|
||||
You now have a fully functional Prometheus exporter for your Caddy WAF metrics! This setup allows you to scrape, store, and visualize your WAF metrics using Prometheus and Grafana.
|
||||
4. Verify: `sudo systemctl status caddywaf-exporter`
|
||||
|
||||
Reference in New Issue
Block a user