Graylog Integration

Overview

Graylog is a powerful open-source log management platform designed for collecting, indexing and analysing large volumes of machine data. By integrating ipinsights.io you can automatically enrich every log message that contains an IP address with threat intelligence — including reputation scores, geolocation, ASN data, Tor/proxy/VPN detection and blocklist membership — giving your analysts instant context without leaving the Graylog interface.

The integration uses Graylog's native Lookup Tables and Processing Pipelines — no external plugins or sidecar agents required. Once configured, every matching message is transparently enriched as it is ingested.

Architecture Overview

The diagram below shows how data flows through the integration:

1. A log message arrives via any Graylog input (Syslog, GELF, Beats, Raw/Plaintext, etc.).
2. A pipeline rule checks whether the message contains an IP field (e.g. src_ip).
3. The rule calls a lookup table backed by an HTTP JSONPath data adapter, which queries the ipinsights.io API.
4. An in-memory cache prevents duplicate API calls for the same IP within the configured TTL.
5. The pipeline rule writes the enriched fields (threat score, country, ISP, etc.) directly onto the message.

Prerequisites

• Graylog 4.x or 5.x (Open, Operations, or Security edition)
• An ipinsights.io API key — available on your profile page (or register for free)
• Admin access to the Graylog web interface (required to create lookup tables and pipelines)
• Outbound HTTPS (port 443) access from the Graylog server to https://ipinsights.io

Step 1 — Create an HTTP JSONPath Data Adapter

Navigate to System → Lookup Tables → Data Adapters and click Create Data Adapter. Select HTTP JSONPath as the adapter type and configure it as follows:

Adapter Configuration

Custom HTTP Headers

Add a custom HTTP header to authenticate with the ipinsights.io API:

Replace your_api_key_here with the API key from your profile page.

Step 2 — Create a Cache

Navigate to System → Lookup Tables → Caches and click Create Cache. Select Node-local, in-memory cache and configure:

The cache prevents duplicate API calls for the same IP within the TTL window. For high-volume environments, you may increase Maximum entries to 50000 or higher, depending on available memory.

Step 3 — Create a Lookup Table

Navigate to System → Lookup Tables and click Create Lookup Table. This ties the data adapter and cache together into a single reusable component:

The default single value of 0 ensures that messages for which the lookup fails still receive a threat score, making downstream pipeline rules simpler.

Step 4 — Create a Pipeline Rule

Navigate to System → Pipelines, create a new pipeline called IPInsights Threat Enrichment, then add a Stage 0 rule. Click Edit Rule and paste the following:

Optional — Enrich Destination IPs

If your logs also contain a dst_ip field, add a second rule in the same stage:

You can adapt the field names (src_ip, dst_ip) to match whatever your extractors or inputs produce. Common alternatives include source_ip, remote_addr and client_ip.

Step 5 — Attach Pipeline to a Stream

Navigate to System → Pipelines and connect the IPInsights Threat Enrichment pipeline to a stream:

1. Click Manage pipelines next to the All messages stream (or a custom stream that receives the logs you want to enrich).
2. Select IPInsights Threat Enrichment from the available pipelines and add it.
3. Save and confirm. New messages flowing through that stream will now be enriched automatically.

If you only want to enrich specific log sources, create a dedicated stream with matching rules (e.g. source: firewall-*) and attach the pipeline there instead.

Step 6 — Create Dashboard Widgets

Once messages are being enriched, you can visualise the threat data using Graylog dashboards. Here are a few useful widget ideas:

High-Risk IPs (threat_score ≥ 75)

Create a Search with the following query and pin it to a dashboard:

Add an Aggregation widget grouped by src_ip and visualised as a table or bar chart to see which IPs are the most threatening.

Tor / Proxy / VPN Traffic

Top Countries by Message Count

Create a Quick Values widget for the ip_country field to see a geographic breakdown of your log traffic.

Verification

After completing the steps above, verify the integration is working:

1. Test the data adapter — go to System → Lookup Tables → Data Adapters, find IPInsights Threat Intel and use the Test button. Enter a known IP address (e.g. 8.8.8.8) and verify that a JSON response is returned.
2. Test the lookup table — go to System → Lookup Tables, select IPInsights Threat Intel Lookup, enter the same test IP and confirm that single-value and multi-value results appear.
3. Send a test message — use the Graylog GELF TCP/UDP input to send a message containing a src_ip field:
   echo '{"version":"1.1","host":"test","short_message":"test enrichment","src_ip":"185.220.101.1"}' \ | nc -w1 -u localhost 12201
4. Inspect enriched fields — search for the test message in the Graylog web interface and expand it. You should see the new fields: threat_score, threat_level, ip_country, ip_isp, etc.
5. Check pipeline metrics — go to System → Pipelines and verify that the pipeline shows a non-zero execution count and no errors.

Best Practices

• Tune cache TTLs — the default 1-hour cache TTL works well for most environments. For high-change environments (e.g. incident response) you may lower it to 900 seconds (15 min). For stable environments, 7200 seconds (2 hours) reduces API calls further.
• Be aware of rate limits — the default free-tier limit is 600 requests per hour. The in-memory cache drastically reduces actual API calls, but in high-volume environments you should monitor cache hit ratios via System → Lookup Tables. If you need a higher limit, use the form below.
• Handle errors gracefully in pipeline rules — if the API is temporarily unreachable, the lookup returns the default value (0). Avoid creating alert rules that trigger on threat_score:0 to prevent false positives during outages.
• Exclude internal/private IPs — add a condition to your pipeline rule to skip RFC 1918 addresses and avoid unnecessary lookups:
  rule "ipinsights — enrich src_ip" when has_field("src_ip") AND NOT cidr_match("10.0.0.0/8", to_ip($message.src_ip)) AND NOT cidr_match("172.16.0.0/12", to_ip($message.src_ip)) AND NOT cidr_match("192.168.0.0/16", to_ip($message.src_ip)) then // … enrichment logic … end
• Secure your API key — Graylog stores data adapter configuration (including HTTP headers) in its database. Restrict Graylog admin access and rotate your API key regularly via your profile page.
• Index enrichment fields — for fast searching, ensure the enriched fields (threat_score, ip_country, is_tor, etc.) are included in your Elasticsearch/OpenSearch index mappings. Graylog typically handles this automatically for new fields.

Troubleshooting

Data adapter returns empty results

• Verify the Lookup URL includes ${key} as the IP parameter — a literal IP address will not work.
• Confirm your API key is correct and active on your profile page.
• Test connectivity from the Graylog server: curl -H "X-API-Key: YOUR_KEY" "https://ipinsights.io/api/v1/lookup?ip=8.8.8.8"
• Check the Graylog server log (/var/log/graylog-server/server.log) for HTTP errors.

Enrichment fields not appearing on messages

• Confirm the pipeline is connected to the correct stream under System → Pipelines.
• Verify the pipeline stage order — enrichment should run at Stage 0 (or before any filtering stages).
• Check that your messages actually contain a src_ip field (or whichever field your rule expects). Use the message detail view to inspect available fields.
• Look for pipeline errors in System → Pipelines → Simulation.

HTTP 429 — Rate limit exceeded

• Increase the cache TTL or maximum entries to improve cache hit rates.
• Add private IP exclusions to your pipeline rule (see Best Practices above).
• Request a higher API limit using the form below.
• Check cache statistics under System → Lookup Tables → Caches to verify the cache is functioning.

Pipeline rule compilation errors

• Verify the lookup table name in your rule exactly matches the Name field (not the Title) — it should be ipinsights_threat_intel.
• Ensure you are using to_string() to cast the IP field before passing it to lookup().
• Check for typos in field access paths like threat_data["threat_assessment"]["level"].

Graylog server performance degradation

• Monitor the data adapter response time in System → Lookup Tables → Data Adapters.
• Increase cache TTL and max entries to minimise external HTTP calls.
• Consider attaching the pipeline only to specific high-value streams rather than All messages.
• If latency persists, reduce the HTTP timeout on the data adapter (default is 10 seconds).

API Key: You can find your API key on your profile page. Don't have an account yet? Register for free.

Request Higher API Limit

Running a high-volume Graylog deployment? If the default rate limit isn't enough for your environment, submit a request below and we'll review it.