Loading

Web Server Suspicious User Agent Requests

This rule detects unusual spikes in web server requests with uncommon or suspicious user-agent strings. Such activity may indicate reconnaissance attempts by attackers trying to identify vulnerabilities in web applications or servers. These user-agents are often associated with automated tools used for scanning, vulnerability assessment, or brute-force attacks.

Rule type: esql
Rule indices:

Rule Severity: low
Risk Score: 21
Runs every: 10m
Searches indices from: now-9m
Maximum alerts per execution: ?
References:

Tags:

  • Domain: Web
  • Domain: Network
  • Use Case: Threat Detection
  • Tactic: Reconnaissance
  • Tactic: Credential Access
  • Data Source: Network Packet Capture
  • Data Source: Nginx
  • Data Source: Apache
  • Data Source: Apache Tomcat
  • Data Source: IIS
  • Resources: Investigation Guide

Version: ?
Rule authors:

  • Elastic

Rule license: Elastic License v2

Disclaimer: This investigation guide was created using generative AI technology and has been reviewed to improve its accuracy and relevance. While every effort has been made to ensure its quality, we recommend validating the content and adapting it to suit your specific environment and operational needs.

This rule flags surges of web requests that advertise scanner or brute-force tool user agents, signaling active reconnaissance against your web servers and applications. A common pattern is dirsearch or gobuster sweeping for hidden paths, firing hundreds of rapid GETs across diverse URLs from one host and probing admin panels, backup folders, and robots.txt.

  • Verify whether the activity aligns with approved scanners or uptime checks by cross-referencing inventories, allowlists, change windows, and egress ranges; otherwise enrich the originating IP with ASN, geolocation, and threat reputation to gauge risk.
  • Sample representative requests to identify targeted paths and payloads (e.g., admin panels, .git/.env, backups, traversal, SQLi/XSS markers) and note any successful responses or downloads that indicate information exposure.
  • Analyze request rate, methods, and status-code distribution to separate noisy recon from successful discovery or brute-force patterns, highlighting any POST/PUT with nontrivial bodies.
  • Correlate the same client across hosts and security layers (application/auth logs, WAF/CDN, IDS) to determine whether it is scanning multiple services, triggering signatures, or attempting credential stuffing.
  • Assess user-agent authenticity and evasiveness by comparing HTTP header order/values and TLS fingerprints (JA3/JA4) to expected clients, and verify true client identity via forwarded-for headers if behind a proxy or CDN.
  • Legitimate, scheduled vulnerability assessments by internal teams (e.g., Nessus, Nikto, or OpenVAS) can generate large volumes of requests with those user-agent strings across many paths.
  • Developer or QA testing using discovery/fuzzing or intercept-proxy tools (Dirsearch, Gobuster, Ffuf, Burp, or OWASP ZAP) may unintentionally target production hosts, producing a short-lived spike with diverse URLs.
  • Immediately contain by blocking or rate-limiting the originating IPs at the WAF/CDN and edge firewall, and add temporary rules to drop or challenge requests that advertise tool user agents such as "nikto", "sqlmap", "dirsearch", "wpscan", "gobuster", or "burp".
  • If traffic is proxied (CDN/reverse proxy), identify the true client via forwarded headers and extend blocks at both layers, enabling bot management or JS challenges on swept paths like /admin, /.git, /.env, /backup, and common discovery endpoints.
  • Eradicate exposure by removing or restricting access to sensitive files and directories uncovered by the scans, rotating any credentials or API keys found, invalidating active sessions, and disabling public access to administrative panels until hardened.
  • Recover by verifying no unauthorized changes or data exfiltration occurred, tuning per-IP and per-path rate limits to prevent path-sweeps while preserving legitimate traffic, and reintroducing normal rules only after fixes are deployed and stability is confirmed.
  • Escalate to incident response if sensitive files are successfully downloaded (HTTP 200/206 on /.git, /.env, or backups), any login or account creation succeeds, multiple hosts or environments are targeted, or activity persists after blocking via UA spoofing or rapid IP rotation.
  • Harden long term by enforcing WAF signatures for known scanner UAs and path patterns, denying directory listing and direct access to /.git, /.env, /backup and similar artifacts, requiring MFA/VPN for /admin and management APIs, and deploying auto-ban controls like fail2ban or mod_security.
from logs-network_traffic.http-*, logs-network_traffic.tls-*, logs-nginx.access-*, logs-apache.access-*, logs-apache_tomcat.access-*, logs-iis.access-*

| eval Esql.user_agent_original_lower = to_lower(user_agent.original)

| where
    (url.original is not null or url.full is not null) and
  (
		Esql.user_agent_original_lower like "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/74.0.3729.169 safari/537.36" or
		Esql.user_agent_original_lower like "nikto*" or
		Esql.user_agent_original_lower like "mozilla/4.0 (compatible; msie 8.0; windows nt 5.1; trident/4.0)" or
        Esql.user_agent_original_lower like "*nessus*" or
		Esql.user_agent_original_lower like "sqlmap/*" or
		Esql.user_agent_original_lower like "wpscan*" or
		Esql.user_agent_original_lower like "feroxbuster/*" or
		Esql.user_agent_original_lower like "masscan*" or
		Esql.user_agent_original_lower like "fuzz*" or
		Esql.user_agent_original_lower like "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/user_agent.original like~ 87.0.4280.88 safari/537.36" or
		Esql.user_agent_original_lower like "mozilla/4.0 (compatible; msie 6.0; windows nt 5.1)" or
		Esql.user_agent_original_lower like "dirbuster*" or
		Esql.user_agent_original_lower like "gobuster/*" or
        Esql.user_agent_original_lower like "*dirsearch*" or
		Esql.user_agent_original_lower like "*nmap*" or
        Esql.user_agent_original_lower like "*hydra*" or
        Esql.user_agent_original_lower like "*w3af*" or
        Esql.user_agent_original_lower like "*arachni*" or
        Esql.user_agent_original_lower like "*skipfish*" or
        Esql.user_agent_original_lower like "*openvas*" or
        Esql.user_agent_original_lower like "*acunetix*" or
        Esql.user_agent_original_lower like "*zap*" or
        Esql.user_agent_original_lower like "*burp*"
  )

| eval Esql.url_text  = case(url.original is not null, url.original, url.full)
| eval Esql.url_lower = to_lower(Esql.url_text)

| keep
    @timestamp,
    event.dataset,
    user_agent.original,
    source.ip,
    agent.id,
    host.name,
	Esql.url_lower,
    Esql.user_agent_original_lower
| stats
    Esql.event_count = count(),
    Esql.url_path_count_distinct = count_distinct(Esql.url_lower),
    Esql.host_name_values = values(host.name),
    Esql.agent_id_values = values(agent.id),
    Esql.url_path_values = values(Esql.url_lower),
    Esql.user_agent_original_values = values(Esql.user_agent_original_lower),
    Esql.event_dataset_values = values(event.dataset)
    by source.ip, agent.id
| where
    Esql.event_count > 50 and Esql.url_path_count_distinct > 10
  1. Nikto
  2. Nikto
  3. Nessus Vulnerability Scanner
  4. Nessus Vulnerability Scanner
  5. SQLMap
  6. WPScan
  7. Feroxbuster
  8. Masscan & masscan-ng
  9. Ffuf
  10. Dirsearch
  11. Dirb
  12. Dirbuster
  13. Gobuster
  14. dirsearch
  15. Nmap Scripting Engine
  16. Hydra Brute Forcer
  17. w3af Web Application Attack and Audit Framework
  18. Arachni Web Application Security Scanner
  19. Skipfish Web Application Security Scanner
  20. OpenVAS Vulnerability Scanner
  21. Acunetix Vulnerability Scanner
  22. OWASP ZAP
  23. Burp Suite

Framework: MITRE ATT&CK

Framework: MITRE ATT&CK