urlhaus-filter/README.md

URLhaus Malicious URL Blocklist

This uBO-compatible filter list is based on the database dump (CSV) of Abuse.sh URLhaus.

Subscribe

Filter is updated twice a day.

Import the following URL into uBO to subcribe:

https://gitlab.com/curben/urlhaus/raw/master/urlhaus-filter.txt

Description

Following URL categories are removed from the database dump:

• Offline URLs
• Well-known domains (top-1m.txt) (using Umbrella Popularity List)
• False positives (exclude.txt)

Database dump is saved as URLhaus.csv, get processed by script.sh and output as urlhaus-filter.txt.

Note

Please report any false positive.

This filter only accepts malware URLs from URLhaus.

Please report malware URL to the upstream maintainer through https://urlhaus.abuse.ch/api/#submit.

This repo is not endorsed by Abuse.sh.

FAQ

• Can you add this very-bad-url.com to the filter?

  • No, please report to the upstream.

• Why don't you use the URLhaus "Plain-Text URL List"?

  • It doesn't show the status (online/offline) of a URL.

• Why don't you wget top-1m.csv.zip and output to stdout?

  • If wget fails, top-1m.txt will be empty. Output as file avoids that.

• Why do you need to clone the repo again in your CI? I thought CI already fetch the repo by default?

  • GitLab Runner clone/fetch the repo using HTTPS method by default (log). This method requires deploy token which is read-only (cannot push).
  • Deploy key has write access but cannot be used with the HTTPS method, hence, the workaround to clone using SSH.
  • See issue #20567 and #20845.