curben
/
blog
mirror of https://gitlab.com/curben/blog
1
0
Fork 0
Code Issues Releases Wiki Activity

post(phishing-filter): clarify about DShield source

This commit is contained in:
MDLeom 2020-07-11 09:20:54 +01:00
parent 1eff521182
commit 6d66e4ee66
No known key found for this signature in database
GPG Key ID: 5D9DB57A25D34EE3
1 changed files with 4 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
source/_posts/phishing-filter.md
View File

@ -2,14 +2,16 @@
title: Block phishing websites with phishing-filter title: Block phishing websites with phishing-filter
excerpt: Many formats available excerpt: Many formats available
date: 2020-07-07 date: 2020-07-07
updated: 2020-07-08 updated: 2020-07-11
tags: tags:
- security - security
--- ---
> Skip to [phishing-filter](#phishing-filter) section > Skip to [phishing-filter](#phishing-filter) section
Recently I switched Firefox's Android app from Preview Nightly to Nightly after the former has been [deprecated](https://old.reddit.com/r/Android/comments/hk37jl/firefox_preview_has_been_merged_into_firefox/). The switch entailed migrating the configurations; a config I need to migrate over is DNS-over-HTTPS (DoH). I verified the Quad9's DoH address through its [instruction page](https://www.quad9.net/doh-quad9-dns-servers/) (tips: you can use "https://9.9.9.9/dns-query" instead of "https://dns.quad9.net/dns-query" so that a browser doesn't need to query the IP behind dns.quad9.net). I also checked out its [recent article](https://quad9.net/dns-blocking-effectiveness-recent-independent-tests/) that talks about how effective it blocks malicious and phishing websites (via DNS-blocking) compared to other well-known DNS service, like Cloudflare and OpenDNS. According to this [replication test](https://www.andryou.com/2020/05/31/comparing-malware-blocking-dns-resolvers-redux/), the effectiveness was measured based on the [DShield.org Suspicious Domain List](https://isc.sans.edu/suspicious_domains.html), which in turn was based on [PhishTank](https://www.phishtank.com/) and [URLhaus](https://urlhaus.abuse.ch/) lists. Recently I switched Firefox's Android app from Preview Nightly to Nightly after the former has been [deprecated](https://old.reddit.com/r/Android/comments/hk37jl/firefox_preview_has_been_merged_into_firefox/). The switch entailed migrating the configurations; a config I need to migrate over is DNS-over-HTTPS (DoH). I verified the Quad9's DoH address through its [instruction page](https://www.quad9.net/doh-quad9-dns-servers/) (tips: you can use "https://9.9.9.9/dns-query" instead of "https://dns.quad9.net/dns-query" so that a browser doesn't need to query the IP behind dns.quad9.net). I also checked out its [recent article](https://quad9.net/dns-blocking-effectiveness-recent-independent-tests/) that mentioned it blocks more malicious and phishing websites (via DNS-blocking) compared to other well-known DNS service--like Cloudflare and OpenDNS--according to an [independent test](https://forums.lawrencesystems.com/t/dns-malware-filtering-compared-quad9-vs-cloudflare-vs-dns-filter-vs-opendns-cisco-umbrella/5072). The test was measured based on [DShield.org Suspicious Domain List](https://isc.sans.edu/suspicious_domains.html).
(Edit) DShield.org dataset [previously](https://web.archive.org/web/20200528232041/https://isc.sans.edu/suspicious_domains.html) did not sourced from [PhishTank](https://www.phishtank.com/) and [URLhaus](https://urlhaus.abuse.ch/) datasets and were not actually used in that DNS test. DShield later [switched](https://web.archive.org/web/20200612030447/https://isc.sans.edu/suspicious_domains.html) after it found out the previous dataset was outdated. In light of this discovery, I conduct a DNS filtering test using datasets sourced from URLhaus and PhishTank, refer to {% post_link dns-filtering 'this post' %} for result.
I was intrigued by the DShield list as I created a blocklist ([urlhaus-filter](https://gitlab.com/curben/urlhaus-filter)) that is also based on URLhaus. I then checked out its another source, the PhishTank list. PhishTank operates similarly to URLhaus, the links are user-submitted. User can vote on submitted links (of other users') are indeed phishing websites. The database is available in [various formats](https://www.phishtank.com/developer_info.php) including CSV. This seemed ideal to be processed into a blocklist, just like what I did in urlhaus-filter. To avoid duplicate effort, I did a search on FilterLists and there is a domain-based blocklist ("[Phishing Bad Sites](https://filterlists.com/lists/phishing-bad-sites)") that is based on PhishTank. I was intrigued by the DShield list as I created a blocklist ([urlhaus-filter](https://gitlab.com/curben/urlhaus-filter)) that is also based on URLhaus. I then checked out its another source, the PhishTank list. PhishTank operates similarly to URLhaus, the links are user-submitted. User can vote on submitted links (of other users') are indeed phishing websites. The database is available in [various formats](https://www.phishtank.com/developer_info.php) including CSV. This seemed ideal to be processed into a blocklist, just like what I did in urlhaus-filter. To avoid duplicate effort, I did a search on FilterLists and there is a domain-based blocklist ("[Phishing Bad Sites](https://filterlists.com/lists/phishing-bad-sites)") that is based on PhishTank.