curben
/
phishing-filter
mirror of https://gitlab.com/malware-filter/phishing-filter
1
0
Fork 0
Code Issues Releases Wiki Activity
1,370 Commits 2 Branches 0 Tags 682 MiB
Commit Graph

166 Commits

Author SHA1 Message Date
MDLeom f07ad2ce4e
refactor: set pipefail conditionally 2024-07-15 08:02:25 +00:00
MDLeom 827342f3e9
fix: expand alias in bash 2024-06-03 08:21:56 +00:00
MDLeom 358003b782
fix: subdomains may be completely excluded 2024-05-03 11:16:01 +00:00
MDLeom 2ee0b2d661
feat(source): disable mitchellkrogza/Phishing.Database 
source does not offer online-only links
closes #86
 2024-05-02 12:00:37 +00:00
MDLeom e9ae4a9f11
refactor: replace got with fetch 2024-04-07 00:45:34 +00:00
MDLeom 607208c171
fix: check file exists and not zero size 2024-03-10 07:49:19 +00:00
MDLeom a1548a5e1c
fix: may not necessarily contain ipv4 entries 2024-03-10 03:06:51 +00:00
MDLeom 00d43e98d3
fix(exclude): click.mail.onedrive.com 2024-03-09 04:28:57 +00:00
MDLeom 6f7cf84de2
chore(exclude): cleanup entries 2024-03-09 04:16:19 +00:00
MDLeom e02ed129d5
fix(exclude): fonts.gstatic.com 2024-03-09 04:12:33 +00:00
MDLeom 5c7b1f4645
feat(source): add mitchellkrogza/Phishing.Database 
ref #40
revert e68268f506
 2024-03-09 04:06:37 +00:00
Ming Di Leom 6b681bc58f Merge branch 'exclude-s3-fix' into 'main' 
Add Amazon S3 dual-stack endpoints to exclude list

See merge request malware-filter/phishing-filter!8
 2024-03-08 08:11:27 +00:00
Alan Turing ecd739a846 fix(exclude): Add Amazon S3 dual-stack endpoints to exclude list 
and add new regions
https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html
https://docs.aws.amazon.com/general/latest/gr/s3.html
 2024-03-08 08:11:27 +00:00
MDLeom 1b2312f492
fix: "phishing-subdomains.txt" may be empty 2024-03-08 07:54:33 +00:00
MDLeom 93b85b00f9
chore: remove remaining phishunt 
no longer used since #43 #45
 2024-03-07 10:14:08 +00:00
MDLeom b3f6e90b9a
feat: remove phishtank source 
frequent interference from cloudflare captcha
 2024-03-07 10:09:32 +00:00
MDLeom 6175179162
refactor: esm 
got is esm only since v12
 2023-10-01 09:56:57 +00:00
MDLeom 07ca1adfd1
refactor: lazy load os-release 2023-05-20 11:23:07 +00:00
MDLeom 667fad0b6f
style: remove debug message 2023-05-20 11:15:29 +00:00
MDLeom 13289d3365
fix: dash does not support pipefail 2023-05-20 10:38:47 +00:00
MDLeom eac902123e
fix: check installed grep is GNU variant 2023-05-20 09:51:12 +00:00
MDLeom eebf51ac47
fix: check existent of busybox 
if dos2unix is not installed
 2023-05-20 09:44:54 +00:00
MDLeom ca23363ef4
fix: reprocess decoded safelink 
- extend 1ea3ce51f5
- also include scope of 0578e6c16a
 2023-05-20 08:20:22 +00:00
MDLeom 0578e6c16a
fix: handle URL of top domains without path 
- ref #62, #43, #44
- 745c81b134, c623542b9a, 8923941376
were not effective previously
 2023-05-19 10:34:04 +00:00
MDLeom 7dbdc85163
fix: sed syntax to recognise newline 
https://gitlab.com/malware-filter/urlhaus-filter/-/issues/79
 2023-04-29 04:11:14 +00:00
MDLeom 745c81b134
fix(exclude): education.gouv.fr 
- close #62
- checked whois record
 2023-04-25 11:24:54 +00:00
MDLeom 4456662716
fix(exclude): *.digitaloceanspaces.com 
- close #61
- close #52
- https://docs.digitalocean.com/products/spaces/details/availability/
 2023-04-25 11:17:05 +00:00
MDLeom 953537642e
fix(exclude): *.safelinks.protection.outlook.com 
- close #58
- enumerated dnsdumpster.com
- related 1ea3ce51f5
 2023-04-25 11:07:03 +00:00
MDLeom 6e68b44c73
fix(exclude): smex-ctp.trendmicro.com 
- closes #59
 2023-04-25 10:51:58 +00:00
MDLeom 99536fa229
fix(exclude): update s3 endpoints 
add new regions
https://docs.aws.amazon.com/general/latest/gr/s3.html#regional-endpoints
 2023-04-25 10:35:23 +00:00
__ f45c7e5299 fix: exclude scaleway S3 object storage 
- https://www.scaleway.com/en/docs/storage/object/quickstart/
- https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/
 2023-04-25 09:54:10 +00:00
MDLeom 8aa4d2334c
fix: cloudflare radar dataset is now in csv format 
instead of zip
 2023-01-16 07:09:35 +00:00
MDLeom b5048417b0
style(sed): avoid backslash in insert option 
- simpler and more readable
- https://unix.stackexchange.com/a/99351
 2022-12-17 00:19:11 +00:00
MDLeom 97cec9d0e8
feat: add csv file for Splunk lookup 
- https://docs.splunk.com/Documentation/Splunk/9.0.2/Knowledge/Aboutlookupsandfieldactions
 2022-12-17 00:06:59 +00:00
MDLeom 53c62b74c3
docs(header): switch date format from RFC 5322 to ISO 8601 
- universally readable
 2022-12-16 08:18:00 +00:00
MDLeom 0f9696c4f5
fix(exclude): interspar.at 
- https://spar-international.com/country/austria/
- close #25
 2022-12-05 08:11:25 +00:00
MDLeom f5e5e95dd8
fix(exclude): atshop.io 
- close #42
 2022-12-05 08:05:47 +00:00
MDLeom 6303ff306f
fix(exclude): lt27.de 
- https://www.tiekoetter.com/en/services/
- close #41
 2022-12-05 08:03:50 +00:00
MDLeom 48b5a4fce1
fix(exclude): short.upm.es 
- https://en.wikipedia.org/wiki/Technical_University_of_Madrid
- closes #37
 2022-12-05 07:56:47 +00:00
MDLeom 13b9740e66
fix(exclude): netbank.takarekbank.hu 
- https://en.wikipedia.org/wiki/Takar%C3%A9kbank
- closes #47
 2022-12-05 07:53:55 +00:00
MDLeom 1ea3ce51f5
feat: decode O365 safelink 
- https://support.microsoft.com/en-us/office/advanced-outlook-com-security-for-microsoft-365-subscribers-882d2243-eab9-4545-a58a-b36fee4a46e2
 2022-12-04 03:53:09 +00:00
MDLeom 5a4a8bb9bc
refactor: xmlstarlet -> html-xml-utils 2022-12-01 10:00:32 +00:00
MDLeom e653ba90c6
fix: remove extra curl option 2022-11-26 01:31:21 +00:00
MDLeom 4bf534bdbc
feat: add Cloudflare Radar top 1m domains dataset 2022-11-25 07:19:20 +00:00
MDLeom c376e2a08f
feat: fallback to busybox dos2unix 2022-11-03 08:48:16 +00:00
MDLeom e51886ff44
feat: fallback to busybox dos2unix 2022-11-03 08:46:39 +00:00
MDLeom a50b2be515
fix: disable phishunt 
- closes #43
- closes #45
 2022-11-03 08:41:25 +00:00
MDLeom c623542b9a
fix(exclude): outlook.com 
- closes #44
 2022-11-03 07:07:26 +00:00
Ming Di Leom d7e71fe41b Merge branch 'main' into 'main' 
fix(exclude): login.microsoftonline.com

See merge request malware-filter/phishing-filter!4
 2022-11-02 10:41:13 +00:00
MDLeom f992002230
ci(cf): snort2.rules path 2022-11-01 09:41:33 +00:00
MDLeom 25207f5708
ci(cf): snort2.rules path 2022-11-01 09:37:52 +00:00
MDLeom b2edb64044
ci(cf): snort2.rules path 2022-11-01 09:22:15 +00:00
MDLeom 4f3a67a21f
ci(cf): remove plain snort2.rules 
- over 25MB limit of cf pages
- use phishing-filter-snort2.rules.gz or phishing-filter-snort2.rules.br
 2022-11-01 07:27:52 +00:00
Aaron Viehl 8923941376 Added login.microsoftonline.com to exclusion list 2022-10-27 17:22:34 +00:00
MDLeom eea9efd0f1
style: alias "curl -L" 2022-10-10 09:15:57 +00:00
MDLeom 93824af81b
fix: add user agent to phishtank request 
https://phishtank.org/developer_info.php
 2022-10-10 09:13:39 +00:00
MDLeom c523b653bb
fix: make PHISHTANK_API optional 
- add TOC
 2022-10-03 03:44:53 +00:00
MDLeom 0447cfb792
fix(exclude): storage.yandexcloud.net 
- close #24
- close #34
 2022-09-27 09:53:18 +00:00
MDLeom e68268f506
fix: remove mitchellkrogza/Phishing.Database source 
- close #35
 2022-09-27 09:43:30 +00:00
MDLeom 6c27bf33aa
fix(exclude): application.axisbank.co.in 
- Close #21
- resolved IP (103.208.248.156) belongs to Axis Bank
 2022-09-27 08:51:15 +00:00
MDLeom 8fa366b37f
feat(sources): add phishunt.io & mitchellkrogza/Phishing.Database 
- inspired by Phishing.Army
 2022-07-31 08:29:10 +00:00
MDLeom b4229b2d56
fix: migrate to malware-filter group 
- BREAKING CHANGE
- gitlab.com/malware-filter/phishing-filter
 2022-05-21 03:04:56 +00:00
MDLeom 61762c4272
docs: to be migrated to gitlab.com/malware-filter 
- https://about.gitlab.com/blog/2021/11/11/public-project-minute-limits
- https://about.gitlab.com/blog/2022/02/04/ultimate-perks-for-open-source-projects
 2022-05-11 07:50:55 +00:00
MDLeom 4dd7298246
fix(exclude): wasabi 
- Closes #20
- https://wasabi.com/locations/
 2022-05-03 05:09:01 +00:00
Ming Di Leom daad15557a Merge branch 'zblach-main-patch-45394' into 'main' 
fix(exclude): add "app.skiff.org"
Closes #16
See merge request curben/phishing-filter!1
 2022-03-29 06:57:00 +00:00
MDLeom d7253c825e
fix: remove oisd exclusion list 
- captcha
 2022-03-28 10:29:01 +00:00
MDLeom 522a8814c4
build: check last pipeline status 
- using pipeline badge
- https://docs.gitlab.com/ee/ci/pipelines/settings.html#pipeline-status-badge
 2022-03-19 05:37:19 +00:00
MDLeom c28cc2b35a
ci(ga): trigger cloudflare & netlify from github 
if gitlab is down
 2022-01-25 10:12:02 +00:00
MDLeom 6c75445b79
build: failover to github if gitlab is unavailable 2022-01-25 10:04:52 +00:00
Zak 7ad3f2e40c Update src/exclude.txt 2022-01-17 19:10:54 +00:00
MDLeom dc834ed04a
fix: use GNU grep instead of busybox 2022-01-13 06:02:34 +00:00
MDLeom 0e7c5bb2af
Revert "fix: oisd.nl is down at the moment" 
This reverts commit d931f137e2.
 2022-01-09 08:33:16 +00:00
MDLeom d931f137e2
fix: oisd.nl is down at the moment 2022-01-09 07:35:42 +00:00
MDLeom 8bb0e6e990
refactor(actions): replicate gitlab ci 2022-01-09 07:32:09 +00:00
MDLeom 6cb2cbff6e refactor: deploy filters to gitlab pages 
- 8c94ddba40
 2022-01-08 03:01:41 +00:00
MDLeom aaab4b82fb
refactor: url encode space on the source 2022-01-02 01:15:46 +00:00
MDLeom ac81d8394e
fix: url encode space 
- Closes #11
 2022-01-02 01:10:57 +00:00
MDLeom a0f8a0b2ce
fix(exclude): facebook IPs 
- 157.240.0.0/16
- closes #9
 2021-09-05 07:26:11 +00:00
MDLeom 0a633e3c57
fix: stricter IPv4 matching 
- avoid excluding domains with IPv4, e.g. static.21.101.69.159.clients.your-server.de
 2021-07-18 09:59:09 +00:00
MDLeom 9fbb4b4686
feat: dnscrypt-proxy blocklists 
- support names and IPs
- https://github.com/DNSCrypt/dnscrypt-proxy
 2021-07-18 09:55:58 +00:00
MDLeom 0e9845b69a
fix: remove port number and deduplicate entries 
- Fixes #8
 2021-06-20 07:38:55 +00:00
MDLeom 43ac6158f1 fix: cleanup oisd-exlusion files 2021-05-05 10:05:18 +00:00
MDLeom 7cba69f1d5 fix: add oisd exclusion list 
- https://oisd.nl/excludes.php
 2021-05-05 10:05:17 +00:00
MDLeom 421e52d340
docs: switch mirror to curben.gitlab.io/malware-filter 
- 40647d12e7
 2021-04-02 03:43:28 +00:00
MDLeom 22c10b4018
feat: DNS Response Policy Zone (RPZ) 
- syntax based on https://rpz.oisd.nl/basic/
 2021-04-02 03:37:21 +00:00
MDLeom cf3a820b24 refactor: dedup ip removal function 2021-04-02 03:21:25 +00:00
MDLeom 3417d86243
docs: limitation of snort2 2021-03-20 00:57:45 +00:00
MDLeom 153970064d
fix: unique sid 
- resolve conflict with urlhaus-filter
 2021-03-20 00:56:59 +00:00
MDLeom 7cea93d141
refactor: simpler sed 2021-03-19 22:10:54 +00:00
MDLeom 4d0b92434b feat: add Snort3 ruleset 2021-03-19 19:04:02 +00:00
MDLeom 2024c75550
docs: link to license section 
- license differs on components and sources
 2021-03-18 22:04:01 +00:00
MDLeom cc9b306076
fix: prod 2021-03-18 18:51:43 +00:00
MDLeom d258cbd714
fix(snort/suricata): escape colons 
- trim snort rule to 2047 chars
 2021-03-18 18:49:50 +00:00
MDLeom d76146515c
feat: add Snort2 and Suricata rulesets 2021-03-18 17:44:15 +00:00
MDLeom eac91a9339
fix(exclude): form.elementform.com 
- website operator actively removes offending forms
- https://www.phishtank.com/phish_detail.php?phish_id=6483650
 2021-03-03 00:52:11 +00:00
MDLeom 99ca30b7de
fix(exclude): pomoc.o2.pl 
- Closes #5
- https://phishtank.com/phish_detail.php?phish_id=6891502
 2021-02-04 05:56:03 +00:00
MDLeom 9e0ecc4364
fix(exclude): encrypted-tbn0.gstatic.com 
- Closes #4
 2021-01-30 10:30:51 +00:00
MDLeom fc4a538223
docs: specify homepage 
- https://help.eyeo.com/en/adblockplus/how-to-write-filters#special-comments
 2020-12-30 23:37:26 +00:00
MDLeom 2b267b9991
feat: add IE-compatible blocklist 
- https://gitlab.com/curben/urlhaus-filter/-/issues/30
 2020-12-07 05:11:47 +00:00
MDLeom 9f92128dc7
fix(exclude): add all S3 endpoints 
- 81425bd0c5
 2020-11-09 04:34:18 +00:00