curben
/
phishing-filter
mirror of https://gitlab.com/malware-filter/phishing-filter
1
0
Fork 0
Code Issues Releases Wiki Activity
1,337 Commits 2 Branches 0 Tags 682 MiB
Commit Graph

134 Commits

Author SHA1 Message Date
MDLeom ab5dca49b4
refactor: handle url-without-path & safelinks without "while read" 
"while read" can be inefficient
 2025-03-16 12:37:56 +00:00
MDLeom 6e359f9a79
fix: remove trailing slash from domain 
to replace previous workaround 0578e6c16a
 2025-03-16 10:05:02 +00:00
MDLeom 993bb958f5
fix: skip phishtank if download fails 2025-03-16 07:37:17 +00:00
MDLeom 56d67d2a41
Revert "feat: remove phishtank source" 
This reverts commit b3f6e90b9a.
https://gitlab.com/malware-filter/phishing-filter/-/issues/40#note_1849507513
 2025-03-16 06:56:41 +00:00
MDLeom be1b6c05d7
fix: remove credential from domain/IP 
fixes #91
 2025-03-11 07:23:02 +00:00
MDLeom e1b051b2fc
fix: remove response header 
showing it will stop showing download progress/size
 2025-03-08 00:07:04 +00:00
MDLeom a500fca678
fix: use redirected tranco link 2025-03-07 23:55:14 +00:00
MDLeom c5fd7f7d34
fix: output response header to stdout 
https://codeahoy.com/general/curl-display-request-response-headers
 2025-03-07 23:53:52 +00:00
MDLeom b94d832896
fix: skip tranco if download fails 2025-03-07 23:42:22 +00:00
MDLeom 7e8139510d
style(rpz): generic syntax 2025-02-16 00:44:21 +00:00
MDLeom 3529e93ba3
feat: wildcard asterisk 2025-02-16 00:23:22 +00:00
MDLeom 8506f18029
chore: remove unused oisd exclusion 2025-02-15 01:12:03 +00:00
MDLeom 7f90191c49
feat: add ipthreat.net source 2025-02-08 06:24:03 +00:00
MDLeom d7145f19e1
refactor: parallel rm() 2024-10-27 07:06:04 +00:00
MDLeom 58e50a2f85
build: remove file >=25MiB in cf pages 2024-10-27 07:02:06 +00:00
MDLeom 4cdf3fa532
build: only remove .rules from Cloudflare Pages 2024-10-27 05:46:42 +00:00
MDLeom 293507fead
refactor: extract zip from stream 2024-10-27 05:34:06 +00:00
MDLeom 8702981a79
fix: unzip alternatives 2024-07-15 09:43:11 +00:00
MDLeom f07ad2ce4e
refactor: set pipefail conditionally 2024-07-15 08:02:25 +00:00
MDLeom 827342f3e9
fix: expand alias in bash 2024-06-03 08:21:56 +00:00
MDLeom 358003b782
fix: subdomains may be completely excluded 2024-05-03 11:16:01 +00:00
MDLeom 2ee0b2d661
feat(source): disable mitchellkrogza/Phishing.Database 
source does not offer online-only links
closes #86
 2024-05-02 12:00:37 +00:00
MDLeom e9ae4a9f11
refactor: replace got with fetch 2024-04-07 00:45:34 +00:00
MDLeom 607208c171
fix: check file exists and not zero size 2024-03-10 07:49:19 +00:00
MDLeom a1548a5e1c
fix: may not necessarily contain ipv4 entries 2024-03-10 03:06:51 +00:00
MDLeom 00d43e98d3
fix(exclude): click.mail.onedrive.com 2024-03-09 04:28:57 +00:00
MDLeom 6f7cf84de2
chore(exclude): cleanup entries 2024-03-09 04:16:19 +00:00
MDLeom e02ed129d5
fix(exclude): fonts.gstatic.com 2024-03-09 04:12:33 +00:00
MDLeom 5c7b1f4645
feat(source): add mitchellkrogza/Phishing.Database 
ref #40
revert e68268f506
 2024-03-09 04:06:37 +00:00
Ming Di Leom 6b681bc58f Merge branch 'exclude-s3-fix' into 'main' 
Add Amazon S3 dual-stack endpoints to exclude list

See merge request malware-filter/phishing-filter!8
 2024-03-08 08:11:27 +00:00
Alan Turing ecd739a846 fix(exclude): Add Amazon S3 dual-stack endpoints to exclude list 
and add new regions
https://docs.aws.amazon.com/AmazonS3/latest/userguide/dual-stack-endpoints.html
https://docs.aws.amazon.com/general/latest/gr/s3.html
 2024-03-08 08:11:27 +00:00
MDLeom 1b2312f492
fix: "phishing-subdomains.txt" may be empty 2024-03-08 07:54:33 +00:00
MDLeom 93b85b00f9
chore: remove remaining phishunt 
no longer used since #43 #45
 2024-03-07 10:14:08 +00:00
MDLeom b3f6e90b9a
feat: remove phishtank source 
frequent interference from cloudflare captcha
 2024-03-07 10:09:32 +00:00
MDLeom 6175179162
refactor: esm 
got is esm only since v12
 2023-10-01 09:56:57 +00:00
MDLeom 07ca1adfd1
refactor: lazy load os-release 2023-05-20 11:23:07 +00:00
MDLeom 667fad0b6f
style: remove debug message 2023-05-20 11:15:29 +00:00
MDLeom 13289d3365
fix: dash does not support pipefail 2023-05-20 10:38:47 +00:00
MDLeom eac902123e
fix: check installed grep is GNU variant 2023-05-20 09:51:12 +00:00
MDLeom eebf51ac47
fix: check existent of busybox 
if dos2unix is not installed
 2023-05-20 09:44:54 +00:00
MDLeom ca23363ef4
fix: reprocess decoded safelink 
- extend 1ea3ce51f5
- also include scope of 0578e6c16a
 2023-05-20 08:20:22 +00:00
MDLeom 0578e6c16a
fix: handle URL of top domains without path 
- ref #62, #43, #44
- 745c81b134, c623542b9a, 8923941376
were not effective previously
 2023-05-19 10:34:04 +00:00
MDLeom 7dbdc85163
fix: sed syntax to recognise newline 
https://gitlab.com/malware-filter/urlhaus-filter/-/issues/79
 2023-04-29 04:11:14 +00:00
MDLeom 745c81b134
fix(exclude): education.gouv.fr 
- close #62
- checked whois record
 2023-04-25 11:24:54 +00:00
MDLeom 4456662716
fix(exclude): *.digitaloceanspaces.com 
- close #61
- close #52
- https://docs.digitalocean.com/products/spaces/details/availability/
 2023-04-25 11:17:05 +00:00
MDLeom 953537642e
fix(exclude): *.safelinks.protection.outlook.com 
- close #58
- enumerated dnsdumpster.com
- related 1ea3ce51f5
 2023-04-25 11:07:03 +00:00
MDLeom 6e68b44c73
fix(exclude): smex-ctp.trendmicro.com 
- closes #59
 2023-04-25 10:51:58 +00:00
MDLeom 99536fa229
fix(exclude): update s3 endpoints 
add new regions
https://docs.aws.amazon.com/general/latest/gr/s3.html#regional-endpoints
 2023-04-25 10:35:23 +00:00
__ f45c7e5299 fix: exclude scaleway S3 object storage 
- https://www.scaleway.com/en/docs/storage/object/quickstart/
- https://www.scaleway.com/en/docs/storage/object/api-cli/object-storage-aws-cli/
 2023-04-25 09:54:10 +00:00
MDLeom 8aa4d2334c
fix: cloudflare radar dataset is now in csv format 
instead of zip
 2023-01-16 07:09:35 +00:00