ethan
/
ethanyoo.com
1
0
Fork 0
Code

Update robots.txt and set X-Robots-Tag header

This commit is contained in:
Ethan Yoo 2024-11-30 11:53:47 -05:00
parent fbf3d9813b
commit bb474315ab
Signed by: ethan
GPG Key ID: 70596474C5D20F85
2 changed files with 5 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
static/.htaccess
View File

@ -15,3 +15,8 @@ ErrorDocument 404 /404.html
<FilesMatch "\.(css|ico|jpg|jpeg|png|svg)$"> <FilesMatch "\.(css|ico|jpg|jpeg|png|svg)$">
Header set Cache-Control "max-age=2592000, public, immutable" Header set Cache-Control "max-age=2592000, public, immutable"
</FilesMatch> </FilesMatch>
# Prevent indexing of PDF files
<FilesMatch "\.(pdf)$">
Header always set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

1
static/robots.txt
View File

@ -1,7 +1,6 @@
# Disallow crawling of specific directories and file types # Disallow crawling of specific directories and file types
User-agent: * User-agent: *
Disallow: /*.asc.txt$ Disallow: /*.asc.txt$
Disallow: /*.pdf$
Disallow: /documents/ Disallow: /documents/
Disallow: /img/ Disallow: /img/