From 13b1aa049acb46149bd78de70d8d6a6dec9ba96c Mon Sep 17 00:00:00 2001 From: Amolith Date: Sat, 16 Jan 2021 03:28:59 -0500 Subject: [PATCH] don't want to lose progress so committing it early THIS IS A DRAFT Whatever content is in the current email post may or may not appear in the final version. I'm just committing so I have a backup :P --- blog.org | 353 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 353 insertions(+) create mode 100644 blog.org diff --git a/blog.org b/blog.org new file mode 100644 index 0000000..915c9c0 --- /dev/null +++ b/blog.org @@ -0,0 +1,353 @@ +#+HUGO_BASE_DIR: ./ +#+HUGO_SECTION: posts +#+HUGO_AUTO_SET_LASTMOD: t + +* Meta :@Meta: +* Technology :@Technology: +** A perfect email setup (for me) +:PROPERTIES: +:EXPORT_FILE_NAME: a-perfect-email-setup-for-me +:EXPORT_HUGO_CUSTOM_FRONT_MATTER: :toc true +:END: +I've never been satisfied with any of the email clients most people use. +I've tried [[https://www.thunderbird.net/en-GB/][Thunderbird]], [[https://wiki.gnome.org/Apps/Evolution][Evolution]], [[https://getmailspring.com/][Mailspring]], [[https://support.apple.com/mail][Mail.app]], [[https://roundcube.net/][Roundcube]], +[[https://sogo.nu/][SOGo]], [[https://wiki.gnome.org/Apps/Geary][Geary]], and /many/ more. /None/ of them handle multiple accounts +particularly well because all of the emails associated with that account +are bound within it. Sure, you can make a new folder somewhere called +~TODO~ and move all of your actionable emails to that folder but, when you +go to move actionable emails from /another/ account into that folder, +you'll likely find that the client simply doesn't let you. If it does, +when you reply, it will likely be sent from the wrong account. This is a +limitation of the IMAP protocol; everything is /managed/ locally but +changes are pushed to the remote server and mixing things the way I want +leads to broken setups. + +Before I go any further, these are a few characteristics of my ideal +email tool. + +- Support for multiple accounts (obviously) +- /Native desktop application/ (*not* [[https://github.com/electron/electron][Electron]]) +- Has stellar keyboard shortcuts +- Doesn't require internet connectivity (other than downloading and + sending of course) +- Organisation can be done with tags + +*** Why tags? +Because they're better. Hierarchies are useful for prose and code but +not for files, emails, notes, or anything where an item may fit within +multiple categories. Imagine you get an email from your Computer Science +professor that includes test dates, homework, and information about +another assignment. In that same email, he asks every student to reply +with something they learned from the previous class as a form of +attendance. In a hierarchy, the best place for this might just be a ~TODO~ +folder /even though/ it would also fit under ~School~, ~CS~, ~Dates~, ~To read~, +and ~Homework~. Maybe you have a few minutes and want to clear out some +emails that don't require any interaction. In a tag-based workflow, this +would be a good time to open ~To read~, get that email out of the way, and +remove the ~To read~ tag. It would still show up under the other tags so +you can find it later and take the time to fully answer the professor's +question, add those dates to your calendar, and add the homework +assignments to your ~TODO~ list. Hierarchies can be quite cumbersome to +work with, especially when one folder ends up getting all the data. Tags +ensure that you only see what you want when you want it. Tags are more +efficient and they will remain my organisation system of choice. + +*** The tools +In short, the tools we will be using are... ++ [[https://www.offlineimap.org/][OfflineIMAP]] to download our emails ++ [[https://notmuchmail.org/][~notmuch~]], the primary way emails will be organised ++ [[https://afew.readthedocs.io/en/latest/][~afew~]] to apply initial ~notmuch~ tags based on subject, sender, recipient, etc. ++ [[https://neomutt.org/][NeoMutt]] to interact with those emails, reply, compose, add/remove + tags, etc. ++ [[https://marlam.de/msmtp/][~msmtp~]] for relaying our replies and compositions to our mail provider + +Yes, it's a lot. Yes, it's time-consuming to set up. Yes, it's worth it +(in my opinion). + +*** OfflineIMAP +As I said above, IMAP is limiting; we need to use some other method of +downloading our emails. There's an awesome piece of software called +[[https://www.offlineimap.org/][OfflineIMAP]] which is built for exactly this purpose. Its configuration +can be rather daunting if you have as many accounts as I do (17) but +it's not /terrible/. + +**** General +#+BEGIN_SRC text +[general] +metadata = ~/.offlineimap +accounts = use_exa +maxsyncaccounts = 1 +ui = basic +ignore-readonly = no +pythonfile = ~/.offlineimap.py +socktimeout = 60 +fsync = true +#+END_SRC + +The first big option is ~accounts~; it tells OfflineIMAP what to actually +sync. What to put there will be defined further down but ~use_exa~ is just +filler text. The example account is ~user@example.com~ and I shortened +that to ~use_exa~. ~maxsyncaccounts~ is also fairly important as it tells +OfflineIMAP to only pull emails from one account at a time. This is +certainly slower than multiple but it's also safer because we'll be +running this in the background and don't want many OfflineIMAP processes +executing concurrently and interfering with each other. ~pythonfile~ will +be discussed later. + +**** Account +#+BEGIN_SRC text +[Account use_exa] +localrepository = use_exa-local +remoterepository = use_exa-remote +quick = 10 +utf8foldernames = yes +postsynchook = notmuch new +#+END_SRC + +In the first block, ~localrepository~ and ~remoterepository~ tell OfflineIMAP where +to look for your emails. ~use_exa-local~ is an arbitrary naming scheme I use to +differentiate between the various local and remote accounts. It can easily be +swapped with something else. + +**** Repository +#+BEGIN_SRC text + [Repository use_exa-local] + type = Maildir + localfolders = ~/mail/use_exa + sync_deletes = yes + + [Repository use_exa-remote] + type = IMAP + remotehost = imap.example.com + starttls = yes + ssl = no + remoteport = 143 + remoteuser = user@example.com + remotepasseval = get_pass("use_exa") + auth_mechanisms = GSSAPI, XOAUTH2, CRAM-MD5, PLAIN, LOGIN + maxconnections = 1 + createfolders = True + sync_deletes = yes +#+END_SRC + +The repository sections describe how the emails are stored or retrieved. +In the ~local~ block, you'll notice that the type is ~Maildir~. In this +format, each email is given a unique filename and stored in a hierarchy +of folders within your account. This is often how your emails are stored +on your provider's mail server. + +~pythonfile~ is used here to authenticate with the remote server. This can +be complicated and depends /entirely/ on how you manage your passwords. I +use [[https://keepassxc.org/][KeePassXC]] and love it. When I set OfflineIMAP up, however, it didn't +have ~libsecret~ compatibility. This would have made setup significantly +easier but, as it already just works™, I don't really see a reason to +change it. + +This new feature allows ~libresecret~-based applications to query +KeePassXC for your passwords or store them there on your behalf. CLI/TUI +applications that need a secure mechanism for background authentication +can use ~secret-tool lookup Title "TITLE_OF_PASSWORD"~ as the password +command. See [[https://github.com/keepassxreboot/keepassxc/pull/2726][the pull request]] for more details. Because this wasn't a +feature when I first set it up, I put my passwords in plaintext files +and encrypted them with the GPG key stored on my YubiKey. As long as my +key is plugged in, OfflineIMAP can authenticate and download all my +emails just fine. The process for using a GPG key /not/ stored on a +hardware token is pretty much the same and I'll talk about that process +instead. + +These are the contents of my ~~/.offlineimap.py~. + +#+BEGIN_SRC python + #! /usr/bin/env python2 + from subprocess import check_output + def get_pass(account): + return check_output(["gpg", "-dq", f" ~/.mail_pass/{account}.gpg"]).strip("\n") +#+END_SRC + +This runs ~gpg -dq ~/.mail_pass/use_exa.gpg~ then strips the newline +character before returning it to OfflineIMAP. ~-d~ tells GPG that you're +passing it a file you want decrypted and ~-q~ tells it not to give any +output other than the file's contents. For a setup that works with this +Python script, put your passwords in plaintext files with the account +name as the file name (e.g. ~use_exa~). You'll then encrypt it with ~gpg +-er use_exa~. Running ~gpg -dq use_exa.gpg~ should display +your password. Repeat for every account and store the resulting files in +~~/.mail_pass/~. + +The other option, ~sync_deletes~, is whether or not to delete remote +emails that have been deleted locally. I enabled that because I want to +have easy control over how much remote storage is used. + +Here's the next block again so you don't have to scroll up: + +#+BEGIN_SRC text +[Repository use_exa-remote] +type = IMAP +remotehost = imap.example.com +starttls = yes +ssl = no +remoteport = 143 +remoteuser = user@example.com +remotepasseval = get_pass("use_exa") +auth_mechanisms = GSSAPI, XOAUTH2, CRAM-MD5, PLAIN, LOGIN +maxconnections = 1 +createfolders = True +sync_deletes = yes +#+END_SRC + +This one's pretty self-explanatory. ~type~, ~remotehost~, ~starttls~, ~ssl~, and +~remoteport~ should all be somewhere in your provider's documentation. +~remoteuser~ is your email address and ~remotepasseval~ is the function that +will return your password and allow OfflineIMAP to authenticate. You'll +want enter the name of your password file without the ~.gpg~ extension; +the script takes care of adding that. Leave ~auth_mechanisms~ alone and +the same for ~maxconnections~ unless you know your provider won't rate +limit you or something for opening multiple connections. ~sync_deletes~ is +the same as in the previous block. + +Copy those three blocks for as many accounts as you want emails +downloaded from. I have 510 lines just for ~Account~ and ~Repository~ blocks +due to the number of address I'm keeping track of. + +*** ~notmuch~ +[[https://notmuchmail.org/][~notmuch~]] is /a fast, global-search, and tag-based email system/. This +what does all of our organisation as well as what provides the "virtual" +mailboxes NeoMutt will display later on. Configuration is incredibly +simple. This file goes in ~~/.notmuch-config~. + +#+BEGIN_SRC text + [database] + path=/home/user/mail/ + + [user] + name=Amolith + primary_email=user@example.com + + [new] + tags=unread;new; + ignore=Trash; + + [search] + exclude_tags=deleted;spam; + + [maildir] + synchronize_flags=true +#+END_SRC + +First section is the path to where all of your archives are, the ~[user]~ +section is where you list all of your accounts, ~[new]~ adds ~tags~ to mail +notmuch hasn't indexed yet and ignores indexing the ~Trash~ folder, and +~[search]~ ignores mail tagged with ~deleted~ or ~spam~. The final section +tells ~notmuch~ to add maildir flags which correspond with ~notmuch~ tags. +These flags will be synced to the remote server the next time +OfflineIMAP runs and things will be somewhat organised in your webmail +interface. + +After creating the configuration file, run ~notmuch new~ and wait for all +of your mail to be indexed. This could take a short amount of time or it +could take minutes up to an hour, depending on how many emails you have. +After it's finished, you'll be able to run queries and see matching +emails: + +#+BEGIN_SRC text + $ notmuch search from:user@example.com + thread:0000000000002e9d December 28 [1/1] Example User; Random subject that means nothing +#+END_SRC + +This is not terribly useful in and of itself because you can't read it +or reply to it or anything. That's where the Mail User Agent (MUA) comes +in. + +*** ~afew~ +[[https://afew.readthedocs.io/en/latest/][~afew~]] is /an initial tagging script for notmuch/. After calling ~notmuch +new~, ~afew~ will add tags based on headers such as ~From:~, ~To:~, ~Subject:~, +etc. as well as handle killed threads and spam. The official [[https://afew.readthedocs.io/en/latest/quickstart.html][quickstart +guide]] is probably the best resource on getting started but I'll include +a few tips here as well. + +*** NeoMutt +*** ~msmtp~ +~msmtp~ is what's known as a /Mail Transfer Agent/ (MTA). You throw it an +email and it will relay that to your mail provider's SMTP server so it +can have the proper headers attached for authentication, it can be sent +from the proper domain, etc. All the necessary security measures can be +applied that prevent your email from going directly to spam or from +being rejected outright. + +~msmtp~'s configuration is also fairly simple if a bit long, just like +OfflineIMAP's. + +#+BEGIN_SRC text + # Set default values for all following accounts. + defaults + + # Use the mail submission port 587 instead of the SMTP port 25. + port 587 + + # Always use TLS. + tls on +#+END_SRC + +This section just sets the defaults. It uses port 587 (STARTTLS) for all +SMTP servers unless otherwise specified and enables TLS. + +#+BEGIN_SRC + account user@example.com + host smtp.example.com + from user@example.com + auth on + user user@example.com + passwordeval secret-tool lookup Title "user@example.com" +#+END_SRC + +This section is where things get tedious. When passing an email to +~msmtp~, it looks at the ~From:~ header and searches for a block with a +matching ~from~ line. If it finds one, it will use those configuration +options to relay the email. ~host~ is simply the SMTP server of your mail +provider, sometimes this is ~mail.example.com~, ~smtp.example.com~, etc. +I've already explained ~from~, ~auth~ simply says that a username and +password will have to be provided, ~user~ is that username, and +~passwordeval~ is a method to obtain the password. + +When I got to configuring ~msmtp~, [[https://keepassxc.org/][KeePassXC]] had just released their +~libsecret~ integration and I wanted to try it. ~secret-tool~ is a command +line tool used to store and retrieve passwords from whatever keyring +you're using. I think KDE has ~kwallet~ and GNOME has ~gnome-keyring~ if +you already have those set up and want to use them; the process should +be quite similar regardless. + +As mentioned above ~secret-tool~ stores and retrieves passwords. For +retrieval, it expects the command to look like this. + +#+BEGIN_SRC text +secret-tool lookup {attribute} {value} ... +#+END_SRC + +I don't know what ~kwallet~ and ~gnome-keyring~'s attributes are but +this can be used with KeePassXC by specifying the ~Title~ attribute. If +the password to your email account is stored in KeePassXC with the +address as the entry title, you can retrieve it by simply running... + +#+BEGIN_SRC text +secret-tool lookup Title "user@example.com" +#+END_SRC + +If you have a different naming system, you'll have to experiment and try +different things; I don't know what KeePassXC's other attributes are so +I can't give other examples. + +You could also just use the same method I described in [[*Repository][the Repository +section]]! It will work perfectly fine here as well. + +#+BEGIN_SRC +passwordeval gpg -dq ~/.mail_pass/use_exa.gpg +#+END_SRC + +Now that the whole block is assembled, copy/paste/edit for as many +accounts as you want to send email from. + +*** Summary + +* Music :@Music: +* Pipe Smoking :@Pipe__Smoking: +* Dungeons & Dragons :@Dungeons__and__Dragons: