Fix email (#2054)

* Modify email sending configuration

We need to move off SendGrid. First step: modify how we send email
so it's much easier to configure.

Signed-off-by: David A. Wheeler <[email protected]>

* Fix documentation so it doesn't say we use SendGrid

Signed-off-by: David A. Wheeler <[email protected]>

* Add link to Amazon SES docs

Signed-off-by: David A. Wheeler <[email protected]>

* Send email using TLS directly (without STARTTLS)

Switch to sending email using TLS directly, without using STARTTLS at all.

STARTTLS lets us request an upgrade to TLS, and we *could* choose to not
send an email if the upgrade fails.

However, it's simpler to simply use TLS right from the beginning. Amazon
supports this, and many others probably do as well. This is simpler, and thus
less prone to failures if an "upgrade to TLS" fails. There's never an opportunity
for any party to be "confused" about whether or not TLS is currently enabled.

Of course, this *only* applies to the hop from our application to the MTA
who will then forward the email on. MTAs communicate with each other in a variety
of ways. But if we can securely get the email to an MTA we trust, the MTA can
then forward the email using the best mechanism available to it.

Signed-off-by: David A. Wheeler <[email protected]>

---------

Signed-off-by: David A. Wheeler <[email protected]>

README.md

@@ -188,7 +188,6 @@ to the CII. The domain name is much shorter, too.
 We use the "www" subdomain because there are technical challenges using
 a top-level domain with our CDN; it's more efficient to use the subdomain.
 
-
 ## License
 
 All material here is released under the [MIT license](./LICENSE).

config/environments/production.rb

@@ -98,20 +98,58 @@
   # https://github.com/ankane/secure_rails
   config.action_controller.asset_host = host
 
-  # Configure email server.
+  # Configure email server we use for sending email to others. This server will,
+  # by definition, be a Mail Transfer Agent (MTA).
   # For discussion about how to do this, see:
+  # https://guides.rubyonrails.org/action_mailer_basics.html
   # https://www.railstutorial.org/book/account_activation_password_reset
-  config.action_mailer.raise_delivery_errors = true
+  # We used to use SendGrid; see its documentation for more about that.
+  # For more about configuring Amazon SES SMTP interface, see:
+  # https://docs.aws.amazon.com/ses/latest/dg/send-email-smtp.html
+  #
+  # TODO: Eventually we should set raise_delivery_errors = true and handle exceptions,
+  # so that we can do retries, and put email sending in a separate thread.
+  # However, in the current code, if we raise delivery errors we cause paths to fail
+  # such as updating a project when the email system isn't working. We'd rather let the
+  # projects update, even when the email system isn't working properly.
+  config.action_mailer.raise_delivery_errors = false
   config.action_mailer.delivery_method = :smtp
   config.action_mailer.default_url_options = { host: host }
   ActionMailer::Base.smtp_settings = {
-    address: 'smtp.sendgrid.net',
-    port: '587',
+    # NOTE: 'address' is NOT the email address of the *sender* but the domain of
+    # the remote mail server (the MTA) that will be *receiving* the email and then send it on.
+    address: ENV.fetch('BADGEAPP_SEND_EMAIL_ADDRESS', 'smtp.sendgrid.net'),
+    # Avoid using port 25, use a TLS port instead.
+    port: ENV.fetch('BADGEAPP_SEND_EMAIL_PORT', '465'),
+    # "plain" authtication would send the password in the clear if sent on port 25.
+    # However, we're assuming that we're using TLS to send email to the MTA.
     authentication: :plain,
-    user_name: ENV.fetch('SENDGRID_USERNAME', nil),
-    password: ENV.fetch('SENDGRID_PASSWORD', nil),
-    domain: 'heroku.com',
-    enable_starttls_auto: true
+    # This is the authentication data we'll send to the MTA to be authorized to send email
+    user_name: ENV.fetch('BADGEAPP_SEND_EMAIL_USERNAME', nil),
+    password: ENV.fetch('BADGEAPP_SEND_EMAIL_PASSWORD', nil),
+    # This is used for the HELO announcement to the MTA:
+    domain: ENV.fetch('BADGEAPP_SEND_EMAIL_DOMAIN', 'heroku.com'),
+    # The following controls how we use TLS for hop-to-hop (point-to-point) encryption of
+    # our emails to the MTA.
+    # *Force* the direct use of TLS when sending email to the MTA. It's simple, which is an
+    # advantage for security. This typically uses port 465 (not 587).
+    # Amazon calls this "TLS Wrapper" in:
+    # https://docs.aws.amazon.com/ses/latest/dg/smtp-connect.html
+    tls: true
+    #
+    # This would use TLS opportunistically, via STARTTLS.
+    # However, it would send email and passwords in the clear
+    # if it can't. It's enough to counter most passive threats, but an active threat agent
+    # could cause TLS to fail & force us to send things in the clear. So we won't use it.
+    # This would typically use port 587.
+    # enable_starttls_auto: true
+    #
+    # This would use STARTTLS but force the use of TLS. This works, but there are concerns
+    # that starting the protocol *not* in TLS mode and *then* switching to TLS creates a
+    # window of opportunity for problems as well as unnecessary complexity.
+    # So while this is a reasonable alternative, for now we won't use it.
+    # This would typically use port 587.
+    # enable_starttls: true
   }
 
   # Enable locale fallbacks for I18n (makes lookups for any locale fall back to

docs/implementation.md

@@ -88,10 +88,10 @@ The application is configured by various environment variables:
   "production" or "fake_production". Plausible values are
   "debug", "info", and "warn". Default is "info". See:
   <http://guides.rubyonrails.org/debugging_rails_applications.html>
-* EMAIL_ENCRYPTION_KEY: Key to decrypt email addresses.
+* EMAIL_ENCRYPTION_KEY: Key to decrypt the stored user email addresses.
   Hexadecimal string, must be 64 hex digits (==32 bytes==256 bits).
   Used by aes-256-gcm (256-bit AES in GCM mode).
-* EMAIL_BLIND_INDEX_KEY: Key for blind index created for email
+* EMAIL_BLIND_INDEX_KEY: Key for blind index created for user email addresses
   (used by PBKDF2-HMAC-SHA256).  Must be 64 hex digits (==32 bytes==256 bits).
 * BADGEAPP_DENY_LOGIN: If a non-blank value is set ("true" is recommended),
   then no on can log in, no one can create a new account (sign up),
@@ -155,6 +155,20 @@ The application is configured by various environment variables:
 * `BADGE_CACHE_STALE_AGE` : Time (in seconds) badges are served by the CDN
   if it can't get a new value from us.
   Default 8640000 (100 days), is forced to be at least 2x`BADGE_CACHE_MAX_AGE`
+* `BADGEAPP_SEND_EMAIL_*`: Various environment variables that configure how the
+  BadgeApp sends email to an email server (a kind of Mail Transfer Agent (MTA))
+  that will then be sent on to others. See `config/environments/production.rb`.
+* `BADGEAPP_SEND_EMAIL_ADDRESS`: The domain of the MTA to send email to, e.g.,
+   `smtp.sendgrid.net`. Note that this is NOT the email address of the
+   *sender* but the domain of the remote mail server (the MTA) that will be
+   *receiving* the email and then send it on.
+* `BADGEAPP_SEND_EMAIL_PORT`: Port of the MTA. Use one that forces TLS or
+   or modify the config file.
+* `BADGEAPP_SEND_EMAIL_USERNAME`: Username for logging into the MTA
+* `BADGEAPP_SEND_EMAIL_PASSWORD`: Password for logging into the MTA
+* `BADGEAPP_SEND_EMAIL_DOMAIN`: Domain to report to the MTA (for HELO)
+* `SENDGRID_USERNAME` and `SENDGRID_PASSWORD`: These were once used for sending email,
+  but these are *not* used any more.
 
 You can make cryptographically random values (such as keys)
 using "rails secret".  E.g., to create 64 random hexadecimal digits, use:
@@ -1112,14 +1126,15 @@ See the Action mailer basics guide at
 and Hartl's Rails tutorial, e.g.:
 <https://www.railstutorial.org/book/account_activation_password_reset#sec-email_in_production>
 
-To install sendgrid on Heroku to make this work, use:
+We used to use SendGrid. You *can* just use SendGrid directly.
+You can also install sendgrid on Heroku to provide some extra functions doing this:
 
 ~~~~sh
 heroku addons:create sendgrid:starter
 ~~~~
 
 If you plan to handle a lot of queries, you probably want to use a CDN.
-It's currently set up for Fastly.
+It's currently set up to use Fastly.
 
 ## Badge SVG
 

docs/security.md

@@ -394,7 +394,8 @@ learning about our users' activities (and thus maintaining user privacy):
   (e.g., to activate a local account), but those links go directly back
   to our site for that given purpose, and do not reveal information to
   anyone else.
-  We use SendGrid to send email, but we have specifically configured the
+  You need to configure the MTA used to send email before you can use it.
+  We used to use SendGrid to send email; we have specifically configured the
   [SendGrid X-SMTPAPI header to disable all of its trackers we know of](https://sendgrid.com/docs/ui/account-and-settings/tracking/),
   which are clicktrack, ganalytics, subscriptiontrack, and opentrack.
   For example, we have never used ganalytics, but by expressly disabling it,
@@ -1956,17 +1957,20 @@ as of 2015-12-14:
    but the data is sufficiently low value, and there aren't
    good alternatives for low value data like this.
    This isn't as bad as it might appear, because we prefer encrypted
-   channels for transmitting all emails. Our application attempts to send
+   channels for transmitting all emails.
+   Historically our application attempts to send
    messages to its MTA using TLS (using `enable_starttls_auto: true`),
-   and that MTA (SendGrid) then attempts to transfer the email the rest
+   and that MTA then attempts to transfer the email the rest
    of the way using TLS if the recipient's email system supports it
    (see <https://sendgrid.com/docs/Glossary/tls.html>).
    This is good protection against passive attacks, and is relatively decent
    protection against active attacks if the user chooses an email system
    that supports TLS (an active attacker has to get between the email
    MTAs, which is often not easy).
+   More recently we're using `enable_starttls: true` which *forces* email to be
+   encrypted point-to-point.
    If users don't like that, they can log in via GitHub and use GitHub's
-   forgotten password system.
+   system for dealing with forgotten passwords.
    The file `config/initializers/filter_parameter_logging.rb`
    intentionally filters passwords so that they are not included in the log.
    We require that local user passwords have a minimum length