Feature Request: Mox for Delta Chat

[s0ph0s-dog]

Hello!

I am working on a hobby project to create an implementation of a chatmail server that is easier to deploy and more user-friendly. My goal is to make it easier for people to use Delta Chat, and hopefully reduce overall reliance on centralized chat platforms like WhatsApp, Telegram, and the like. I was originally going to use Maddy as the email server for this project, but its developer seems to have been inactive on GitHub for a long time. I've stumbled across Mox though, and it seems like a better-maintained choice—one to which I'd be happy to contribute!

In my (limited) testing so far, Mox seems to work well as an email server for Delta Chat:

• It delivers mail relatively quickly
• It implements CONDSTORE (for read state syncing between devices)
• It implements quotas

Chatmail servers also need to do several other things though. Some of them I already have plans to implement on my own, because they don't require any cooperation from the mail server (sign-up web page generation, a P2P data relay, etc.). Here are the things I'd like to discuss adding directly to Mox, because directly building them in might make the most sense:

• Milter support: In order to prevent abuse, chatmail servers need to reject any outgoing mail that isn't PGP encrypted. I've already written a milter plugin for this that works with Maddy, so milter support in Mox should enable that to work as-is.
• External authentication: New accounts on chatmail servers are created automatically when the first login is attempted. I don't see a SASL mechanism (or similar) that I can use to implement external authentication, so that probably requires some work.
• Data retention time limits: Chatmail servers delete unused accounts after a period of time (typically 90 days) and delete emails after a shorter period of time (typically 30 days). I'm not sure how bstore handles concurrent access by different processes, so I'm unsure whether this functionality can be built separately from Mox, or whether it needs to be built in.
• Delivery speed: Email delivery latency to local mailboxes with Mox seems to be slower than Maddy, but only by a few hundred milliseconds (with the junk filter turned off, by very unscientific measurements). I think profiling the code and optimizing this would be a fun challenge!
• TLS ALPN Multiplexing: Chatmail servers offer multiplexing of IMAP, SMTP, and HTTPS over port 443 using TLS ALPN to help users behind aggressive firewalls continue to use Delta Chat. I have some (uncommitted) sketch code that partially implements this alongside Maddy, but it might be better to build the ALPN proxy directly into Mox, given that Mox also runs a web server.

I'm quite happy to help contribute to the development of all of these features, if the help would be welcome! Is this sort of thing something you'd like to see Mox used for? If so, would you be willing to review my pull requests and offer guidance to me as I work on implementing the parts of these that are inside Mox?

[mjl-]

Hi, interesting topic and there certainly is overlap in goals.

It's a goal of mox itself to be easy to install and run. Automating the
creation of dns records, and maintaining them, is on the short list.

About the topics:

• Milter support: Others have asked milter support too. It could be added.
  But is relatively intrusive, with hooks through the smtp process. Especially
  full milter support, which requires being able to change messages
  (add/remove/rewrite headers), which mox cannot do at the moment. From your
  description, you would only run the milter for submission connections? Have
  you considered adding the check for pgp-encryptedness of submitted messages
  directly to mox, without milter? Part of the philosophy of mox is to not
  require many tools to get a running mail server. Mox could get a
  chatmail-mode (either inside mox, or maintained as a fork) that enables
  this functionality.
• External authentication: Mox indeed handles accounts and authentication
  itself. We could add integration with an external auth provider. I think
  it's been suggested before. It's not great for the general case for mox,
  because modern auth mechanisms may not work properly (eg
  scram-sha-256-plus). Another option could be to add autoprovisioning to mox
  itself? Again enabled with the chatmail mode. Are there multiple chatmail
  clients, or is deltachat the only one (in active use)? I presume it does
  plain auth during signup. Will it use better auth mechanisms in later
  connections (eg scram-sha-256-plus)?
  Btw, the mox admin interface is implemented with a simple web api. You
  could reuse that for creating accounts if signups through a website instead
  of signup-through-login are an option.
• Data retention: Bstore database indeed cannot be opened by multiple
  processes. So the cleanup would have to be done inside mox. Could be
  implemented in the chatmail mode.
• Delivery speed: Will certainly be interesting to see why mox is slower
  and ways to improve it!
• TLS ALPN Multiplexing: Indeed makes sense to have this done inside mox. For
  HTTPS, we would have to change the TLS listener to look at the nextproto
  chosen and possibly call a different protocol handler.

The chatmail page mentions push notifications. Mox does implement IMAP IDLE,
but does not send push notifications. Will that be an issue? I don't know
exactly what the latest state is around push notifications on IMAP and
iOS/Android. I've seen a mechanism with a custom imap extension, and I've may
have heard about JMAP push functionality being reused in some way.

I'll be happy to answer questions and the mox code, and give pointers & help
think about approaches. Can also be over email/call etc.

[s0ph0s-dog]

Awesome! I'm glad to hear that this sort of thing would be welcome. I'll reply point-by-point (with some minor re-ordering) below:

It's a goal of mox itself to be easy to install and run. Automating the creation of dns records, and maintaining them, is on the short list.

Oooh, cool! That's a nice bit of overlap :)

Milter support

Others have asked milter support too. It could be added.
But is relatively intrusive, with hooks through the smtp process. Especially
full milter support, which requires being able to change messages
(add/remove/rewrite headers), which mox cannot do at the moment.

Maddy avoids the complexities of full milter support by… also not having it, lol. (See the first three paragraphs of Maddy's documentation of their milter support.) That said, even just being able to accept/reject emails via a standardized interface is a useful feature.

From your description, you would only run the milter for submission connections?

Yep, that's correct.

Have you considered adding the check for pgp-encryptedness of submitted messages
directly to mox, without milter? Part of the philosophy of mox is to not
require many tools to get a running mail server. Mox could get a
chatmail-mode (either inside mox, or maintained as a fork) that enables
this functionality.

This is an interesting possibility. I am initially hesitant to go all-in on building things directly into Mox, because, from Mox's perspective, all of the chatmail stuff is a weird growth stuck to the side of an email server. I wouldn't want to write all of this stuff, upstream it into Mox, and then (heaven forbid) get hit by a bus and leave you with the maintenance burden for code that you may or may not even want to use. If the chatmail mechanisms are separated through standardized interfaces (like milter, Dovecot SASL, etc.), Mox can make good use of those interfaces regardless of what kind of thing is connected to them!

(For the record, I have no plans to disappear—I just want to make realistic plans for the long-term maintenance of the things I build! I have learned the hard way that if I don't make those plans up front, I end up building things that suck to maintain, and which I eventually give up on.)

The major wrinkle in building a standardized interface for all of the pieces here is that I don't know of any existing mechanism for providing external tools access to all of the emails for every account—except for maildir, which would be an enormous rewrite and would sacrifice many performance benefits gained by having a database. I don't think that maildir would be a good idea. I'll describe how I was planning to implement the interfaces for data cleanup and push notifications below.

External Authentication

Are there multiple chatmail clients, or is deltachat the only one (in active use)?

As far as I know, all of the Delta Chat clients (including the forks and third-party ones) use a core library written by the main developers called deltachat-core-rust.

I presume it does plain auth during signup. Will it use better auth mechanisms in later
connections (eg scram-sha-256-plus)?

The deltachat-core library only uses XOAUTH2 or PLAIN/LOGIN SASL mechanisms for authenticating to the server (IMAP code, SMTP code). I don't know for sure, but I think that XOAUTH2 is only used when talking to one of the major email providers (Gmail, Office 365, etc.), because they don't allow anything else. None of the code in the main chatmail server sets up an OAuth identity provider.

We could add integration with an external auth provider. I think
it's been suggested before. It's not great for the general case for mox,
because modern auth mechanisms may not work properly (eg
scram-sha-256-plus).

Maddy accomplishes this by allowing server administrators to delegate authentication to external modules using the Dovecot SASL protocol. Here's the documentation, but I can't pinpoint exactly where in the code this happens.

Mox indeed handles accounts and authentication
itself. […] Another option could be to add autoprovisioning to mox itself? […]
Btw, the mox admin interface is implemented with a simple web api. You
could reuse that for creating accounts if signups through a website instead
of signup-through-login are an option.

All of these could absolutely work! The upstream chatmail server has a /new HTTP API which the client uses to ask the server to hand it a random username and random password. The behavior of this endpoint could quite easily be modified to use the Mox admin API to create the account. That said, I've been trying to stick as close to the same behavior as the upstream, in order to minimize the odds of weird bugs. (Additionally, the trust-on-first-use mechanism prevents broken web crawlers or a miscreant running cURL in a loop from creating thousands of accounts in seconds. Setting up an IMAP connection is a much higher barrier than HTTP, simply by virtue of obscurity.)

Data Retention

Bstore database indeed cannot be opened by multiple
processes. So the cleanup would have to be done inside mox. Could be
implemented in the chatmail mode.

This makes sense. When I was figuring out how to do this with Maddy, I planned to read the message metadata out of the database. It's backed by SQLite (or PostgreSQL), so that is a very easy way to look at message dates and clean up old ones. There was also a "last login" field in the user table that I could use to erase old, unused accounts. SQLite and PostgreSQL both support concurrent access by multiple processes though, so the right choice for Mox might be to build it in.

Delivery Speed

Will certainly be interesting to see why mox is slower and ways to improve it!

I poked at this with the /debug/pprof tools last night, but I haven't figured out anything useful yet. I probably need to spend more time reading the code to understand what the profiling data means :P

TLS ALPN Multiplexing

Indeed makes sense to have this done inside mox. For
HTTPS, we would have to change the TLS listener to look at the nextproto
chosen and possibly call a different protocol handler.

This sounds good—plus it should hopefully mean that the rate limiting code needs no changes!

Push Notifications

The chatmail page mentions push notifications. Mox does implement IMAP IDLE, but does not send push notifications.

The deltachat-core-rust code uses IMAP IDLE when it's running in the foreground, but it also uses APNS/Play push notifications when in the background on mobile.

Will that be an issue? I don't know exactly what the latest state is around push notifications on IMAP and iOS/Android. I've seen a mechanism with a custom imap extension, and I've may have heard about JMAP push functionality being reused in some way.

It shouldn't be an issue—the Delta Chat folks have solved the complexities around IMAP push notifications by stepping sideways around them. The upstream chatmail server sends a "hey there's a new message for [email protected]" notification via HTTP to notifications.delta.chat, which maintains a registry of APNS/Play push notification tokens and chatmail email addresses, in order to deliver push notifications. I didn't mention this in the initial list because the solution is approximately the same as the data retention code. Based on what you mentioned above, the best choice seems to be a module which is enabled as part of "chatmail mode" that watches for new emails arriving and makes the required HTTP request to notifications.delta.chat.

When planning to do this with Maddy, I intended to use the IMAP Filter mechanism to send the HTTP request every time a new email was delivered. The "filter" wouldn't actually do any filtering. Instead it would write the email address for the account to a pipe, which the rest of the daemon code could read and then start an HTTP request retry loop. This plan was definitely a hack, and building it in would certainly be cleaner.

I'll be happy to answer questions and the mox code, and give pointers & help think about approaches. Can also be over email/call etc.

Fantastic, thank you! I'll try to ask major planning-related questions here, so that other folks can get involved if they'd like (such as the upstream chatmail server developers, who have expressed interest in my side project), and send more nitty-gritty questions about Mox's internals by email to avoid sidetracking the architectural stuff.

[mjl-]

Good to hear your consideration about adding chatmail-specific vs generic functionality, and the potential vanishing vs finishing. I'm not sure yet what the right approach is. If we end up with many config options that are specific for chatmail, than a single mode would probably have been simpler. It looks like some chatmail-specific thing (notifications and cleaning up) have to be in mox anyway. But some functionality would be good to have in general.

For milter support, indeed a simplified version may be enough. I think milters are intended to be run during each step in the smtp session. But perhaps we can just do the whole exchange at the end. Would simplify implementation quite a bit, I wasn't looking forward to adding it throughout the smtpserver code.

For authentication, if we were to check with another sasl server, we would probably require "plain" as only mechanism. The announced mechanisms would have to be trimmed when such a config option is active. The external server would call back to mox to create the account & email address.
The alternative of building this into mox would be to change the current calls to store.OpenEmailAuth with a new mox.OpenEmailAuth that creates new accounts if they don't exist yet (if autoaccounts are enabled), and calls store.OpenEmailAuth otherwise.

For data retention, a periodic goroutine that goes through all accounts and removes old message is probably easiest. Each account has its own message database that we have to go through. For removing stale accounts, we don't have "last login". The store.LoginSession type is only for web-based logins (from memory). I have been wanting to keep track of more details of active and recent sessions, this may be a good trigger to implement that.

For push notifications, indeed there is no hook for running custom code when a message is delivered...