Running your own email server can be quite rewarding. You are in charge of your data. It also allows you more flexibility with your delivery options. However, there are a few challenges. You run the risk of opening your server up to vulnerabilities, as well as making your server a potential relay for spammers to use.
This guide is clearly aimed at those experienced with FreeBSD, networking, DNS, and email. If you don't know the difference between an MTA or MDA, this guide is probably not for you. I don't want to sound harsh, but I would argue that running a mail server requires a certain amount of in-depth knowledge of various mechanisms due to the overall complexity and general complexity of the thing called "email".
If you're inexperienced and want to learn: Great! I'll include very rough explanations for the terms I've written, but be sure to read them in detail.
OpenSMTPd is a mail transfer agent (MTA) and mail delivery agent (MDA). This means that it can communicate with other mail servers over the SMTP protocol, and it also handles delivering mail to the individual users’ mailboxes. We’ll be setting up OpenSMTPd so that it can communicate to external servers (filtered through spamd) and deliver mail to local users, as well as delivering local mail from user to user.
Dovecot is an MDA which reads local mailboxes and serves them up over IMAP or POP3 to the users. It will use the local users’ mailboxes to serve this content.
Spamd is a mail filtering service. We can forward mail through spamd, and it will filter mail based on a variety of blacklists, whitelists, and a greylist.
DNS setup
There are some things to set up DNS-wise. First off, make sure that your reverse DNS record is set to the name of your future mail server: If your mail domain is example.com, your mail server domain will probably be something like mail.example.com, at least this is what I’m using in this guide. Then, set up according DNS records that point to your server.
You also might wanna set up a SPF record (“Sender Policy Framework”). When you send an email, the receiving MTA queries the DNS TXT records for your mail domain. A special TXT record provides your SPF policy, which defines the mail servers that are allowed to send emails for the given domain
Prepare host system & jail
My host system looks like that: I have one public facing IP address but internally I want to split up my services to different jails, so that I can maintain them more easily. For that to work, we set up jails using ezjail and create an internal network for the jails. On the host system, we create firewall rules in pf to set up a NAT between our internal network and the public facing IP address. This is pretty much the way your network at home works most probably: You have one public facing IPv4 address but your internal network might be full of hosts and your router creates the NAT.
For inbound connections, we forward the requests based on the port to our jails in the internal network.
I already wrote an article about that some time ago that you can follow for setting up ezjail and the NAT, so I will not describe that again, just have a look here: FreeBSD jails with a single public IP address.
Create user
For this guide, we will use local system users for SMTP authentication and mail delivery. Later on, we will switch to virtual users that map to a single system user, but for the start, let’s keep it simple:
root@ns3:~ # adduser myuser
Create a user, give it a password and use nologin as a shell, since the user is not really supposed to log in. The home directory will contain the Maildir, which receives all the email for now.
Fetch certificates
Before we start installing the STMP server, we need to get a valid TLS certificate. Let’s Encrypts certbot can help us with that:
root@ns3:~ # pkg install py39-certbot
We will start certbot to fetch a single certificate for now. It will use the HTTP-01 challenge, therefore we need to make sure that port 80 on the host system’s IP forwards to our mail server jail. If you followed the guide so far, you already have set that up in your host’s /etc/pf.conf.