Email delivery failure messages no such grain with email not configured

[jdougan]

With mail not configured in my selfhosted sandstorm, I got a piled of weird email failure messages. The especially weird part is there has never been grains with those names on my sandstorm installation. For that matter port 25 is blocked. Any idea what is happening?

As an aside, the lack of timestamps in the log file is making troubleshooting very difficult.

E-mail delivery failure: Error: No such grain: hr2008_007
    at imports/server/drivers/mail.js:167:25
    at imports/server/async-helpers.ts:24:17
    at imports/server/async-helpers.ts:12:3
    at runWithEnvironment (packages/meteor.js:1286:24)
    at packages/meteor.js:1299:14
    at imports/server/async-helpers.ts:22:5
    at new Promise (<anonymous>)
    at inMeteor (imports/server/async-helpers.ts:21:10)
    at tryDeliver (imports/server/drivers/mail.js:159:22)
    at imports/server/drivers/mail.js:201:20
    at Array.map (<anonymous>)
    at imports/server/drivers/mail.js:155:45
    at /programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40

E-mail delivery failure: Error: No such grain: jobmnc
    at imports/server/drivers/mail.js:167:25
    at imports/server/async-helpers.ts:24:17
    at imports/server/async-helpers.ts:12:3
    at runWithEnvironment (packages/meteor.js:1286:24)
    at packages/meteor.js:1299:14
    at imports/server/async-helpers.ts:22:5
    at new Promise (<anonymous>)
    at inMeteor (imports/server/async-helpers.ts:21:10)
    at tryDeliver (imports/server/drivers/mail.js:159:22)
    at imports/server/drivers/mail.js:201:20
    at Array.map (<anonymous>)
    at imports/server/drivers/mail.js:155:45
    at /programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40

[ocdtrekkie]

Does your config file say it's using port 25 for inbound email? In most cases, it's port 30025 by default. And is that port blocked? I'm wondering if port scanners or something might be causing it.

[jdougan]

My sandstorm.conf. I just did a normal installation, taking defaults. The initial version installed was 303, and it has updated successfully to 305.

SERVER_USER=sandstorm
PORT=80
MONGO_PORT=6081
BIND_IP=0.0.0.0
BASE_URL=https://corvid.sandcats.io
WILDCARD_HOST=*.corvid.sandcats.io
UPDATE_CHANNEL=dev
ALLOW_DEV_ACCOUNTS=false
SMTP_LISTEN_PORT=25
SANDCATS_BASE_DOMAIN=sandcats.io
HTTPS_PORT=443

Since this says 25, I assumed 25. I was (and still am) using a hosting mail port block feature, and after a quick test it appears to be working. Telnetting to 30025 gets connection refused, as it should. So it is unlikely to be an external port scanner.

Bug 1: Incorrect docs or installation script.

I can get to 25 via localhost, so that suggests one of the other sandstorm apps may be doing something. Sandstorm (tt-rss and davros) is all I am running on this VPS.

The part that confuses me, is where it is getting the grain names from. As I understand it, the mail server maps randomly generated ids back into into grains then delivers them. None of the grain names is anything thing I have ever had running. Where did they come from? Is it residue from installation?

And this appeared in in the last 20 minutes, so whatever it is it is still an issue.

E-mail delivery failure: Error: No such grain: spameri
    at imports/server/drivers/mail.js:167:25
    at imports/server/async-helpers.ts:24:17
    at imports/server/async-helpers.ts:12:3
    at runWithEnvironment (packages/meteor.js:1286:24)
    at packages/meteor.js:1299:14
    at imports/server/async-helpers.ts:22:5
    at new Promise (<anonymous>)
    at inMeteor (imports/server/async-helpers.ts:21:10)
    at tryDeliver (imports/server/drivers/mail.js:159:22)
    at imports/server/drivers/mail.js:201:20
    at Array.map (<anonymous>)
    at imports/server/drivers/mail.js:155:45
    at /programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40

So, lets see what the mail server does:

root@sandstorm:/opt/sandstorm# telnet localhost 25
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 sandstorm ESMTP Sandstorm Mail Server
HELO opencobalt.net
250 sandstorm Nice to meet you, []
MAIL FROM: <jack@opencobalt.net>
250 Accepted
RCPT TO: <68768766678666687687687676@corvid.sandcats.io>
250 Accepted
DATA
354 End data with <CR><LF>.<CR><LF>
From: jack@opencobalt.net
To: 68768766678666687687687676@corvid.sandcats.io
Subject: Test 01

This is a test 01
.
450

Gives me

E-mail delivery failure: Error: No such grain: 68768766678666687687687676
    at imports/server/drivers/mail.js:167:25
    at imports/server/async-helpers.ts:24:17
    at imports/server/async-helpers.ts:12:3
    at runWithEnvironment (packages/meteor.js:1286:24)
    at packages/meteor.js:1299:14
    at imports/server/async-helpers.ts:22:5
    at new Promise (<anonymous>)
    at inMeteor (imports/server/async-helpers.ts:21:10)
    at tryDeliver (imports/server/drivers/mail.js:159:22)
    at imports/server/drivers/mail.js:201:20
    at Array.map (<anonymous>)
    at imports/server/drivers/mail.js:155:45
    at /programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40

Bug 2: Error message should probably be changed to "No such grain id"

This implies that either Davros or TT-RSS is sending mail. Is that even allowed? I figured there would be a capability to allow mail sending.

[ocdtrekkie]

I believe the current mail implementation is a legacy hack, but I'm reasonably sure neither TTRSS nor Davros use it.

I am almost positive this refers to mail coming from outside directed at Sandstorm. 🤔

[jdougan]

I am almost positive this refers to mail coming from outside directed at Sandstorm. 🤔

How? Port 25 is blocked. Try telnet corvid.sandcats.io 25 and you'll get a connection timeout.

root@ss20220603a:/opt/sandstorm# telnet corvid.sandcats.io 25
Trying 64.34.218.58...
telnet: Unable to connect to remote host: Connection timed out
root@ss20220603a:/opt/sandstorm#

In Admin/Email Configuration, is says "Email is unconfigured. Email-related features like email login, notifications, sharing invitations, and requesting access will be unavailable until you configure email." and the SMTP host field is blank. Is there anything else that has to be done to turn outbound mail off?

Is there any way of just turning inbound mail off?

[ocdtrekkie]

Is there anything locally on the machine that could be sending to port 25? 🤔

I mean, I'm not sure it can do any harm, you have no apps that can process inbound mail anyways. But it is strange.

[jdougan]

I've got a support email into the hosting provider to see if there is anything weird about the port block.

Should be either a config option or something in the sandstorm admin pages to turn off incoming mail. The less attack surface the better.

[ocdtrekkie]

I agree. Honestly the docs suggest if you remove the config line entirely it will just use 30025, and it seems to me if the config is missing we should just not do inbound mail. It's currently not critical functionality to begin with.

[zenhack]

30025 as a default agrees with what it looks like the code does.

I think I agree that the "right" behavior would be to just not listen via SMTP. I'm mildly hesitant to change it though since it is possible someone is relying on the current behavior. Either way, providing some way to disable inbound mail would probably be good.

[jdougan]

Bug 2: Error message should probably be changed to "No such grain id"

My attempt at fixing the error message: #3683

[ocdtrekkie]

Have you had any luck determining the source of these?

[jdougan]

Apparently a config glitch at the hosting company, the port block was not applied to the internal datacenter IP. The hosting co is in the process of determining who is spamming us.

[jdougan]

Sadly, since the logs in sandstorm aren't very good, I can't help them with mail headers or anything. I've moved the mail port off to 30025 for now and stuck a python smtpd debugging server on 25 to see if it catches anything.