custom domains - middleware and verification #2145

Soxasora · 2025-05-01T23:59:27Z

Description

Part of #1942
Focuses on custom domain insertion, providing a working form and verification process via pgboss.
Uses ACM with LocalStack to provide certificates CRUD simulation.

Screenshots

TBD

Additional Context

#1958 is being split to facilitate review and build greater stuff on top of it.
Doesn't contain Auth Sync or custom branding.

This PR provides:

custom domains
-- context
-- crud, resolvers, fragments
-- form with basic UI/UX
-- ACM certificates support
-- local DNS server via dnsmasq support
middleware
-- custom domain detection via endpoint
-- doesn't apply referrals yet (no rewrites here)
domain verification
-- job every 5 minutes
-- AWS error handling
-- log every verification attempt and its steps
-- normalized schema for Domain, Record, Attempt, Certificate
triggers
-- on territory stop, HOLD the domain
-- on territory takeover, delete the domain
-- on domain/domainCertificate deletion, delete from ACM via pgboss job
workers
-- full domain verification
-- clear domains that have been on HOLD for more than 30 days
-- delete certificate from ACM (job)

Checklist

Are your changes backwards compatible? Please answer below:
Yes, everything is an addition. An exception is the way we handle S3 via localstack, now the container is called AWS and also allows ACM simulations other than S3.

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:
7, QA OK, domain verification handles errors correctly and follows its step, middleware correctly detects a custom domain and obtains its connected sub.

For frontend changes: Tested on mobile, light and dark mode? Please answer below:
Yes, it uses standard SN colors and styles that fits with theme changes.

Did you introduce any new environment variables? If so, call them out explicitly here:
An endpoint for localstack that can be contacted by our worker container
LOCALSTACK_ENDPOINT=http://aws:4566

local DNS server via dnsmasq, with fallback to 1.1.1.1 in both dnsmasq and code.
DOMAINS_DNS_SERVER=172.20.0.2:53

- ACM support - custom domains crud, resolvers, fragments - custom domains form, guidelines - custom domains context - domain verification every 5 minutes via pgboss - domain validation schema - basic custom domains middleware, to be completed - TODOs tracings

- CustomDomain -> Domain - DomainVerification table - CNAME, TXT, SSL verification types - WIP DomainVerification upsert

…ange status of a Record from its Attempt, multi-purpose dns verification

…tch as they can fail

- use DomainVerificationStatus enum for domains and records - adapt Territory Form UI to new schema - return 'records' as an object with its types - wip: prepare for attempts and certificate usage for prisma

…tion triggers

fix: - fix setDomain mutation transaction - fix schema typedefs enhance: - DNS records guidelines with flex-wrap for longer records cleanup: - add comments to worker - remove console.log on validation values

…main

huumn · 2025-05-14T20:00:05Z

https://github.com/stackernews/stacker.news/blob/c7321351bfef74dc2cfa9d0c2542bb3fbbc5db2c/docs/dev/custom-domains-base.md

So to QA this I need to use real DNS? I suspect we'll want to mock DNS somehow, like we mock ACM etc, going forward to make this trivial to iterate on.

huumn

I did a very casual high level first pass to get acquainted. It looks very good.

Thank you for separating it into its own PR. Also thanks to @ekzyis for his first review which I'm sure led to lots of improvements.

I left some questions and nits. I'll make another high level pass tomorrow. Then maybe again depending on when we can test this in a self-contained way.

api/acm/index.js

api/resolvers/domain.js

pages/api/domains/index.js

huumn · 2025-05-15T00:38:58Z

prisma/migrations/20250504202129_custom_domains_base/migration.sql

+-- !QUIRK: If someone else takes over the sub, the new guy needs to delete the domain
+CREATE OR REPLACE FUNCTION hold_domain_and_delete_certificate()


That is weird. I guess it's not clear to us whether the transfer is from an alt to another alt or to another person entirely.

I think we can actually just remove the custom domains settings and certificate if the owner is different on re-activation.

And, instead, on transfer, we retain settings

huumn · 2025-05-15T00:42:30Z

worker/territory.js

+      // make sure to delete the certificate from ACM if the sub is stopped, if we have it.
+      if (nextStatus === 'STOPPED' && sub.domain?.certificate?.certificateArn) {
+        await deleteDomainCertificate(sub.domain.certificate.certificateArn)
+      }
+
      await models.sub.update({
        include: { user: true },
        where: {
          name: subName
        },
        data: {
-          status: nextBillingWithGrace(sub) >= new Date() ? 'GRACE' : 'STOPPED',
+          status: nextStatus,


hmmm hard to tell if this is the right behavior unless we automatically re-add the cert when it's paid.

Hmm, a new certificate from ACM is fast and free, what's not really good (for the territory owner) is that they will be subjected again to ACM DNS validation.

Maybe we don't need to remove the certificate at all as long as the domain remains the same but this is also somewhat connected to what you said on the previous comment:

I guess it's not clear to us whether the transfer is from an alt to another alt or to another person entirely.

… HOLD handle territory changes via triggers - on territory stop, HOLD the domain - on territory takeover from another user, delete the domain and its associated records handle ACM certificates via trigger - on domain/domainCertificate deletion, ask ACM to delete the certificate via a pgboss job; removes the need to ask ACM in multiple places clear domains that have been on HOLD for more than 30 days, check every midnight via pgboss schedule use 'domains' profile for worker jobs

Soxasora · 2025-05-18T14:49:11Z

prisma/migrations/20250504202129_custom_domains_base/migration.sql

+-- everytime a domain (relation) or a domainCertificate is deleted, also ask ACM to delete the certificate
+CREATE TRIGGER trigger_ask_acm_to_delete_certificate


Instead of jumping around the code to ask ACM to delete a certificate (mostly in shady places), a trigger on domain/domainCertificate deletion offloads this job to our worker via pgboss.

Soxasora · 2025-05-18T14:52:12Z

prisma/migrations/20250504202129_custom_domains_base/migration.sql

+-- SCENARIO: Territory got stopped after grace period
+-- HOLD the domain when the sub is stopped
+-- this is to prevent the domain from being used by another sub;
+-- won't delete anything but it will require a new verification attempt if the sub is resumed
+CREATE OR REPLACE FUNCTION hold_domain_on_sub_stop()


Instead of deleting a domain from a territory, we put it on HOLD so that the original owner can come back without losing any customization or their ACM certificate.

The user will inevitably need to re-verify the domain to make it active again.
edit: well it can be avoided by triggering ACTIVE when a territory goes up again but imo it's not ideal

Soxasora · 2025-05-18T14:53:41Z

prisma/migrations/20250504202129_custom_domains_base/migration.sql

+-- SCENARIO: Territory got taken over by a different user
+-- clear the domain when the sub is taken over by a different user;
+-- this will delete the domain, its certificates, verification attempts, DNS records and brandings.
+-- will also trigger a request to ACM to delete the certificate because the domain is being deleted
+CREATE OR REPLACE FUNCTION clear_domain_on_sub_takeover()


Now that we only HOLD the domain on territory expiration, I think it's fair to delete any domain information, with its certificate (thanks to trigger_ask_acm_to_delete_certificate), if the territory gets resumed by another user instead of the original owner.

Soxasora · 2025-05-18T14:58:05Z

prisma/migrations/20250504202129_custom_domains_base/migration.sql

+-- Clear domains that have been on HOLD for 30 days or more every midnight.
+INSERT INTO pgboss.schedule (name, cron, timezone) VALUES ('clearLongHeldDomains', '0 0 * * *', 'America/Chicago') ON CONFLICT DO NOTHING;


I've been wanting to add this for a long time but didn't know if we actually need it, regardless, now we check every midnight if a domain has been on HOLD for 30 days or more.

There might be a better way to do this so this might get iterated?

The primary reason for this is to delete unused certificates. In the future, it would ideally be coupled with routine checks on domain ownership just to be safer.

edit: still the same but encapsulated in a function for plpgsql

…unction

…not valid or we're in production

Soxasora added 2 commits May 1, 2025 18:38

Domains refactor, Domain Verification normalization

5e80c3f

- CustomDomain -> Domain - DomainVerification table - CNAME, TXT, SSL verification types - WIP DomainVerification upsert

Soxasora force-pushed the custom_domains_base branch from cd46aeb to 5e80c3f Compare May 2, 2025 23:33

Soxasora added 3 commits May 3, 2025 17:26

Domains normalization: Attempts, Records, Certificates

4d24845

Domain Verification worker adjusted to new schema; use triggers to ch…

b624c59

…ange status of a Record from its Attempt, multi-purpose dns verification

wip: Domain Verification worker, log all verification steps

89f1eb4

Soxasora closed this May 6, 2025

wip: clearer Domain Verification flow, surround AWS calls with try ca…

9d1c137

…tch as they can fail

Soxasora reopened this May 6, 2025

Soxasora added 9 commits May 6, 2025 14:51

Domain Verification schema updates

dc119a8

- use DomainVerificationStatus enum for domains and records - adapt Territory Form UI to new schema - return 'records' as an object with its types - wip: prepare for attempts and certificate usage for prisma

HOLD the domain and delete the certificate when a territory expires

67fb2c8

delete the certificate from ACM when we're about to STOP a territory

725ce81

Domain resolver refactor, use transactions, add comments

f3930f7

Stages for Domain Verification attempts logging, fix certificate dele…

01e319e

…tion triggers

separate ACM certificate requests and validation values

e132ad0

Domains UI/UX enhancements; core fixes to schema; general cleanup

cd9cb68

fix: - fix setDomain mutation transaction - fix schema typedefs enhance: - DNS records guidelines with flex-wrap for longer records cleanup: - add comments to worker - remove console.log on validation values

delete any existing domain verification jobs if we're updating the do…

9e96d7c

…main

Log AWS-related error messages; fix deleteCertificate recursion

82a71f5

Soxasora force-pushed the custom_domains_base branch from 775a966 to 82a71f5 Compare May 8, 2025 23:33

Soxasora and others added 2 commits May 8, 2025 18:47

fix missing await on async customDomainMiddleware

f95ab6a

Merge branch 'master' into custom_domains_base

4f49382

Soxasora marked this pull request as ready for review May 8, 2025 23:55

Soxasora and others added 2 commits May 8, 2025 19:04

hotfix: delete certificate from ACM also on domain removal

2382f3b

Merge branch 'master' into custom_domains_base

c732135

huumn reviewed May 15, 2025

View reviewed changes

Soxasora added 2 commits May 16, 2025 10:52

prepare for dnsmasq, light cleanup

ca13d80

fix DNS server typo

2a77fd1

Soxasora mentioned this pull request May 16, 2025

local DNS server via dnsmasq #2168

Open

don't ask ACM to delete a certificate in a db transaction

072c1ae

Soxasora added 2 commits May 17, 2025 08:07

fix typo

7da660a

Soxasora commented May 18, 2025

View reviewed changes

Soxasora added 2 commits May 18, 2025 10:33

address plpgsql syntax issues, move INSERT for pgboss.schedule in a f…

2c4ca44

…unction

fallback to system's default DNS servers if dnsmasq is not available/…

52dd035

…not valid or we're in production

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

custom domains - middleware and verification #2145

custom domains - middleware and verification #2145

Soxasora commented May 1, 2025 •

edited

Loading

huumn commented May 14, 2025 •

edited

Loading

huumn left a comment

huumn May 15, 2025

Soxasora May 15, 2025

huumn May 15, 2025

Soxasora May 15, 2025 •

edited

Loading

Soxasora May 18, 2025

Soxasora May 18, 2025 •

edited

Loading

Soxasora May 18, 2025

Soxasora May 18, 2025 •

edited

Loading

		-- !QUIRK: If someone else takes over the sub, the new guy needs to delete the domain
		CREATE OR REPLACE FUNCTION hold_domain_and_delete_certificate()

		-- everytime a domain (relation) or a domainCertificate is deleted, also ask ACM to delete the certificate
		CREATE TRIGGER trigger_ask_acm_to_delete_certificate

		-- Clear domains that have been on HOLD for 30 days or more every midnight.
		INSERT INTO pgboss.schedule (name, cron, timezone) VALUES ('clearLongHeldDomains', '0 0 * * *', 'America/Chicago') ON CONFLICT DO NOTHING;

custom domains - middleware and verification #2145

Are you sure you want to change the base?

custom domains - middleware and verification #2145

Conversation

Soxasora commented May 1, 2025 • edited Loading

Description

Screenshots

Additional Context

Checklist

huumn commented May 14, 2025 • edited Loading

huumn left a comment

Choose a reason for hiding this comment

huumn May 15, 2025

Choose a reason for hiding this comment

Soxasora May 15, 2025

Choose a reason for hiding this comment

huumn May 15, 2025

Choose a reason for hiding this comment

Soxasora May 15, 2025 • edited Loading

Choose a reason for hiding this comment

Soxasora May 18, 2025

Choose a reason for hiding this comment

Soxasora May 18, 2025 • edited Loading

Choose a reason for hiding this comment

Soxasora May 18, 2025

Choose a reason for hiding this comment

Soxasora May 18, 2025 • edited Loading

Choose a reason for hiding this comment

Soxasora commented May 1, 2025 •

edited

Loading

huumn commented May 14, 2025 •

edited

Loading

Soxasora May 15, 2025 •

edited

Loading

Soxasora May 18, 2025 •

edited

Loading

Soxasora May 18, 2025 •

edited

Loading