Last week was the last email I sent via Substack! I have been wanting to migrate my newsletter over to a self-hosted platform for a while now and Substack's recent decisions was just what I needed to make the push.
The migration however hasn't been plain sailing and there are a few bits I still need to sort out before I am happy with the setup. I thought for this issue I would share some of the original thinking for the migration and some of the particular issues I have encountered along the way.
Hopefully this will be useful for anyone who is looking to migrate as well or an otherwise interesting read.
The Initial Plan #
In addition to sending out my newsletter via Substack I have been posting all the issues on my website as well. For SEO purposes these have all had the canonical link pointing back to the original article on Substack.
I would have liked to have a canonical link on Substack pointing to my website instead but Substack doesn't allow this.
My original plan was to host Ghost as a headless CMS used purely in the backend for sending the emails out. The content would then publicly sit under /newsletter on my website.
As a backend service I wasn't expecting much traffic, so I decided to host Ghost on Railway (referral link). If you have not heard of Railway they are similar to Vercel or Heroku but generally a lot faster to set up and cheaper. You get $5 free a month so in most cases hosting is free.
I am already hosting Strapi and n8n on Railway, so it seemed like a good fit for another backend service.
I then planned to set up a redirect via an AWS CloudFront lambda to redirect all requests to https://newsletter.alexhyett.com/p/* to https://newsletter.alexhyett.com/*. This is so that any content someone has linked to on Substack still works.
Seems like a good plan but as always in software engineering there are always some unexpected consequences.
Gotchas #
To make Ghost a headless CMS they say to make the site private which stops it from being discoverable from the search engines and adds a password to the main site.
I originally set Ghost up on ghost.alexhyett.com and everything seemed to be working well. Ghost has a nice embed form as well which I integrated into my website. It doesn't quite match the style I had before, so I might change this in the future, but it works.
The first gotcha I discovered was on the opt-in emails. When someone signs up they are emailed to confirm their address. The email gives them a link back to Ghost to make them a subscriber.
Naturally the link goes back to ghost.alexhyett.com so there goes having a hidden backend service. The main issue is that subscribers were then greeted with a password required page. It seems the confirmation worked as I can see the subscriber in the list, but it would definitely be confusing to all my subscribers.
It turns out I am not the only one to have this issue and Ghost just isn't designed to work as a headless CMS for sending newsletters, which is a shame.
So I switched the ghost installation over to newsletter.alexhyett.com and made the website public. I then imported my Substack posts and subscribers over using the import tool under Settings > Labs > Beta > Substack migrator. This worked really well. I am not running a paid newsletter, but I have heard that you can easily migrate your paid subscribers over as well and then not have to pay Substack 10% of your revenue.
Now that I am having to host the website on newsletter.alexhyett.com I don't have the option to use a separate redirect to my website.
However, I still needed to make sure the old substack URLs would work. It turns out this is quite easy with Ghost as they have a redirect feature under Settings > Labs > Redirects. I just needed to upload a redirects.yaml file with the following contents:
301:
^\/p\/([a-z0-9-]+)\/$: /$1
/subscribe?: /#/portal/signupThis redirects all posts under /p/ to the correct place as well redirects any subscribe links I had set up.
There are still a few of things I need to set up:
- Images from my imported posts are coming from Substack's AWS bucket
https://substack-post-media.s3.amazonaws.com. So I need to go through each post manually and update the images. This is why my Substack hasn't been deleted yet. - Ghost doesn't send a welcome email after they have confirmed their email address. I need to set up an integration with Mailgun to send this out.
- As I am using Docker, the above redirect file isn't persisted when the app is redeployed.
If you want to set up Ghost on Railway you can use this Dockerfile:
FROM ghost:5-alpine as cloudinary
RUN apk add g++ make python3
RUN su-exec node yarn add ghost-storage-cloudinary
FROM ghost:5-alpine
COPY $GHOST_INSTALL/node_modules $GHOST_INSTALL/node_modules
COPY $GHOST_INSTALL/node_modules/ghost-storage-cloudinary $GHOST_INSTALL/content/adapters/storage/ghost-storage-cloudinary
RUN set -ex; \
su-exec node ghost config storage.active ghost-storage-cloudinary; \
su-exec node ghost config storage.ghost-storage-cloudinary.upload.use_filename true; \
su-exec node ghost config storage.ghost-storage-cloudinary.upload.unique_filename false; \
su-exec node ghost config storage.ghost-storage-cloudinary.upload.overwrite false; \
su-exec node ghost config storage.ghost-storage-cloudinary.fetch.quality auto; \
su-exec node ghost config storage.ghost-storage-cloudinary.fetch.secure true; \
su-exec node ghost config storage.ghost-storage-cloudinary.fetch.cdn_subdomain true; \
su-exec node ghost config mail.transport "SMTP"; \
su-exec node ghost config mail.options.service "Mailgun";You then need to set up a MySQL instance on Railway and then set the variables for your Ghost app to:
url=<your newsletter url>
database__client=mysql
database__connection__host=${{MySQL.MYSQLHOST}}
database__connection__user=<mysql user>
database__connection__password=${{MySQL.MYSQLPASSWORD}}
database__connection__database=${{MySQL.MYSQLDATABASE}}
CLOUDINARY_URL=<cloudinary url>
database__connection__port=${{MySQL.MYSQLPORT}}
mail__options__auth__user=<mailgun smtp username>
mail__options__auth__pass=<mailgun smtp password>
mail__options__host=<mailgun smtp server>
mail__options__port=<mailgun smtp port>
mail__options__secure=false
mail__options__requireTLS=true
mail__from=<mailgun email address>This is working for now, but I am likely going to move my Ghost installation over to an AWS Lightsail instance in the near future. Having it hosted via Docker causes a few limitations when it comes to theming and saving some of the settings in Ghost.
❤️ Picks of the Week #
👤 Portfolio - Gameboy inspired developer portfolio. This is one awesome portfolio inspired by Gameboy. It even has a little character that you can move around the screen with the arrow keys. Well done Matteo this is awesome and very memorable.
📝 Article - The Hacker News Top 40 books of 2023. I love finding new books to read and what better way than to scan Hacker News. I will definitely be adding some of these to my to read list.
📝 Article - FAQ on leaving Google. It seems lay-offs are in full swing again as I know other ex-colleagues from other companies have been let go too. Time to update your CV if you haven't already.
📝 Article - How Apple built iCloud to store billions of databases. If you wanted to know how iCloud works then this might be of interest to you.
🛠️ Tool - Willow Protocol. Willow is a new protocol for peer-to-peer data stores. One of its creators describes it better than I could:
More than ever, we need networking protocols which are resilient, privacy preserving, bandwidth conserving, able to run on low-spec hardware, and not quite as preoccupied with being the global network for everyone ever.
📝 Article - 6174 - Wikipedia. Warning this might make your brain hurt. I am not sure if there is a bigger reason for why this works, but it is interesting.
📝 Article - The Labor of Inspiration - "More to that" posts are always great. This is an interesting one about inspiration which I can relate with. I write this newsletter every week and generally have a set time that I sit down and write. However, most weeks I am not entirely sure what I am going to write about. Most of the time I just let the inspiration fairy do it's thing.
💬 Quote of the Week #
"Life is a series of tradeoffs, and greater results usually require greater tradeoffs. The question is not, "Do you want to be great at this?" The question is, "What are you willing to give up in order to be great at this?"