Moving to Amazon S3

AWS Logo

Amazon Web Services (AWS) have always baffled me, but I have been determined to untagle them. To this end, I’ve been migrating my static sites into S3 and using Cloudfront distributions as CDN. This gave me some real problems to solve, which makes learning more fun.

The final candidate for migration was this here blog (baty.net) and if you’re reading this, it worked.

Here’s a rough sketch of the steps I took…

Created S3 buckets

I created an S3 bucket for www.baty.net and configured it to host a static website. I also wanted to redirect requests for the apex domain (baty.net) to www.baty.net, so I had to create a second bucket named baty.net. The baty.net bucket is configured as a redirect-only website with all requests redirected to www.baty.net. Creating a separate bucket for each host name is a little cumbersome, but not difficult.

Provisioned SSL certificate

There’s no way (that I know of) to add an SSL certificate to an S3-hosted site, so for that I had to request a certificate using Certificate Manager and attach it to a Cloudfront Distribution. I used the wildcard *.baty.net so I could reuse the cert for other subdomains.

Created Cloudfront distributions

I created a Cloudfront distribution using the www.baty.net S3 bucket as its origin1 and configured it to use the new SSL certificate. Now, Amazon is managing updates for the cert at no extra cost and, as a bonus, I get a nice CDN for content.

I needed to create a second distribution using the the same SSL certificate and the baty.net bucket’s website endpoint so that I could redirect both http and https requests for baty.net to https://www.baty.net. This wasn’t necessary but I’m sure there are old links to pages in https://baty.net and I wanted to make sure those were properly handled.

Moved DNS into Route 53

Another change I made was to move DNS for the baty.net zone into Route 53. I’ve used DNS Made Easy for years and have never had any trouble, but I wanted to go through the process with Route 53 just to see how well it worked. For $0.50/month per zone, Route 53 sure makes working with DNS and other AWS tools easy.

Updated my deployment script to use s3deploy

I use a Makefile to run my site deployment. The deploy recipe used rsync and synced files to a directory on my DigitalOcean instance. The move to S3 required that I change it so that the site’s files are synced to the S3 bucket instead. I’m using s3deploy for this. To deploy the site I just type make deploy and Make builds the site, commits and pushes to my github repo, then uploads changes to S3 using the following command:

s3deploy -bucket=www.baty.net -region=us-east-1 -source=public/

My AWS credentials are kept in environment variables2 so I don’t need to keep them in the Makefile. s3deploy is smart enough to sync based on etag hashes rather than file modification time so only files that have actually changed get uploaded3.

Invalidate objects after deployment

The nice thing about a CDN is that content is efficiently cached, sometimes for a very long time. The trouble this causes is that updates to the site are not immediately visible. Cloudfront needs to be told to re-fetch changed objects from the origin. To force this, I added an invalidate recipe to the Makefile. I tell Cloudfront to re-fetch the pages that should be updated immediately, namely /index.html, /index.xml, and /page/*. I do this using the following command:

1
2
aws cloudfront create-invalidation --distribution-id=$(DISTRIBUTION_ID) \
                                   --paths /index.html /index.xml "/page/*"

I may decide to tweak the paths later. For example, I’m not invalidating /tags/* but probably should. I only get 1000 path invalidations per month for free so I’m being conservative.

That’s it. It took me a long time to figure all of this out. AWS has deep and thorough documentation but few useful resources for getting started. I’m finally starting to get my head around how AWS services4 work and how to use them.

I’ll be watching costs closely, but best I can figure this entire setup should cost me less than $2.00/month. That includes hosting, DNS, SSL, CDN, and storage.


  1. You must use the S3 website URL as the origin rather than the bucket itself otherwise it won’t serve /index.html files when visiting just the directory (e.g. /about/ will not find /about/index.html) [return]
  2. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY [return]
  3. Amazon’s aws command line tool uses timestamp and file size when determining whether to upload a file [return]
  4. “services” is redundant here but it sounds better to me this way [return]

Dialing Down Facebook

I wish I could just delete my Facebook account and be done with it, but I can’t. Or at least I’ve been unable to convince myself that everything would be fine if I didn’t have a Facebook account. Deleting the account entirely would prevent me from seeing things that I really want to see. As disappointing as it is, Facebook is where my family communicates and no sane person would suggest I try getting them all to move elsewhere.

So, while I’d love to be one of those people who just deletes Facebook and then writes a smug blog post about it, for now I’ll be trying to at least limit my usage and exposure.

The goal is to use Facebook for keeping up with friends, browsing groups about specific topics, and occasionally posting stuff my family may find interesting. This way they can also keep up with me without me having to constantly remind them that “I still have a blog, you know!”. And let’s be honest, I’m not always worth the extra effort.

As a side effect, this should drastically reduce my exposure to the usual noise and nonsense.

First, I’ve downloaded all of my content, just in case I decide to take more permanent measures later.

Facebook archive note

Second, I’ve updated privacy settings so that only “friends” can see my stuff.

Third, I’ve unfriended everyone who is not a coworker, collaborator, or family member. (Please don’t take it personally.)

Fourth, I’ve unfollowed all pages and groups. I’m still a member of the groups but they don’t appear in my timeline. I can just visit them directly whenever I want to catch up.

My timeline is now filled only with posts from those I’m interested in seeing updates from. Well, that, and ads…so many ads. Still it’s a huge improvement.

I look forward to the day I can delete Facebook entirely, but for now I’ll just dial it down to a manageable level.

Retweets Are Trash - The Atlantic

Alexis C. Madrigal (The Atlantic)

So I began to take note each time I experienced a little hit of outrage or condescension or envy during a Twitter session. What I found was that nearly every time I felt one of these negative emotions, it was triggered by a retweet.

I’ve been noting my reactions to tweets and found the same thing - that retweets are the single most toxic component of my current Twitter feed. I will use whichever Twitter client allows me to completely disable retweets. Some, like Twitterrific let me disable retweets for individual users, but not for everyone all at once. Lifehacker posted a script for doing this on twitter.com but it doesn’t work for quoted tweets and I’d have to remember to rerun it each time I follow someone new. It might be worth a try while I wait.

The Tyranny of Convenience - NYT

Tim Wu (NYT):

An unwelcome consequence of living in a world where everything is “easy” is that the only skill that matters is the ability to multitask. At the extreme, we don’t actually do anything; we only arrange what will be done, which is a flimsy basis for a life.

And

Today’s cult of convenience fails to acknowledge that difficulty is a constitutive feature of human experience. Convenience is all destination and no journey. But climbing a mountain is different from taking the tram to the top, even if you end up at the same place. We are becoming people who care mainly or only about outcomes. We are at risk of making most of our life experiences a series of trolley rides.

From 1Password to Pass

My recent experiment with Linux has me rethinking my password management strategy.

I’ve been using 1Password to manage passwords on my Mac and iPhone. 1Password is a great way for most people with Macs to manage passwords. Since I recently started using Linux alongside my Macs, I need better cross-platform support. I could use something similar to 1Password that also runs on Linux such as KeePassX. Or, I could run 1Password for Windows under Wine.

As it happens, I’ve also been using the wonderful command line password tool “Pass” for years and love it.

Pass is basically a simple wrapper around a folder full of gpg-encrypted text files. This means I can manage password files using the usual set of either command line tools or the Mac Finder or whatever else that can shuffle files around. I can also maintain history by keeping them in a Git repo. Pass has extensions for Alfred on Mac and dmenu on Linux. Using these, I can find and copy my passwords at least as fast as with 1Password…probably faster.

The thing that had kept me from going all-in with Pass is that I didn’t have acccess to my Pass passwords on my iPhone. Then I discovered Pass for iOS. Pass for iOS is an iOS app that works with Pass passwords. Using a git repo of the encrypted password store with ssh and gpg keys, I have full access to all of my passwords from pass. It also has an iOS extension for Safari so that it works just like 1Password on iOS. Pass for iOS even has support for one-time passwords, although I haven’t tried that yet.

I have exported all of my 1Password passwords and converted them using Pass and now all my passwords are stored as encrypted text files in a nice, organized folder hierarchy. I can copy them, move them, and back them up using any of the usual tools. This feels liberating.

I’ll keep 1Password around for a month or two just in case, but unless something weird happens with Pass, I’ll be able to cancel my 1Password subscription. The fewer subscriptions I have to deal with, the better.

UPDATE 2018-05-18 Seemed like a good idea at the time. Now that 1Password X supports Firefox on Linux I’m much happier with that setup. Pass is fine and free but 1Password is really very good and sooo much easier.

Back to Lightroom...for now

Photo: Petapixel.com

I keep trying to move my photo processing and library workflow out of Lightroom, and I keep failing. This post is just me thinking it through (again).

First, why would I want to do that? Good question. It’s popular to hate on Adobe, and some of this is justified, but sometimes it feels like bandwagoning.

For me, the urge to move away from Lightroom is mostly due to Subscription Fatigue. I would love to not have to pay $10/month for the rest of my life just to manage my photo library. It’s not the price, as $10/month for the photography package is a fine deal. It’s just that it weighs on my mind, and wouldn’t it be nice to not think about it?

Here are the things I’ve tried.

Apple Photos

This is the quick and easy option. With iCloud Photo Library I can have everything everywhere all the time. The editing features are pretty good and it can use things like Luminar as an external editor. I don’t have to import my iPhone photos separately.

But, a requirement for me is that all of my photos must be kept in a set of folders that I create and maintain. I don’t mind so much if there’s a “Library” for metadata and edits and such, but I can’t abide not having a “Show in Finder” command available. I know I should just get with the program, but I’m a little stuck in my ways here so Apple Photos is not yet the right thing for me.

Photo Mechanic and an editor

Photo Mechanic is by far the best way I’ve found to ingest, caption, and keyword photos. It’s super fast and is made for this. It works directly with the Finder so my precious folder system remains intact. There’s no library at all - it’s just a browser. Select an image, hit “E” and the image opens in whatever editor I have configured. It does the right thing with raw files. It can export/rename/upload to just about any service or format.

Photo Mechanic is the option I want to use. It’s simple, lightweight, fast and flexible. But sometimes I want a library. I want to make collections or do some fancy searching. I also kind of want to edit “in place” without sending files to a separate editor and back. I may end up back here someday, but for now the benefits don’t outweigh the effort.

Capture One

I love Capture One, and it’s the most likely alternative. Earlier this year I tried going all-in. I’ve done this before, and it almost stuck this time. Capture One does a great job with Fuji Raw files. Although it’s easy to be brainwashed by The Internet into thinking that Capture One is great with Fuji and Lightroom sucks. I don’t find that to be true any more. Lightroom does fine, and Capture One is slightly better.

The UI of Capture One takes a bit of getting used to, but it’s very customizable and quite nice once I got settled in.

It just doesn’t have the export options or plugins or ecosystem of Lightroom. I’ve tweaked my process in Lightroom over the years to the point where I can crank through image processing, filing, exporting, and sharing without thinking much about it. With Capture One it feels like I’m always swimming upstream. That would eventually pass, but why bother? Capture One costs pretty much the same as Lightroom if I pay for the upgrades each year or so.

Lightroom CC

I really wanted to like the new Lightroom CC, but I didn’t. I’ll keep an eye on it but at this point it feels like an Apple Photos workalike with the overhead of the Adobe subscription. Also, no “Show in Finder” command which I still can’t get past. It’s certainly one to watch.

Luminar

Luminar is a very cool new photo editor. There’s no library yet, and it’s a bit slow on my machines. We’ll see what happens when they introduce the library features but I’m not ready to throw all my efforts behind it yet.

Lightroom CC Classic

So here I am, back in Classic. For me, right now, the familiarity, ease, flexibility, and power that Classic gives me is unmatched by any of the other options. Even though I’m not always comfortable with Adobe, I’m comfortable with Lightroom. I’m used to it. My fingers are hardwired for using it, and I get the results I want quickly and easily.

There is a fear, probably justified, that Adobe will break their promise to keep Classic around indefinitely the way they did with the downloadable version. I worry about Adobe abandoning Classic, leaving me hanging.

But let’s say they do discontinue Classic in, say, three years. What’s the difference between being forced to move off Classic in three years and deciding to move off it now, other than it would be on my terms if I did it now? I guess I could congratulate myself for ending one more subscription but is it really worth it? Right now, for me, it’s not.

And who knows, the new CC could become just as good or better than Classic. Or one of the alternatives could be even better. Or something new could come along and sweep me off my feet.

Point is, I’m already invested in Lightroom so fighting to move to something else just in case seems like unnecessary effort, and for what? So I can be free of one of the most useful subscriptions I pay for? So that if it goes away someday I’ll be able to say “I told you so!” but only because by that time I’ll have forgotten the pain I’d already suffered through when switching?

Adobe may do things I don’t love, and no, I don’t completely trust them, but right now they offer a darn good photo management and editing solution that I’m comfortable with.

I’ll try to remember all of this next time I get the sudden urge to move to something else for one reason or another that day.

A New Home for Josie

Josie (2017)

It’s breaking my heart, but I decided I needed to find a new home for Josie.

I adopted Josie last spring from the Humane Society in Grand Rapids. I knew the minute we met that I wanted to bring her home, so I did. And I fell in love.

She is sweet, funny, playful, and affectionate. She doesn’t chew things she’s not supposed to. She doesn’t bark. She doesn’t poop in the house. She curls up next to me on the couch. She loves car rides and chewies and sticks.

Josie is 99% perfect, but she simply can’t be around other dogs.

I’d never met a dog that just never figured out how to get along with other dogs, so I just assumed I’d figure things out. It’s not that I didn’t work at it. I worked at it. I spent thousands of dollars on training and consulting. I brought her to “finishing” school, where she did pretty well in groups. This was encouraging, but whenever she was one on one with another dog she just couldn’t control herself. And even with all the training, I had trouble controlling her.

I don’t give up easily on dogs. I’ve had dogs that needed to be carried around in order to pee or get up stairs. I had dogs needing multiple surgeries and long, expensive, recovery periods. I’ve had dogs that couldn’t control their bladders and would just pee everywhere all the time. I never once considered getting rid of them.

My life involves dogs. Everyone I know has dogs. My girlfriend has two small dogs. None of us can ever be in the same place when Josie is with me. I worry that someone’s dog is going to get hurt. After nearly a year of working at it, I didn’t know what I could do to resolve all this.

So, a couple weeks ago I contacted one of Josie’s trainers and asked for advice. It turns out that his sister had recently lost a dog, lived alone with no other pets, and was interested in meeting her. She took Josie home for a week to see how they got along. She (of course) fell in love immediately and expressed an interest in keeping Josie. I said yes.

I’m told by people who know this woman that Josie will be well cared for and probably spoiled a little. I’m ok with that.

Part of the reason I’m writing this is to convince myself that I shouldn’t hate myself for failing her. Adopting a dog isn’t something to be taken lightly with just an easy “never mind” if things get inconvenient. I hope that’s not what I’ve done. I don’t think I gave up too easily. I think I’ve done the right thing, but damn it hurts.

Josie will be fine. Considering the easy access she’ll have to professional dog people, I expect her to be happy and to thrive.

Still, I miss her terribly.

Book: The Lathe of Heaven ★★★½

The Lathe of Heaven cover

I finally got around to reading Ursula K. Le Guin’s “The Lathe of Heaven” and I hate to say that I was a tad underwhelmed. It’s a novel beloved by so many that perhaps I was expecting too much. It’s short enough that I may read it again some day to see if it strikes me differently.