Update - AWS has since released a private registry service
I attended this talk at a GoLangPhilly meetup back in August where the speaker, Peter Shannon, detailed the setup his company has been using to run docker in production. Every host there runs a local docker registry server backed by S3.
That was the first I’d heard of this setup. Usually posts discuss running a private registry in a dedicated spot (usually on AWS, behind an ELB) and funneling all docker pulls and pushes through it. Eliminating authentication issues and a single point of failure (except s3 of course) seem like a big win with little downside. I decided to try it out.
I realize tutorials for creating an S3-backed private registry are easily found on the web and the docs are good, but I haven’t seen a single version I’d follow beginning to end yet. Here is the way we do it at RealScout. Briefly, the steps covered involve:
- Create a bucket
- Grant bucket access
- Run the registry container
- Using the registry
In the end, we’ll be able to run docker containers on EC2 and outside of EC2 in pretty much the same way, both using a highly available S3-backed private registry.
Create the bucket
I’ve given up on doing this with Cloudformation and just do this rare operation via the AWS console.
Configure the bucket
Configure bucket access permissions
Access to the bucket backing the registry currently works two different ways in our environment, depending on whether you’re accessing from within EC2 or outside. Within EC2, IAM roles can be utilized to control access. Outside of EC2, we use IAM user policies to manage access. I hope to spend some time with aws-mock-metadata or hologram to eliminate this disparity.
The policy looks like this for the role and user, using this gist:
This permits only pulls. Add
s3:PutObject for pushing.
Run the registry container
Using this config,
dev-config.yml in my case:
docker-compose up -d on an EC2 box with the above policy attached to its IAM Role.
Outside of EC2, use a similar
docker-compose.yml with credentials mounted in via
That credentials file follows the standardized credentials file format.
If all is well, you should be able to list the available repos:
The docker docs are really quite good, https://docs.docker.com/registry/introduction/ is a great place to continue on from here.
We haven’t hit any issues with this configuration yet but do worry a bit about eventual consistency issues with s3 that a single registry server would address.