Guess the utility

I invented a couple of utilities at work today. They’re really useful little go programs. So useful, in fact, that they have already existed for many years. I haven’t written in a while so I figured I’d post them here as a little guessing game. See if you can figure out what they are from the code, just note that they are not faithful reimplementations, as I didn’t originally set out to reimplement anything. Expand the description for a hint.

A tech job search in 2017

I started searching for a new role just over 2 months ago. While I had passively reviewed messages from recruiters for awhile, I finally decided to give the search a real go, getting in touch with my network, switching my status to searching on AngelList & StackOverflow jobs, and applying to a few promising positions I had found through Indeed, in #golang-jobs on gophers slack, and elsewhere. I was not interested in relocating so I targeted only Philadelphia-based, 1 day/week in NYC, or fully remote gigs.

Routetable - transit scheduling with GTFS

I put together as a little project to get to know the General Transit Feed Specification and address some of the issues I have with Google’s otherwise excellent transit routing. First, I’m willing to walk pretty far to a transit stop. Second, I want to see more options at one time. Finally, I want a hard limit on transfers (Google does provide a less transfers preference). I also really wanted to show off my amazing UX skills.

EC2 Metadata service mocking with Docker

My former employer quietly open sourced a handy utility for mocking out the metadata service on AWS’ EC2 hosts a few weeks ago. As I’ve mentioned previously, there are a few projects in this space, but none quite as simple as this one to get going.

ectou-metadata includes an implementation of the IAM security credential API that, to applications, feels exactly like running on an EC2 instance launched with an IAM Role. This means no more injecting credentials into containers needing AWS API access. It also helps increase dev prod parity.

Caffe with Python3 on Centos 7 Install Notes

I’ve posted a gist here with a Dockerfile for building a Centos 7 + Python 3 + caffe container image. I don’t expect many will use this directly but the steps and various patches might be useful for incorporation into your own Dockerfiles. As such, I’ve put no effort into minimizing the size or cacheability of layers. docker build as-is weighs in at 2.4+ GB.

The default docker run expects a /notebooks volume and exposes jupyter-notebook on port 8888.

AWS EC2 Container Registry is Better

I spent some of today with EC2 Container Registry following it’s promotion to GA status and it’s clear that it’s far better than what I was using. For $0.07 more per GB-month, you get the same features as any other private registry service:

• no need to run a local registry on every host
• global https access to a highly available registry
• registry browser UI

But you also get IAM permissions based access control!

Elasticsearch magic for IPython

Scratched an itch on Friday night while waiting for my flight back from HQ. It’s basically sense + ipython-sql and looks like this in jupyter-notebook under python3:

I’m pleasantly surprised the name was available on pypi. Check out the source on github.

Docker S3-Backed Private Registry On Every Host

Update - AWS has since released a private registry service

I attended this talk at a GoLangPhilly meetup back in August where the speaker, Peter Shannon, detailed the setup his company has been using to run docker in production. Every host there runs a local docker registry server backed by S3.

That was the first I’d heard of this setup. Usually posts discuss running a private registry in a dedicated spot (usually on AWS, behind an ELB) and funneling all docker pulls and pushes through it. Eliminating authentication issues and a single point of failure (except s3 of course) seem like a big win with little downside. I decided to try it out.

Docker Data Container Snapshots

It’s straightforward to create a data container with pre-baked archives included: COPY them on during docker build and off you go. But what if you can’t or don’t want to create the data ahead of time?

At RealScout, we recently started experimenting with the approach detailed below to create docker images from postgres snapshots for distribution to development environments. Briefly, it looks like:

1. Create and initialize a data container
2. Start postgres using the volume from that data container
3. Restore data
4. Stop postgres
5. Create an image from the populated data container

Setup

First, build a data container image, for example with this Dockerfile. :

FROM debian:jessie

ENV PGDATA /var/lib/postgresql/data

# ids must match postgres container

RUN groupadd -r -g 999 postgres && useradd -u 999 -r -g postgres postgres
RUN mkdir -p -m 0700 ${PGDATA} && chown -R postgres:postgres ${PGDATA}

VOLUME ${PGDATA}

CMD /bin/true