g.raphaelli.com

Meeting Recording Sharing as a Service

Sat, 02 Dec 2023 12:00:16 -0500

I made an Elastic Advent Calendar post here: https://discuss.elastic.co/t/dec-2nd-2023-meeting-recording-sharing-as-a-service/347282

Guess the utility

Mon, 19 Mar 2018 22:22:22 -0500

I invented a couple of utilities at work today. They’re really useful little go programs. So useful, in fact,that they have already existed for many years. I haven’t written in a while so I figured I’d post them here as a littleguessing game. See if you can figure out what they are from the code, just note that they are not faithfulreimplementations, as I didn’t originally set out to reimplement anything. Expand the description for a hint.

#1

{ ctx, _ := context.WithTimeout(context.Background(), timeout) cmd := exec.CommandContext(ctx, os.Args[1], os.Args[2:]...) cmd.Stdout, cmd.Stderr, cmd.Stdin = os.Stdout, os.Stderr, os.Stdin var err error if err = cmd.Start(); err != nil { fmt.Fprintln(os.Stderr, err) } else { err = cmd.Wait() } os.Exit(exitCode(err))}

description

solution

#2

{cmd := exec.Command(os.Args[1], os.Args[2:]...)cmd.Stdout, cmd.Stderr, cmd.Stdin = os.Stdout, os.Stderr, os.Stdinvar err errorstart := time.Now()if err = cmd.Start(); err != nil {fmt.Fprintln(os.Stderr, err)} else {err = cmd.Wait()}end := time.Now()var rusage syscall.Rusagesyscall.Getrusage(syscall.RUSAGE_CHILDREN, &rusage)fmt.Printf("rss:%d user:%s sys:%s real:%s\n", rusage.Maxrss, time.Duration(rusage.Utime.Nano())*time.Nanosecond, time.Duration(rusage.Stime.Nano())*time.Nanosecond, end.Sub(start))os.Exit(exitCode(err))}

description

solution

#3

{ var rlimit syscall.Rlimit rlimit.Cur = 134217728 rlimit.Max = 134217728 syscall.Setrlimit(syscall.RLIMIT_AS, &rlimit) cmd := exec.Command(os.Args[1], os.Args[2:]...) cmd.Stdout, cmd.Stderr, cmd.Stdin = os.Stdout, os.Stderr, os.Stdin var err error if err = cmd.Start(); err != nil { fmt.Fprintln(os.Stderr, err) } else { err = cmd.Wait() } os.Exit(exitCode(err))}

description

solution

If you like this kind of thing, many other utilities can be found in go-coreutils.

A tech job search in 2017

Sat, 11 Nov 2017 22:22:22 -0500

I started searching for a new role just over 2 months ago. While I had passively reviewed messages from recruiters for awhile, I finally decided to give the search a real go, getting in touch with my network, switching my status to searching on AngelList & StackOverflow jobs, and applying to a few promising positions I had found through Indeed, in #golang-jobs on gophers slack, and elsewhere. I was not interested in relocating so I targeted only Philadelphia-based, 1 day/week in NYC, or fully remote gigs.

Here’s part that might be helpful to others - turns out it’s kind of a mess out there.

Sometimes you won’t hear any news for a long time. That was the biggest surprise for me, especially when doing well during a round of interviews. When I’ve been in a position to hire and a good candidate comes in - someone I think would be a great addition to the team - I work to create a sense of urgency and enthusiasm do get that candidate fired up to join. That means feedback within 24 hours, end of day or first thing the following morning. During my experience in the last few weeks, I had no word for 2+ weeks after just an HR screen, twice! I was pretty sure I didn’t mess up quite that badly in either. In both cases I followed up with a “hey I like you so far, do you still want to figure out if you like me” email and both times had an initial phone screen with team set up shortly after, both times with an apology for the delay. I’m still waiting on a code challenge following the second round of talks with one of those.

Despite making it clear that relocation was not an option, some recruiters still wanted to chat and try to convince me to move San Francisco (it was always SF). While I’m flattered, it’s not a great use of either of our time. I learned this one pretty quickly and cleared it up early in the process with each company. One phone screen ended amicably after about 90 seconds but I could have avoided it altogether with an email.

Some companies, especially very small startups, require an extreme amount of time to complete the interview process. I went along with a few of these at first, investing many hours in video calls, coffees, and pairing sessions. After having some success without investing quite that much time, I was less enthusiastic about going back for yet another round at the other places and politely bowed out of their next rounds.

Getting in through the front door is pretty tough. I’m fortunate to have many years of experience, in various roles, and tales of scaling and hard problems at some notable companies - I was rejected from a number of companies without even an initial phone screen. Referrals are of course great for getting that initial interview, but most of us only have access to so many of those. I felt more successful when replying to posts in slack channels and hacker news and those sent through a mailing list than through companies’ HR portals though my supporting evidence on this is a little thin. I considered but ultimately decided against services like Hired, TripleByte, and VolkScience this time.

The point of all of this is not really to get recruiters, HR, and hiring managers to improve their processes, though it would be great if that happened, it’s more of a note to my fellow job seekers that there is a lot of strangeness in [tech] recruiting out there and you’re not weird or alone if you’ve had similar experiences. I’ve confirmed with a bunch of other developers that what I’ve described here was not unique to my search. My best advice is to stick with it and really go all in on the search with some deadline in mind for making decision. Be flexible and understanding with possible future employers but don’t let them undervalue your time.

Good luck out there.

Routetable - transit scheduling with GTFS

Thu, 21 Jul 2016 22:22:22 -0500

I put together as a little project to get to know theGeneral Transit Feed Specification and address some of the issues I havewith Google’s otherwise excellent transit routing. First, I’m willing to walk pretty far to a transit stop. Second, Iwant to see more options at one time. Finally, I want a hard limit on transfers (Google does provide a less transferspreference). I also really wanted to show off my amazing UX skills.

If you share the pleasure of riding SEPTA, you may find http://g.raphaelli.com/rt/ helpful for planning yournext commute.

EC2 Metadata service mocking with Docker

Thu, 03 Mar 2016 00:00:00 -0500

My former employer quietly open sourced a handy utility for mocking out the metadata service on AWS’ EC2hosts a few weeks ago. As I’ve mentioned previously, there are a fewprojects in this space, but none quite as simple as this one to get going.

ectou-metadata includes an implementation of the IAM security credential API that, to applications, feels exactly likerunning on an EC2 instance launched with an IAM Role. This means no more injecting credentials into containers needingAWS API access. It also helps increase dev prod parity.

Here’s how I’m using it out under Mac OSX with a stock docker-machine virtualbox VM.

First, build the docker container image. I’ve contributed a Dockerfile.

Next, plumb the well-known address in the docker-machine VM:

docker-machine ssh <machine-label> sudo ip addr add 169.254.169.254 dev eth0

Finally, launch a container bound to that IP. This works with --net if using docker networking.

docker run \ --env=MOCK_METADATA_ROLE_ARN=arn:aws:iam::ACCOUNT:role/ROLE \ --publish=169.254.169.254:80:5000 \ --volume=$HOME/.aws/mock-metadata-credentials:/home/ec2-user/.aws/credentials:ro \ ectou-metadata

If everything is wired up correctly you’ll see 200s logged by the ectou-metadata service when making API calls, like:

172.17.0.1 - - [01/Feb/2016 18:54:29] "GET /latest/meta-data/iam/security-credentials/ HTTP/1.1" 200 9172.17.0.1 - - [01/Feb/2016 18:54:29] "GET /latest/meta-data/iam/security-credentials/role-name HTTP/1.1" 200 641

Other containers running on this VM that have network access will be able to use these credentials.

I’d love to figure out how to run multiple instances of this within a single VM so different containers automaticallyhave access to different roles but for now I’ve settled on one instance per VM/set of containers. If there’s anyone outthere, please get in touch with thoughts.

There are someother goodiesbaked in that look useful. Big thanks to Monetate for opening up this code!

Caffe with Python3 on Centos 7 Install Notes

Tue, 05 Jan 2016 01:00:16 -0500

I’ve posted a gist here with a Dockerfile for building aCentos 7 + Python 3 + caffe container image. I don’t expect many will use this directly but the steps and variouspatches might be useful for incorporation into your own Dockerfiles. As such, I’ve put no effort into minimizing thesize or cacheability of layers. docker build as-is weighs in at 2.4+ GB.

The default docker run expects a /notebooks volume and exposes jupyter-notebook on port 8888.

AWS EC2 Container Registry is Better

Wed, 23 Dec 2015 00:00:16 -0500

I spent some of today with EC2 Container Registry following it’s promotion to GA status and it’s clearthat it’s far better than what I was using. For $0.07 more perGB-month, you get the same features as any other private registry service:

no need to run a local registry on every host
global https access to a highly available registry
registry browser UI

But you also get IAM permissions based access control!

The docs and AWS console walkthrough are great for getting started, the brief version is:

pip install -U awscliaws ecr create-repository --repository-name <name>$(aws ecr get-login --region us-east-1)# docker tag, push ...

docker pull permissions

The policy looks like this for the role and user, using awacs and troposphere:

# Permit AWS-hosted docker registry accessfrom awacs.aws import Allow, Everybody, Policy, Statementimport awacs.iam as iamimport troposphere.iamtroposphere.iam.Policy(PolicyName="DockerRegistryConsumerPolicy",PolicyDocument=Policy(Statement=[Statement(Effect=Allow,Action=[iam.Action("ecr", "BatchCheckLayerAvailability"),iam.Action("ecr", "BatchGetImage"),iam.Action("ecr", "DescribeRepositories"),iam.Action("ecr", "GetAuthorizationToken"),iam.Action("ecr", "GetDownloadUrlForLayer"),iam.Action("ecr", "GetRepositoryPolicy"),iam.Action("ecr", "ListImages"),],Resource=[Everybody],)]))

and generates:

"Policies": ["PolicyName": "DockerRegistryConsumerPolicy",{"PolicyDocument": {"Statement": [{"Action": ["ecr:BatchCheckLayerAvailability","ecr:BatchGetImage","ecr:DescribeRepositories","ecr:GetAuthorizationToken","ecr:GetDownloadUrlForLayer","ecr:GetRepositoryPolicy","ecr:ListImages"],"Effect": "Allow","Resource": ["*"]}]}}]

This permits login, pulling images, and browsing the repositories. Since we’re using AWS, command line tools tosubstitute for docker search are easily within reach

aws ecr describe-repositories to view the repos
aws ecr list-images --repository-name <name> to see image:tag availability.

docker push permissions

So far, I’ve simply logged in as a registry administrator (Allowed to ecr:*) when needing to push images -eventually CI will be the only user allowed to assume that role. For now that is:

docker logout https://$acct.dkr.ecr.us-east-1.amazonaws.com

followed by

$(aws ecr get-login --region us-east-1 --profile registry-admin).

Elasticsearch magic for IPython

Mon, 14 Dec 2015 05:00:16 -0500

Scratched an itch on Friday night while waiting for my flight back from HQ. It’s basically sense + ipython-sql and looks like this in jupyter-notebook under python3:

I’m pleasantly surprised the name was available on pypi. Check out the source on github.

Docker S3-Backed Private Registry On Every Host

Fri, 20 Nov 2015 02:04:18 -0500

Update - AWS has since released a private registry service

I attended this talk at a GoLangPhilly meetup back in August where the speaker,Peter Shannon, detailed the setup his company has been using to run docker in production. Every host thereruns a local docker registry server backed by S3.

That was the first I’d heard of this setup. Usually posts discuss running a private registry in a dedicated spot(usually on AWS, behind an ELB) and funneling all docker pulls and pushes through it. Eliminating authentication issuesand a single point of failure (except s3 of course) seem like a big win with little downside. I decided to try it out.

I realize tutorials for creating an S3-backed private registry are easily found on the web and thedocs are good, but I haven’t seen a single version I’d follow beginning to end yet.Here is the way we do it at RealScout. Briefly, the steps covered involve:

Create a bucket
Grant bucket access
Run the registry container
Using the registry

In the end, we’ll be able to run docker containers on EC2 and outside of EC2 in pretty much the same way, both using ahighly available S3-backed private registry.

Create the bucket

I’ve given up on doing this with Cloudformation and just do this rare operation via the AWS console.

Configure the bucket

Optional, but it’s probably a good idea to turn on server side encryption for this bucket. We usetroposphere to generate templates. This gist generates:

{ "Resources": { "DockerRegistryBucket": { "Properties": { "Bucket": "docker-registry", "PolicyDocument": { "Id": "DockerRegistryBucketPolicy", "Statement": [ { "Action": [ "s3:PutObject" ], "Condition": { "StringNotEquals": { "s3:x-amz-server-side-encryption": [ "AES256" ] } }, "Effect": "Deny", "Principal": "*", "Resource": [ "arn:aws:s3:::docker-registry/*" ], "Sid": "DenyUnEncryptedObjectUploads" } ], "Version": "2012-10-17" } }, "Type": "AWS::S3::BucketPolicy" } }}

Configure bucket access permissions

Access to the bucket backing the registry currently works two different ways in our environment, depending on whetheryou’re accessing from within EC2 or outside. Within EC2, IAM roles can be utilized to control access. Outside of EC2,we use IAM user policies to manage access. I hope to spend some time with aws-mock-metadata orhologram to eliminate this disparity.

The policy looks like this for the role and user, using this gist:

"Policies": ["PolicyName": "DockerRegistryConsumerPolicy",{"PolicyDocument": {"Statement": [{"Action": ["s3:ListBucket","s3:GetBucketLocation"],"Effect": "Allow","Resource": [ "arn:aws:s3:::docker-registry"]},{ "Action": [ "s3:GetObject"],"Effect": "Allow","Resource": [ "arn:aws:s3:::docker-registry/*"]}]}}]

This permits only pulls. Add s3:PutObject for pushing.

Run the registry container

Using this config, dev-config.yml in my case:

+++version: 0.1log: level: info formatter: text fields: service: registrystorage: cache: blobdescriptor: inmemory s3: region: us-east-1 bucket: docker-registry encrypt: true secure: true v4auth: true rootdirectory: /myroothttp: addr: :5000 headers: X-Content-Type-Options: [nosniff]health: storagedriver: enabled: true interval: 10s threshold: 3

and this docker-compose.yml:

+++registry: container_name: registry image: registry:2 ports: - 5000:5000 restart: always volumes: - ./dev-config.yml:/etc/docker/registry/config.yml:ro

You can docker-compose up -d on an EC2 box with the above policy attached to its IAM Role.

Outside of EC2, use a similar docker-compose.yml with credentials mounted in via volumes:

- ${HOME}/.aws/docker-credentials:/${REGISTRY_USER}/.aws/credentials:ro

That credentials file follows the standardized credentials file format.

If all is well, you should be able to list the available repos:

$ curl -s http://localhost:5000/v2/_catalog{"repositories":[]}

The docker docs are really quite good, https://docs.docker.com/registry/introduction/ is a great placeto continue on from here.

Notes

We haven’t hit any issues with this configuration yet but do worry a bit about eventual consistency issues with s3 thata single registry server would address.

Docker Data Container Snapshots

Fri, 13 Nov 2015 17:10:18 -0500

It’s straightforward to create a data container with pre-baked archives included: COPY them on during docker build and off you go. But what if you can’t or don’t want to create the data ahead of time?

At RealScout, we recently started experimenting with the approach detailed below to create docker images from postgres snapshots for distribution to development environments. Briefly, it looks like:

Create and initialize a data container
Start postgres using the volume from that data container
Restore data
Stop postgres
Create an image from the populated data container

Setup

First, build a data container image, for example with this Dockerfile. :

FROM debian:jessieENV PGDATA /var/lib/postgresql/data# ids must match postgres containerRUN groupadd -r -g 999 postgres && useradd -u 999 -r -g postgres postgresRUN mkdir -p -m 0700 ${PGDATA} && chown -R postgres:postgres ${PGDATA}VOLUME ${PGDATA}CMD /bin/true

$ docker build .Successfully built 221c07e4f85c

Now create a postgres data container using that image:

$ docker run --name postgres-data 221c07e4f85c$ docker ps -af ancestor=221c07e4f85cCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESd13320582ff2 221c07e4f85c "/bin/sh -c /bin/true" 1 seconds ago Exited (0) Less than a second ago hopeful_boyd

Now start a postgres server (using an image from Docker Hub here) using the volume from that data container:

$ docker run -d --volumes-from postgres-data --name postgres postgres146ad1bcbb45c74100f5345facff8be2ba9a532cbe0c5642a0c0bf840b4234e4

Now we can load up some data. At this point, you should probably scrub out anything from the database backup that you don’t want to include in the image.

pg_restore ...

The clever part

We’re going to docker build inside of a third container that has the data volume mounted inside of it. It’s not quite docker in docker, but it’s related.

First, take the Dockerfile above and COPY data/ ${PGDATA} to it so it looks like:

FROM debian:jessieENV PGDATA /var/lib/postgresql/data# ids must match postgres containerRUN groupadd -r -g 999 postgres && useradd -u 999 -r -g postgres postgresRUN mkdir -p -m 0700 ${PGDATA} && chown -R postgres:postgres ${PGDATA}COPY data/ ${PDATA}VOLUME ${PGDATA}CMD /bin/true

(We just have a single Dockerfile used for initial and snapshot builds and an empty data/ sitting around for use by the initial build.)

Now stick this in snapshot.sh:

{%raw>}}#!/bin/bash -eBUILD_WD=/tmp/buildDATA_CONTAINER=${1:-postgres-data}DOCKERFILE=${2:-Dockerfile}# like --volumes-from, but allow for mounting in a different locationDATA_VOLUME=$(docker inspect --format '{{range $mount := .Mounts}}{{if eq $mount.Destination "/var/lib/postgresql/data"}}{{$mount.Source}}{{end}}{{end}}' $DATA_CONTAINER)# stop sends TERM, like postgres wantsdocker stop -t 60 postgres# create a docker container with data mounted at /data and run docker build inside itdocker run --rm \ -v ${PWD}/${DOCKERFILE}:/tmp/build/Dockerfile \ -v ${DATA_VOLUME}:${BUILD_WD}/data \ -v /var/run/docker.sock:/var/run/docker.sock \ -v $(which docker):/bin/docker \ -w ${BUILD_WD} \ debian:jessie \ docker build .{%endraw>}}

if you used the --name flag on docker run earlier, you can snapshot with:

./snapshot.sh

I hope that helps. Please let me know if I screwed anything up.

About

Mon, 01 Jan 0001 00:00:00 +0000

Hey, I’m Gil. I’m graphaelli on most services including gmail, linkedin, github, and gitlab.