One of the things we allow our customers to do is purchase datasets for download. This data can be anything from a whole region, to a very drilled down query.

The setup⌗

In order to make this happen, it requires effectively three parts:

  • The user’s data request
  • A way to convert that request in to a SQL query
  • A way to convert that SQL query in to a downloadable file

Sometimes these datasets can be extremely resource intensive as well, so we don’t want to generate them each request if nothing underlying has changed. This requires some way to cache the generated file in an easy-to-recall way.

So how does this relate to Kubernetes, jobs, or management?⌗

Kubernetes is a natural choice; we wanted the ability to scale our backends independently and possibly across availability zones without have to constantly baby it. We originally considered virtual machines for this, but the overhead was far too great.

This brings us to jobs and job managers, the exciting part. Depending on the size and complexity of a user’s request, we’ve observed that a single export job can take upwards of an hour or more of constant run time. Due to the nature of the data that the user is requesting from us, we have no way of knowing upfront how long an export will run or what the size of the resulting file is. Running these exports in a background job was the obvious choice. Kubernetes has a build in Job type which was close to what we wanted on its own. The problem was that we need it to be always running, accepting an infinite number of jobs with a new pod for each job over time with an unknowable number of completions.

I needed to build something that could lie between the request to start a job and Kubernetes’ Jobs.⌗

I decided to use Redis lists for enqueuing a job, having Redis allowing a pub/sub channel for keyspace events. For the manager itself, I went with Python. Python is a nice low-overhead language that is nice and familiar with built-in async – vital to the architecture.

The job manager listens for any lpush event in any list matching *-export using an async coroutine, since we have several possible formats a user can choose from. Once Redis publishes an lpush event, the manager will pop the job ID and use the Kuberenetes API to create a new Job. The containers used for the jobs are all named such that the list name can simply be used as the container name, making the manager dynamic and somewhat future-proof. The job manager sets the job ID as an environment variable for the container, so that the actual job runner knows which export it should be processing.

When Kubernetes marks a job ‘Complete’ or ‘Failed’, another async coroutine watching for Job status changes uses another Kubernetes API to delete the job and clean up after itself. We use external error monitoring, so we don’t really care about the pod’s logs in the case of a failure.

With the manager itself being only two looping coroutines and having the container names match the Redis list name, this service can run essentially forever unless we find ourselves needing to change the Job definition.

In part 2, I’ll write about the job processing system itself. It’s a pretty fun service that leverages the best part of Kubernetes: microservices.