In the last post, we introduced the Overseer workflow engine that ran Framed, and saw how to use plain Clojure data structures and functions to wire up an example pipeline. In this post, we’ll talk about simplicity and how we were able to leverage Overseer and a variety of techniques to do away with Hadoop and other heavyweight processing frameworks.

Overseer is a library for building and running data pipelines in Clojure. Workflows are designed as a graph (DAG) of dependent tasks, and it handles scheduling, execution, and failure handling among a pool of workers. It ran the vast majority of the Framed predictive analytics platform for over a year up to the acquisition by Square, and today we’re happy to release the project as open source to the community.

Note: This is an unmodified re-publication of a post that originally appeared on Medium.com in October 2015.

At Framed, machine learning is our bread and butter. Every day, we run a lot of models over a lot of data, and recently ran up against some limits in the disk serialization format we were using to store our training datasets, which models use to discover patterns between input attributes and the outcome being predicted. The result is an efficient new serialization format we’re excited to introduce.

Now that the dust has settled from Square’s acquisition of Framed, I wanted to take some time to talk about the building blocks of simple software, and the things I think we got right at Framed.

The Framed platform was written entirely in Clojure, and at its peak it was processing hundreds of millions of analytics events per day and running machine learning models over tens of millions of input points with thousands of features.

I have always been an anxious person. It’s a pervasive undercurrent, the constant voice of ‘no’ in my head; No, I don’t want to go to that party where I only know one person. No, I’ll talk to that girl tomorrow. It’s all too easy to succumb to this voice, and I’ve been spending a lot of time thinking of what to do about it.