ClojureQL: 'totally different in every regard'
Interview: ClojureQL Overhaul
ClojureQL creator Lau Jensen recently took the drastic measure of deleting his two-and-half-year long project and starting from scratch. JAXenter caught up with him, to find out why this was necessary, and what the overhaul has added to the project.....
JAXenter: Can you introduce us to your ClojureQL project?
Lau Jensen: ClojureQL is an implementation of the primitives defined in Relational Algebra (RA) written in and for Clojure. RA gives you a lot of power to express database queries and relations without having to learn specific low-level languages and their respective syntax, ie. SQL. With ClojureQL you have a relatively small API, with a very consistent design, that lets you express complex queries through very simple lisp expressions.
JAXenter: After two and a half years, you recently deleted the ClojureQL project and started from scratch. What prompted such a drastic move?
Lau Jensen: The initial project was created shortly after Clojure's release to the public. Since that time both Clojure and I have improved substantially.
The first project aimed at abstracting away all of the SQL statements, both DDL (creation) and DML (queries/modification)and we wanted to accomplish this for a myriad of backends. Because we didn't settle on using RA, we also had to design the api from scratch. Our great ambitions also meant a great burden of work, much greater than what our initial 2 man crew were able to carry ourselves. Redefining ClojureQL also meant redefining our ambitions: Primarily target SQL92 (PostgreSQL and MySQL), implement primitives from RA and focus on DML. So esentially, there was no way to continue the first ClojureQL, since the revised version was totally different in every regard.
JAXenter: What improvements has going back to the start brought to ClojureQL, which wouldn't have otherwise been reached?
Lau Jensen: Firstly and most importantly, queries now compose giving the user great power of expression. We only achieved this by using the primitives from RA. The elegance of ClojureQL is the combination of Clojure and RA.
A very clean and modern implementation. The original ClojureQL spanned about 20 files and even in its uncomplete state exceeded 3000 lines of code. The revised version spans only 5 files, where the core.clj only amounts to 390 lines and the total of the project is just over 1000 lines. This makes for a simpler and more easily maintained project, which contributors are finding much easier to work with.
Better Clojure integration. Because of advances in Clojure itself (protocols and records) we are able to make a much tighter integration with Clojure since we can now implement methods for interfaces such as IDeref, IFn, IKeywordLookUp etc which previously were not available. Right now our implementation of IDeref is a big part of why composing queries looks so elegant.
JAXenter: And what are the next steps for the new-look ClojureQL?
Lau Jensen: There are several things on the roadmap, but its important to stress that this is much more than a new-look, it's a completely revised, rewritten and redesigned project.
Firstly I expect the release of 1.0.0 will trigger many requests for special SQL operators like WITH RECURSIVE etc, many of these will likely make it into versions before 2.0.0.
Secondly I would like to simplify the way the compiler works, possibly by internally redesigning the AST (Abstract Syntax Tree, our internal representation of queries and relations). Right now it is very possible for users to contribute entire compilers and plug them into ClojureQL without needing any assistance from me. But it's not as easy as I would like it to be, because they have to first read and understand how the AST works (which isn't documented yet) and secondly they have to read and understand the sql92compiler.clj - the latter being the simplest of the two. If most of the compiler work could be done in the AST, writing a compiler would be as simple as rearranging the words of a sentence. This is a likely milestone for 2.0.0.
Thirdly I would like to implement some high quality connection pool management.
And finally, I would like to see more backends be contributed so that people could enjoy all the features of ClojureQL on Sqlite, Derby, H2 etc. Currently most things actually do work, but its the final 10% that will eventually annoy users.