4

There was a discussion on Kotlin Slack about a possibility of adding code trees to support things like C# LINQ.

In C# LINQ has many applications, but I want to focus on only one (because others a already presumably covered by Kotlin syntax): composing SQL queries to remote databases.

Prerequisites:

  • We have a data schema of an SQL database expressed somehow in the code so that static tools (or the type system) could check the correctness (at least the naming) of an SQL query

  • We have to generate the queries as strings

  • We want a syntax close to SQL or Java streams

Question: what do expression trees add to the syntax that is so crucial for the task at hand? How good an SQL builder can be without them?

voddan
  • 31,956
  • 8
  • 77
  • 87
  • 2
    I have a suspicion that this question would be better asked on http://cs.stackexchange.com – spender Jul 06 '16 at 09:21
  • Out of curiosity, what's the benefit of something like LINQ over an ORM library like JOOQ? I must confess that I don't know LINQ (I've never touched any MS technology) and my knowledge of JOOQ is kind of basic. Thanks in advance. – EPadronU Jul 06 '16 at 13:57
  • I think that question is synonymous with 'whats the advantage of LINQ over [insert pre-C#-3.0 ORM scheme here]'. I'm not really familiar with using those connectors _without_ Linq, so I cant really answer that, but I bet there are a fair number of blog posts on the subject. – Groostav Jul 06 '16 at 17:41
  • Worth mentioning: Dmitry Jemerov, who seems to be an integral part of Kotlin and works for JetBrains (though a title eludes me) said "so as for the LINQ question, expression trees are something that has been brought up, but definitely not in 1.1 timeframe, and it’s not clear when" – Groostav Jul 06 '16 at 17:42
  • I am partially interested in using Kotlin for that problem. I am not at all sure C#-2.0 has analogies of all features Kotlin 1.0 has – voddan Jul 06 '16 at 17:49

2 Answers2

3

How good a query dsl can be without expression trees?

As JINQ has shown you can get very far by analyzing the bytecode to understand the intent of developer and thus translate predicate to SQL. So in principle expression trees are not essential to build a great looking query dsl:

val alices = database.customerStream().where { it.name == "Alice" }

Even without a hackery such as bytecode analysis it's possible to get a decent query dsl with code generation. Querydsl and JOOQ are great examples. With a little bit of Kotlin wrapping code you can then write

val alices = db.findAll(QCustomer.customer, { it.name.eq("Alice") }) 

How do expression trees help building query dsl

Expression tree is a structure representing some code that resolves to a value. By having such structure generated by compiler one does not need bytecode analysis to understand what's the supposed to do. Given the example

val alices = database.customerStream().where { it.name == "Alice" }

The argument to where function would be an expression that we can inspect in runtime and translate it to SQL or other query language. Because the expression trees represent code you don't need to switch between Kotlin and SQL paradigm to write queries. The query code expressed using linq/jinq look pretty much the same regardless if they are executed in memory using POCO/POJO or in the database engine using its query language. The compiler can also do more type checking. Furthermore it's very easy to replace the underlying database with in memory representation to make running tests much faster.

Further reading:

Community
  • 1
  • 1
miensol
  • 39,733
  • 7
  • 116
  • 112
  • I can build a DSL to express your last example using only kotlin api (no reflection, no code generation, no byte code tinkering) – voddan Jul 06 '16 at 09:52
  • Correct me if I am wrong, but JOOQ uses code generation to convert a DB scheme into a static representation. In this question I assume it may be done by hand – voddan Jul 06 '16 at 09:53
  • @voddan Assuming `it` in the last example is of domain/model type `Customer` how would you do that? – miensol Jul 06 '16 at 09:55
  • `it` is `Customer`, and `it.name` is its property of some type referencing `String`, but not `String` itself. Then `==` can be redefined on it – voddan Jul 06 '16 at 10:04
  • @voddan given `class Customer(var name:String)` how would that be possible? As I mentioned the expression trees allow you to write queries as if they were executed in memory using poko objects. – miensol Jul 06 '16 at 10:19
  • I assume here that the data schema is defined in a special DSL, not POJOs. That's what ORM frameworks like Exposed do https://github.com/JetBrains/Exposed – voddan Jul 06 '16 at 10:28
  • 1
    @voddan Yes, assuming there's a schema representation in code the expression trees seem less usable. However you'll then still have 2 worlds in query code: db world where schema classes querying is used, in memory world where kotlin collections functions are used. – miensol Jul 06 '16 at 10:32
  • 1
    yes, and that's a good thing, because those situations have absolutely different different requirements and limitations. I specified that in the question – voddan Jul 06 '16 at 10:38
2

JOOQ and Querydsl:

The typical solution to ORM has been to employ either a DSL or an embedded DSL from the application logic. While great advances have been made with these schemes, culminating in JOOQ and Querydsl, there are still many caveats to such a system:

  • many of the paradigms the people writing these queries are used to (namely type safety) are either missing or different in key ways
  • the exact semantics are non-obvious: in the previous answer it is suggested that we use an extension method eq to perform a db-native equality filter. It is highly probably that a new developer will mistakenly use equals instead of eq.
  • this second point is compounded by the difficulty in testing: using a live connector with fake data is a nutoriously difficult problem, so depending on the testing procedure, the incorrect code jooqDB.where { it.name.equals("alice") } may not be discovered until much further down the development pipeline.

Jinq

Jinq is not a first party data connector. While I think the importance of this is largely psychosomatic, it is important none the less. Many projects are going to use the tools suggested by the vendors, and all of the major DB providers have Java connectors, so it is likely that most developers will simply use them.

While I have not used Jinq, it is my belief that another reason Jinq has not seen wide-spread adoption is largely because it's attempting to use a much tougher domain to solve the problem: building queries from AST's is much easier than building queries from byte code for the same reason that building the back-end of a compiler is easier than building a transcompiler. While I cannot help but tip my hat to the Jinq team for doing such an amazing job, I also cannot help but think they are hampered by their tools: building queries out of bytecode is hard. By definition, java bytecode is committed to running on the JVM, trying to retrofit that commitment for another interpreter is a very hard problem.

My current work does not permit me to use a traditional database, but if I was to switch projects, knowing that I would need a great deal of data exposure in my DAL, I would likely retreat from Kotlin and Java back to .net, largely because of Linq, rather than investigate Jinq. "Linq from Kotlin" might well change my mind.

Support from DB vendors:

The LINQ-to-SQL and LINQ-to-mongo database connectors have seen wide-spread adoption in the .net community. This is because they are first party, high quality, and act in a reasonably straightforward manner: compiling an AST to SQL (or the mongo-query-language) is at least conceptually straight forward. Many of the traditional caveats of ORM's apply, but the vendors (Microsoft and Mongo) continue to address these problems.

If Kotlin supported runtime code trees in a similar vein to Linq, and if Kotlin continues to gain traction at its current rate, then I believe the MongoDB and the Hibernate teams would be quick to start retrofitting their existing LINQ-to-X connectors to support Kotlin clients, and eventually even the bigger slower companies like Microsoft and IBM would begin to support the same flow.

Linq from Kotlin

What’s more, the exact roles the Kotlin-unique concepts of a "receiver type" and the aggressive implementation of inline might play in the Linq space is interesting. Linq-from-Kotlin might well be more effective than LINQ-from-C#.

where C# has

someInterface
  .where(customer -> customer.Name.StartsWith("A") && ComplexFunctionCallDoneByMongoDriver(customer))
  .select(customer -> new ResultObject(customer.Name, customer.Age, OtherContext()))

Kotlin might be able to make advances:

someInterface
  .filter { it.name startsWith "A" && inlinedComplexFunctionCallDoneOnDB(it) } 
  //inlined methods would have their AST exposed -> can be run on the DB process instead of on the driver.
  .map { ResultObject(name, age, otherContext()) } 
  //uses a reciever type, no need to specify "customer ->"
  //otherContext() might also be inlined

This is off the top of my head, I suspect that smarter brains than mine can put these tools to better use.

Other uses:

Its worth mentioning, the assumption made about the applications of runtime-code-AST's is false:

other [runtime AST problem domains] [are] already presumably covered by Kotlin syntax

The reason I brought this up in the first place was because I was annoyed with Kotlin's null-safety feature and its interaction with Mockito: Having spent some time researching the issue, there is no Mocking framework designed for Kotlin, only java frameworks that can be used from Kotlin, with some pain.

Some currently unsolved problems in both the java domain and the Kotlin domain:

  • Mocking frameworks, as above. With access to the AST all of the clever but bizarre tricks around argument-operation-order employed by Mockito become obsolete. Other more traditional mocking frameworks gain a much more intuitive and type-safe front-end.
  • binding expressions, for UI frameworks or otherwise, often devolve to strings. Consider a UI framework where developers could write notifyOfUIElementChange { this.model.displayName } instead of notifyOfUIElementChange("model.displayName"). The latter suffers from the crippling problem of being stale if somebody renames the property.
    • I'm very excited to see what the ControlsFX guys or Thomas Mikula might do with a feature such as this.
  • similar to Kotlin specific Linq: I suspect Kotlin’s applications here might provide for a number of tools I'm not aware of. But I'm very confident that they do exist.

I really like Linq, and I cannot help but think that with Kotlin's focus on industry problems, a Linq-from-Kotlin module would be a perfect fit and make a number of peoples lives, including mine, a fair bit easier.

Groostav
  • 3,170
  • 1
  • 23
  • 27
  • 1
    thanks for your input. I am not sure I follow your logic. Your answer contains several sections, I'll try to replay in order below – voddan Jul 06 '16 at 19:47
  • 1
    I don't have a working implementation yet, so I can't comment on the type safety. There are concerns that it is impossible to support all popular SQL operations with type safety, but I have to try to know for sure – voddan Jul 06 '16 at 19:49
  • 1
    In the implementation I have in mind (the most strait forward one?) it is impossible to mix up `eq` and `equals` since `eq` must return not a boolean, but a complex object, allowing the framework to generate SQL – voddan Jul 06 '16 at 19:51
  • 1
    I agree that LINQ is more straight forward in this field than any Kotlin since it looks a lot like SQL (that was the original idea). I don't think this is a definitive advantage because there are differences, so one can not solely rely on knowlage of SQL – voddan Jul 06 '16 at 19:55
  • 1
    In this question I do not want to discuss LINQ for local data-structures, because kotlin is much better in this respect thanks to the `inline` you've mentioned. I don't see how inlining can possibly be used for SQL queries (which are the subject here) – voddan Jul 06 '16 at 19:59
  • 1
    This is off-topic, but recently there were several mocking frameworks emerging: mockito-kt, mockito-kotlin. Some recommend powermock – voddan Jul 06 '16 at 20:06