Internals

Overview

This package is modeled closely after LINQ. If you are not familiar with LINQ, this is a great overview. It is especially recommended if you associate LINQ mainly with a query syntax in a language and don't know about the underlying language features and architecture, for example how anonymous types, lambdas and lots of other language features all play together. The query syntax is really just the tip of the iceberg.

The core idea of this package right now is to iterate over NamedTuples for table like data structures. Starting with a DataFrame, query will create an iterator that produces a NamedTuple that has a field for each column, and the collect method can turn a stream of NamedTuples back into a DataFrame.

If one starts with a queryable data source (like SQLite), the query will automatically be translated into SQL and executed in the database.

The wording of methods and types currently follows LINQ, not julia conventions. This is mainly to prevent clashes while Query.jl is in development.

Readings

The original LINQ document is still a good read.

The The Wayward WebLog has some excellent posts about writing query providers:

• LINQ: Building an IQueryable Provider – Part I
• LINQ: Building an IQueryable Provider – Part II
• LINQ: Building an IQueryable Provider – Part III
• LINQ: Building an IQueryable Provider – Part IV
• LINQ: Building an IQueryable Provider – Part V
• LINQ: Building an IQueryable Provider – Part VI
• LINQ: Building an IQueryable provider – Part VII
• LINQ: Building an IQueryable Provider – Part VIII
• LINQ: Building an IQueryable Provider – Part IX
• LINQ: Building an IQueryable Provider – Part X
• LINQ: Building an IQueryable Provider – Part XI
• Building a LINQ IQueryable Provider – Part XII
• Building a LINQ IQueryable Provider – Part XIII
• Building a LINQ IQueryable provider – Part XIV
• Building a LINQ IQueryable provider – Part XV (IQToolkit v0.15)
• Building a LINQ IQueryable Provider – Part XVI (IQToolkit 0.16)
• Building a LINQ IQueryable Provider – Part XVII (IQToolkit 0.17)

Joe Duffy wrote an interesting article about iterator protocolls:

• Joe Duffy on enumerating in .Net

On NULL values and 3VL in .Net.