Vectorized (Columnar) Execution Engine

Introduction

• This is a very basic Vectorized Execution Engine extracted from MatrixOne database.

• The query engine is in query engine pkg. The core code is in txn_executor

• Some of the packages have be renamed and ordered for easy understanding.

• This supports a very primitive Projection and Abs function. Hopefully, this should give you a high level overview of how Columnar Execution Engines Works.

• For Volcano based execution model, refer to my TinyDB Java Project.

  • You can see the use of Next() being applied at row/record level.
  • You can see how this is a pull based approach.

• In Vectorized execution

  • We do a push based approach, by storing the batch in Process Register.
  • We execute operation on the Column level, as see in the ColExec.
  • These operations can make use of SIMD if required.

Pending issues

• Passing Attr to the result Batch
• Documentation
• Implement a Parquet based Storage Engine/Reader
• Implement JOIN
• Replace FunctionName with FunctionID

Good Reads

• CockroachDB Article
• A good structure rust project
• SQL Parser

Pending Reads

• Crate Issue with Links
• Push vs Pull based
• SIMD

Disclaimer

Some of the code could be wrong or buggy. I am also in the learning phase. If you find a bug, please feel free to raise a PR.