rayner.notes

How Apache Doris embeds a Rust-based Lance reader in its C++ BE via the Arrow C Data Interface, a single-threaded Tokio runtime, and Corrosion.

How Parquet’s Flatbuffer proposal, Lance, and Doris segment v3 each tackle columnar footer explosion when wide tables push metadata from KB into MB.

JSON stopped being a supporting format and became core business data. Why analytics engines need Variant types to keep up — and how Snowflake, Doris, and Iceberg are solving it.

Adding a secondary index to Apache Iceberg isn’t a file-layout problem — it’s an ecosystem problem. A walkthrough of the design tradeoffs the Iceberg community is navigating.