Skip to content

Platform· Case study 09

Incremental ingestion for Rill’s ClickHouse driver

Designed the incremental ingestion strategy for the ClickHouse driver in Rill, an open-source real-time BI platform, which became the foundation for ClickHouse data modeling in Rill.

Open source

shipped in rilldata/rill

Architecture at a glance

  1. Partition-aware loads
  2. Idempotent models
  3. Rill ClickHouse driver

01 The problem

ClickHouse models in Rill needed to refresh incrementally at scale without double-counting when loads were re-run or backfilled.

02 What I built

  1. 1Incremental, partition-aware loads for ClickHouse models in Rill.
  2. 2Idempotent semantics so re-runs and backfills never double-count.
  3. 3A design that became the foundation for ClickHouse data modeling in Rill.

03 Impact

ClickHouse models in Rill refresh incrementally and safely, and the approach is available to every Rill user through the open-source project.
rilldata/rill on GitHub

Related case studies

08 · Contact

Let’s build something that holds up in production.

If something here resonates, whether it’s a data problem, an idea, or a conversation worth having, I’d love to hear from you. The fastest way to reach me is a message on LinkedIn.

Message me on LinkedIn Résumé (PDF)

Good reasons to reach out

  • Data platform help

    ClickHouse or Druid performance, Kubernetes operations, migrations, and cost reviews.

  • Collaboration

    Open-source work, writing, talks, or comparing notes on a hard data problem.

  • Just to talk shop

    Real-time analytics, AdTech data, MCP and agent tooling, or life as an FDE.

esc
↑ ↓ navigate↵ select