Apache Fluss: unified streaming storage

Apache Fluss (incubating in Apache Foundation, 2024-2025) — это новый streaming storage system, спроектированный как columnar Kafka для analytics. Если Kafka — universal log (row-oriented, app-agnostic), то Fluss — narrow analytical streaming (column-oriented, query-aware).

Этот урок: что такое Fluss, чем он отличается от Kafka, как column format даёт server-side projection и filter, и в каких case-ах Fluss заменяет Kafka + OLAP stack.

Проблема: гибрид Kafka + OLAP

Classical real-time analytics stack:

1. Kafka — ingestion, durable log.
2. Flink streaming job — consumes Kafka, transforms.
3. OLAP store (ClickHouse / Druid / Pinot) — fast aggregation queries.
4. Dashboard / API — queries OLAP.

Boundaries несколько:

Issue 1: Two storage layers. Kafka хранит raw events (TBs), OLAP — aggregated/materialized (TBs). Дублирование.

Issue 2: Latency between layers. Kafka -> Flink -> OLAP = pipeline latency 100ms-1s. Query OLAP — query OLAP latency. Total time от event arrival до query result.

Issue 3: Operational complexity. Maintain Kafka cluster, Flink jobs, OLAP cluster. 3 different ops domains.

Issue 4: Storage cost. Kafka retention 7 days = TBs. OLAP retention 90 days = TBs. Same data в both, different formats.

Fluss approach: columnar streaming

Fluss заявляет: для analytical workloads нам не нужны 2 storage layers. Если streaming storage уже columnar и query-aware, мы можем:

• Read с filter / projection pushdown (как Parquet, но streaming).
• Subscribe на stream (как Kafka).
• Materialize в lakehouse для long-term storage (как через connector).

Fluss combines streaming semantics (Kafka-like subscribe, log retention) с analytical capabilities (columnar storage, predicate pushdown, native lakehouse integration).

Stripped middleware. Producers write to Fluss directly. Consumers read Fluss directly — но с column projection и filter, не bulk records.

Columnar storage в streaming

Kafka stores records as binary blobs:

[batch 1: record(id=1, name=foo, price=10), record(id=2, name=bar, price=20)]
[batch 2: record(id=3, name=baz, price=30), ...]

Consumer reads byte stream, parses каждый record целиком (deserialize all columns even if нужна одна).

Fluss stores records as columnar blocks:

[column-block id:    [1, 2, 3, ...]]
[column-block name:  ['foo', 'bar', 'baz', ...]]
[column-block price: [10, 20, 30, ...]]

Consumer может request only columns it needs:

SELECT id, price FROM fluss_topic;
-- читает только column-blocks id и price, не name

Это reduce network IO especially для wide rows. Если у тебя topic с 100 columns и query нужны 3 — Fluss reads 3% data vs Kafka 100%.

Server-side projection и filter

Beyond just column read, Fluss supports predicate pushdown:

SELECT id, price FROM fluss_topic
WHERE price > 100;

Fluss server (broker аналог Kafka) evaluates price > 100 на server-side (before sending to client). Только matching rows ship. Network savings.

Comparable to Kafka:

• Kafka: ship all records to consumer, consumer filters.
• Fluss: ship only matching records.

Effective для selective filters (1% records match) — 100x network reduction. Less effective для unselective (90% match) — почти same.

Pushdown limitations:

• Simple predicates (column op value) — supported.
• Complex expressions (function calls) — usually не supported, client-side filter.
• Joins — не supported (Kafka тоже не supports, естественно — нет другой data на server).

Subscribe model

Fluss supports Kafka-like subscribe:

FlussConsumer<RowData> consumer = FlussConsumer.builder()
    .bootstrapServers("fluss:9092")
    .topic("orders")
    .columns(List.of("id", "amount"))  // projection
    .filter("amount > 100")            // predicate pushdown
    .startingOffsets(OffsetsInitializer.committedOffsets())
    .build();

while (true) {
    List<RowData> batch = consumer.poll(Duration.ofMillis(100));
    for (RowData row : batch) {
        process(row);
    }
}

Subscribe semantics: consumer follows tail of stream, gets new records as they arrive. Like Kafka subscribe. Но only requested columns, only matching rows.

Flink integration:

CREATE TABLE orders WITH (
  'connector' = 'fluss',
  'bootstrap.servers' = 'fluss:9092',
  'topic' = 'orders'
);

-- Streaming read с pushdown
INSERT INTO sink
SELECT id, amount
FROM orders
WHERE amount > 100;
-- Flink Fluss connector applies pushdown automatically

Native lakehouse integration

Fluss + Paimon — designed-together combo. Pattern:

• Fluss — hot streaming layer. Retention 1-7 days. Sub-second consume.
• Paimon — cold lakehouse layer. Retention years. Batch + streaming queries.

Fluss can automatically materialize to Paimon:

CREATE TABLE orders WITH (
  'connector' = 'fluss',
  'bootstrap.servers' = 'fluss:9092',
  'topic' = 'orders',
  'paimon.warehouse' = 's3://bucket/paimon/',
  'paimon.materialize' = 'true',
  'paimon.materialize.interval' = '1 minute'
);

Fluss server tier-down old data to Paimon. Queries get unified view: live in Fluss + historical in Paimon. Without manual ETL.

Это похоже на tiered storage в новых Kafka versions (cold storage в S3), но более integrated:

• Kafka tiered storage: still row-oriented, optimal для full-scan.
• Fluss + Paimon: columnar throughout, optimal для analytics.

Architecture: tablet / chunk

Fluss internally organized in tablets (как Kafka partitions, но more flexible). Каждая table split в N tablets.

Tablet = ordered sequence of chunks (columnar blocks of records):

Table "orders"
  Tablet 0 (partition by hash(id) mod N)
    Chunk 0 (records 0-9999)
      column id:    sorted array
      column name:  sorted by id array
      column price: sorted by id array
      footer:       min, max, count per column
    Chunk 1 (records 10000-19999)
    ...
  Tablet 1
    ...

Chunks immutable after write (like parquet row groups). Tablet append-only (new chunks added).

Reading:

1. Reader subscribes to tablet (one or more).
2. Server reads chunk footer first (small): min/max per column.
3. If filter matches (price > 100 might match this chunk if max > 100), reads only requested columns from chunk.
4. Ships filtered records to reader.

Это classic columnar query optimization (Parquet, ORC, ClickHouse), но в streaming context — chunks created on-the-fly как records arrive.

When Fluss vs Kafka

Decision matrix:

Не “Fluss replaces Kafka”. Это complementary. Многие orgs будут иметь оба: Kafka для general messaging, Fluss для analytics layer.

Performance characteristics

Approximate (workload-dependent):

Trade-offs: Fluss heavier на server CPU (columnar encoding) и server logic (filter eval). Kafka simpler — just write/read bytes. Fluss wins when query patterns are selective (filter + projection), Kafka wins для full-record streaming.

Production-перспектива: adoption considerations

Fluss is incubating в Apache (как of 2025). Это значит:

Pros:

• Active development by Alibaba (creator) и community.
• Open source, Apache governance.
• Designed для modern streaming + lakehouse stacks.

Cons / risks:

• Younger than Kafka (years vs decade).
• Smaller ecosystem (SDKs, monitoring tools, third-party connectors).
• Less battle-tested at extreme scale.
• Operational expertise rare — fewer ops engineers know Fluss.

When to adopt Fluss now (2025-2026):

• Greenfield real-time analytics platform.
• Already deep в Flink + Paimon stack.
• Have Alibaba/Ververica support (commercial).

When to wait:

• Critical production workloads needing high SLA, 24/7 ops.
• Existing Kafka org с big investment in Kafka tooling.
• Need wide language SDK support (Fluss SDKs may be limited).

Migration paths

If wanting to evaluate Fluss without big-bang switch:

Pattern 1: Dual-write.

Flink job mirror writes to Kafka + Fluss. Analytical consumers point to Fluss, app consumers — Kafka. Compare performance. Eventually wean off Kafka в analytical domain.

Pattern 2: Layered.

Kafka stays for ingestion (apps write Kafka). Flink job reads Kafka, writes to Fluss (with column structure, schema enforcement). Analytical queries read Fluss. Best of both — Kafka simple ingestion, Fluss analytical optimization.

Pattern 3: Tiered.

Kafka holds last 1 hour (fast retention). Flink tier данные older в Fluss. Recent queries — Kafka. Historical — Fluss. Combines latency (Kafka recent) с analytics (Fluss historical).

Comparison: Kafka tiered storage

Modern Kafka (3.6+) has tiered storage — Kafka cluster уверенно offloads old segments в S3. Это решает retention cost issue: keep weeks/months в Kafka without huge local storage.

Fluss + Paimon delivers similar but architecturally cleaner:

Если ты “просто” replacing Kafka local storage с cheap object store — Kafka tiered storage simpler. Если ты wanting analytical-first streaming + automatic lakehouse — Fluss.