Apache Fluss Quickstart: Real-Time Analytics with Flink

AI Guide

Apache Fluss just graduated from the Apache incubator — it's now a top-level project, approved unanimously by the board. If you're building agentic applications, this one is worth your time: agents need sub-second-fresh context to make decisions, and a classic Lakehouse (Paimon, Iceberg, Hudi) is great at storing history but terrible at "what happened two seconds ago." Fluss closes that gap with a lakehouse-native streaming storage layer.

In this guide you'll run the official quickstart end-to-end: spin up a Fluss + Flink cluster with Docker Compose, stream data in, enrich it with lookup joins, and run real-time analytics — all in Flink SQL. No Java required.

Why Fluss, in one paragraph

Fluss (German for "river") sits on top of your data lake, not instead of it. Hot, frequently-updated data lives in Fluss; long-tail history stays in Paimon/Iceberg/Hudi. Engines read a unified view via Union Read, so agents get context that's both fresh and complete. Under the hood it's columnar streaming on Apache Arrow, with server-side column pruning and predicate pushdown — engines only read the bytes they actually need.

The features that matter for agent workloads:

  • Primary-key tables — high-QPS point lookups, dedup, partial updates, delta joins
  • Changelogs — append-only history of state/decision changes, useful for auditing and reproducibility
  • Sub-second freshness — data is queryable as soon as it lands
  • Compute-storage separation — Fluss owns state and storage; Flink/Spark just compute

Step 1: Spin up the cluster

You need Docker and the Compose v2 plugin. Create a working directory with this docker-compose.yml (trimmed to the essentials):

services:
  # S3-compatible storage for tiered data
  rustfs:
    image: rustfs/rustfs:1.0.0-alpha.83
    ports: ["9000:9000", "9001:9001"]
    environment:
      - RUSTFS_ACCESS_KEY=rustfsadmin
      - RUSTFS_SECRET_KEY=rustfsadmin

  # Fluss cluster
  coordinator-server:
    image: apache/fluss:0.9.1-incubating
    command: coordinatorServer
    environment:
      - FLUSS_PROPERTIES=
          zookeeper.address: zookeeper:2181
          bind.listeners: FLUSS://coordinator-server:9123
          remote.data.dir: s3://fluss/remote-data
          s3.endpoint: http://rustfs:9000
          s3.access-key: rustfsadmin
          s3.secret-key: rustfsadmin

  tablet-server:
    image: apache/fluss:0.9.1-incubating
    command: tabletServer
    environment:
      - FLUSS_PROPERTIES=
          zookeeper.address: zookeeper:2181
          data.dir: /tmp/fluss/data
          remote.data.dir: s3://fluss/remote-data
          s3.endpoint: http://rustfs:9000
          s3.access-key: rustfsadmin
          s3.secret-key: rustfsadmin

  zookeeper:
    image: zookeeper:3.9.2

  # Flink cluster (bundles the Fluss connector + flink-faker)
  jobmanager:
    image: apache/fluss-quickstart-flink:1.20-0.9.1-incubating
    ports: ["8083:8081"]
    command: jobmanager

  taskmanager:
    image: apache/fluss-quickstart-flink:1.20-0.9.1-incubating
    command: taskmanager

  sql-client:
    image: apache/fluss-quickstart-flink:1.20-0.9.1-incubating
    command: /opt/sql-client/sql-client

Start everything and verify:

docker compose up -d
docker compose ps

Flink UI is at http://localhost:8083. The RustFS console (your S3 bucket, fluss) is at http://localhost:9001, credentials rustfsadmin/rustfsadmin.

Step 2: Create the Fluss catalog and tables

Enter the SQL client and point it at Fluss:

docker compose run sql-client
CREATE CATALOG fluss_catalog WITH (
  'type' = 'fluss',
  'bootstrap.servers' = 'coordinator-server:9123'
);
USE CATALOG fluss_catalog;

Create primary-key tables — these power the fast point lookups:

CREATE TABLE fluss_order (
  `order_key`   BIGINT,
  `cust_key`    INT NOT NULL,
  `total_price` DECIMAL(15, 2),
  `order_date`  DATE,
  `order_priority` STRING,
  `ptime` AS PROCTIME(),
  PRIMARY KEY (`order_key`) NOT ENFORCED
);

CREATE TABLE fluss_customer (
  `cust_key`  INT NOT NULL,
  `name`      STRING,
  `phone`     STRING,
  `nation_key` INT NOT NULL,
  `acctbal`   DECIMAL(15, 2),
  `mktsegment` STRING,
  PRIMARY KEY (`cust_key`) NOT ENFORCED
);

Step 3: Stream data in

The quickstart image pre-creates faker source tables that generate demo data. Sync them into Fluss:

EXECUTE STATEMENT SET
BEGIN
  INSERT INTO fluss_nation   SELECT * FROM `default_catalog`.`default_database`.source_nation;
  INSERT INTO fluss_customer SELECT * FROM `default_catalog`.`default_database`.source_customer;
  INSERT INTO fluss_order    SELECT * FROM `default_catalog`.`default_database`.source_order;
END;

Step 4: Enrich with lookup joins

This is where Fluss shines: joining a streaming order stream against PK tables for dimension lookup is a high-QPS operation, not a scan.

INSERT INTO enriched_orders
SELECT o.order_key, o.cust_key, o.total_price, o.order_date, o.order_priority,
       c.name, c.phone, c.acctbal, c.mktsegment, n.name
FROM fluss_order o
LEFT JOIN fluss_customer FOR SYSTEM_TIME AS OF `o`.`ptime` AS c ON o.cust_key = c.cust_key
LEFT JOIN fluss_nation   FOR SYSTEM_TIME AS OF `o`.`ptime` AS n ON c.nation_key = n.nation_key;

Step 5: Real-time analytics

SET 'sql-client.execution.result-mode' = 'tableau';
SET 'execution.runtime-mode' = 'batch';
SET 'table.dml-sync' = 'true';

SELECT * FROM enriched_orders LIMIT 2;

-- COUNT(*) is fast: Fluss maintains table-level stats, no full scan
SELECT COUNT(*) FROM enriched_orders;

-- point lookup by primary key
SELECT * FROM fluss_customer WHERE `cust_key` = 1;

Run COUNT(*) a few times — the number grows as faker keeps producing, because Fluss ingests continuously. Re-run it and the result updates in real time.

Step 6: Update and delete

UPDATE fluss_customer SET `name` = 'fluss_updated' WHERE `cust_key` = 1;
SELECT * FROM fluss_customer WHERE `cust_key` = 1;  -- name is now fluss_updated

DELETE FROM fluss_customer WHERE `cust_key` = 1;
SELECT * FROM fluss_customer WHERE `cust_key` = 1;  -- empty

Done. Clean up with quit then docker compose down -v.

Practice notes

  • Don't ship the demo credentials. The compose file uses rustfsadmin with AssumeRole STS for the local RustFS. In production, use cloud-specific credential provider chains.
  • Use it as an agent's real-time context store. Fluss maintains dedup, partial updates, and delta joins natively — the typical "real-time feature store" pattern maps directly onto a PK table.
  • Changelogs are free decision logs. For agent systems you want an audit trail of state changes; Fluss generates append-only changelogs out of the box.
  • Ecosystem today: Flink + Spark connectors are available, StarRocks is coming. Alibaba, Xiaohongshu, JD, and Ant run it in production on billion-scale traffic.

Resources

Scroll to Top