| Concept | Description | |---------|-------------| | | Hierarchical (Namespace → Topic → Partition). | | Retention | Configurable time‑based (e.g., 7 days) or size‑based (e.g., 1 TB). | | Exactly‑once | Transactional writes via two‑phase commit between LogStore and StreamEngine. | | Back‑pressure | gRPC flow‑control + token‑bucket per consumer group. | | Replay | Log‑based replay using offsets stored in MetaStore ; supports snapshot‑based re‑processing (e.g., for ML feature pipelines). |
| Sub‑System | Role | Main Technologies (examples) | Critical Guarantees | |------------|------|------------------------------|----------------------| | | Global namespace, object versioning, lease management. | etcd , Consul , Raft‑based KV . | Strong consistency (linearizable reads/writes). | | LogStore | Immutable write‑ahead log (WAL) + LSM compaction. | RocksDB , Pebble , LevelDB forks. | Write durability (≥ 2‑f + 1 replication). | | StreamEngine | Real‑time event ingestion & replay (pub/sub). | Apache Pulsar , Kafka‑compatible APIs , gRPC streams . | Exactly‑once delivery (transactional writes). | | Cache Layer | Hot‑data tier (in‑memory & NVMe‑tier). | Redis‑cluster , Memcached , Aerospike . | Sub‑ms read latency, automatic promotion/demotion. | | Storage Nodes | Persistent data (object + block). | Hybrid SSD/HDD , Erasure‑coded erasures (Reed‑Solomon) , NVMe‑over‑Fabric . | 99.999% durability, configurable R/W parity. | | Global Router | Multi‑region request routing + geo‑affinity. | Anycast DNS , BGP‑based Anycast , SD‑WAN . | < 2 ms cross‑region routing, fail‑over within 30 s. | | API Gateways | Unified REST, gRPC, S3‑compatible, Kafka‑compatible endpoints. | Envoy , Traefik , NGINX , Istio . | Auth / AuthZ (OAuth2, mTLS). | FSDSS-536
★★★★☆ (4 / 5) – approve for merge after the minor items are resolved. | Concept | Description | |---------|-------------| | |
In the meantime, the following structure gives you a that you can copy‑paste, edit, and flesh out. Every major heading includes bullet‑point prompts and suggested content so that you can fill in the specifics quickly. | | Back‑pressure | gRPC flow‑control + token‑bucket