I guess we're working our way through the Greek alphabet with these architectures. Here it goes:
λ Lambda architecture
The Lambda Architecture is a mature paradigm in big data computing. The name comes from a functional point of view of data processing — all data processing is understood as the application of a function to all data.
Lambda is popular for handling huge amounts of data by taking advantage of both a batch layer and a speed (stream-processing) layer. It balances latency, throughput, and fault tolerance: batch processing provides comprehensive, accurate views of batch data, while real-time stream processing provides views of online data. The outputs from both layers merge before presentation.
The efficiency shows up as increased throughput, reduced latency, and negligible errors — a linearly scalable architecture that scales out rather than scaling up.
Basic tenets
Lambda achieves high scalability and low latency through three principles:
→ Immutability of data
→ Data denormalization
→ Precomputed views
The three layers
Lambda solves the problem of computing arbitrary functions on arbitrary data in real time by decomposing it into:
1. Batch layer — the cold path
2. Speed layer — the hot path
3. Serving layer
κ Kappa architecture
Kappa is a simplification of Lambda — a Lambda system with the batch processing system removed, which avoids duplicating logic. Rather than a relational DB or a key-value store, the canonical data store in a Kappa system is an append-only immutable log. Data streams from the log through a computational system into auxiliary stores for serving. To replace batch processing, data is simply fed through the streaming system quickly.
LinkedIn and Yahoo are some of the users of this architecture.
ζ Zeta architecture
The next-generation enterprise architecture cultivated by Jim Scott — a pluggable architecture with seven components working with a global resource manager:
→ Distributed file system
→ Real-time data storage
→ Pluggable compute model / execution engine
→ Deployment / container management system
→ Solution architecture
→ Enterprise applications
→ Dynamic and global resource management
Zeta represents the number 7 — seven components in total. It gives you dynamic compute resources, a common storage platform, real-time application support, flexible programming models, deployment management, and a solution-based approach to operating a business: reduced opex and capex, improved time to market, and a solid scalable platform.
Next up: IoT-a architecture.
Originally published on LinkedIn. Republished here from the archive.