A Short Introduction to Portals
At its core, Portals unifies the Distributed Dataflow Streaming Model and the Actor Model, providing unparalleled flexibility and data-parallel processing capabilities with strong guarantees. The framework’s programming model is tailored for edge stateful serverless processing, and provides the following key features:
- Multi-Dataflow Applications. Multiple stateful dataflow streaming pipelines can dynamically be composed together on top of atomic streams, a transactional type of data streams.
- Inter-Dataflow Services. The Portal abstraction binds dataflow pipelines together to create and expose reusable services. This enables a request-reply type of communication between pipelines by providing serverless access to remote operator states on top of a continuation-style execution.
- Decentralized Cloud and Local Execution. The decentralized runtime can be executed on, and across, cloud and edge devices, whilst still providing end-to-end exactly-once processing guarantees.
- End-to-End Exactly-Once Processing: Portals provides end-to-end exactly-once processing guarantees for applications. This guarantee holds for arbitrary failures.
- Transactional Processing: Atoms are processed transactionally with ACID guarantees in a serializable schedule.
- Static Compile-Time Correctness: The Portals compiler statically checks the correctness of applications at compile-time for most common application bugs.
- Real-Time Data Processing: Events are processed at real-time low-latency speeds.
- Parallelism: Portals provides data-parallelism, pipeline-parallelism and task-parallelism for applications through stateful per-key execution semantics and pipelined execution.
- Elastic Scalability, Serverless Execution: Applications are automatically, elastically scaled up and down with the data demand, fully managed by the runtime.
- Dynamic Service Composition: Services can be composed together dynamically at runtime, forming dynamic connections between applications.
- Decentralized Service Composition: The Portals API and runtime support composing applications across decentralized runtimes.
- Cyclic Composition of Applications: Applications can be composed together as cyclic microservices.
- Dynamic Actor-Like Communication: The Portals service abstraction enables dynamic communication between applications, reminiscent of the actor model, integrated with futures.