Draft:KSML (Apache Kafka)

KSML is an expression language and accompanying interpreter that allows users to specify Kafka Streams topologies as YAML. It was open sources by Axual in May 2021 under Apache License 2 with code published on GitHub.

The idea
The general idea of KSML is to express stream processing operations as a series of operations applied to a Kafka topic. The simplest example of a copying application is expressed as follows:

pipelines: main: from: some_topic to: some_other_topic

This example defines a topology that copies messages from a given source topic to a target topic. Custom functions can be added to the YAML specification as Python functions. For more information, see the KSML documentation.

Implementation
Under the hood, KSML delivers an interpreter that reads in the YAML definitions and maps them onto a Kafka Streams topology. This topology is then passed onto the Kafka Streams framework itself to be executed continuously.

Distribution
KSML is distributed as source code and as Docker images for plain Kafka and Axual Platform / Cloud.