Akka Streams is Legos for stream processing

Use it to glue streams of messages together:

• Streams of Kafka messages
• Streams of messages over Websockets
• High-level network IO: Streams of HTTP Requests
• Low-level network IO: streams of bytes over TCP
• Files (local or on S3) as streams of bytes
• Input and output streams of processes

Goal

Server that consumes events via many websocket connections and publishes them to a single Kafka topic.

Events

case class Event(msg: String, clientId: String, timestamp: Long)

(serialization/deserialization boilerplate omitted)

Implicit Boilerplate

Provides a context for running actors

  implicit val system: ActorSystem = ActorSystem()

Provides a thread pool for executing callbacks

  implicit val ec: ExecutionContext = system.dispatcher

Provides a materializer, backed by the ActorSystem system, for materializing stream processing graph blueprints

  implicit val materializer: Materializer = ActorMaterializer()

Kafka Service

case class KafkaServiceConf(bootstrapServers: String)

class KafkaService(kafkaClient: ReactiveKafka, conf: KafkaServiceConf) {
  // publish a stream of json-serializable messages to a kafka topic
  def publish[T](topic: String)(implicit writes: Writes[T], actorSystem: ActorSystem): Sink[T, Unit] =
    Flow[T].map(e => ProducerMessage(e)).to(
      Sink.fromSubscriber(
        kafkaClient.publish(
          ProducerProperties(
            bootstrapServers = conf.bootstrapServers, //IP and port of local Kafka instance
            topic = topic, // topic to publish message to
            valueSerializer = KafkaService.serializer[T]
          )
        )))

  // consume messages from a kafka topic. messages must be deserializable from json
  def consume[T](topic: String, groupId: String)(implicit writes: Reads[T], actorSystem: ActorSystem): Source[T, Unit] =
    Source.fromPublisher(kafkaClient.consume(
      ConsumerProperties(
        bootstrapServers = conf.bootstrapServers, // IP and port of local Kafka instance
        topic = topic, // topic to consume messages from
        groupId = groupId, // consumer group
        valueDeserializer = KafkaService.deserializer[T]
      )
    )).map(_.value())
}

Using KafkaService

val publishEvents: Sink[Event, Unit] = kafkaService.publish[Event](eventTopic)

    Event      +----------+
 ------------->|   Kafka  |
               +----------+

val consumeEvents: Source[Event, Unit] = kafkaService.consume[Event](eventTopic, group)

+-------+  Event   
| Kafka +--------->
+-------+

Bridging streams with SourceQueue

trait SourceQueue[T] {
  def offer(elem: T): Future[Boolean]
}

Connecting a SourceQueue[Event] to the Kafka Sink

  val kafkaPublisherGraph: RunnableGraph[SourceQueue[Event]] =
    Source.queue[Event](1024, OverflowStrategy.backpressure).to(kafka.publish[Event](eventTopic))

  val sourceQueue: SourceQueue[Event] = kafkaPublisherGraph.run

Stream processing graph

+--------------------+   Event   +---------+
| SourceQueue[Event] +----------->  Kafka  |
+--------------------+           +---------+

Creating a Sink to write to the SourceQueue

  val queueWriter: Sink[Event, Unit] =
    Flow[Event].mapAsync(1){ elem =>
      sourceQueue.offer(elem)
        .andThen{
          case Success(false) => println(s"failed to publish $elem to topic $eventTopic")
        }
    }.to(Sink.ignore)

Stream processing graph

            +--------------------+
  Event     |                    |
+-----------> SourceQueue[Event] |
            |                    |
            +--------------------+

Parse incoming websocket messages

  val parseMessages: Flow[Message, Event, Unit] =
    Flow[Message].collect{
      case TextMessage.Strict(t) =>
        val js = Json.parse(t)
        Json.fromJson[Event](js).get
      //ignore binary or streaming messages
    }

Stream processing graph

 Message   +---------------+   Event
+----------> parseMessages +----------->
           +---------------+

Defining a Flow to handle each websocket connection

  val wsHandlerFlow: Flow[Message, Message, Unit] =
    Flow.fromSinkAndSource(
      sink = parseMessages.to(queueWriter),
      source = Source.maybe
    )

Stream processing graph

                               Flow.fromSinkAndSource
        +--------------------------------------------------+-----------------+
        |                                                  |                 |
Message | +----------------+ Event +--------------------+  | +-------------+ | Message
+---------> parseMessages  +-------> SourceQueue[Event] |  | |Source.maybe +----------->
        | +----------------+       +--------------------+  | +-------------+ |
        |                                                  |                 |
        +--------------------------------------------------+-----------------+

Defining a Flow to handle incoming HTTP Requests

  val routes: Flow[HttpRequest, HttpResponse, Unit] =
      get {
        path(PathEnd) {
          getFromResource("test.html")
        } ~
          path("ws") {
            println("ws connection accepted")
            handleWebsocketMessages(wsHandlerFlow)
          }
      }

  Http().bindAndHandle(routes, "localhost", port)

Stream processing graph

HttpRequest +--------+ HttpResponse
   +--------> routes +-------->
            +--------+

Simple frontend page: sends 1 event per second

<!DOCTYPE html><html><body><script type="text/javascript">
    var clientId = Math.random();
    var webSocket = new WebSocket("ws://localhost:9000/ws");

    webSocket.onopen = function(event){
        console.log(event);
    };

    webSocket.onmessage = function(event){
        console.log(event);
    };

    webSocket.onclose = function(event){
        console.log("Connection closed");
    };


    window.setInterval(function(){
        var data = JSON.stringify({'msg': "test msg",  'clientId': "" + clientId, 'timestamp': Date.now()});
        console.log("send " + data);
        webSocket.send(data);
    }, 1000);
</script></body></html>

Consuming and logging messages from Kafka

object KafkaListener extends App with AppContext {
  val graph = kafka.consume[Event](eventTopic, "kafka_listener").to(Sink.foreach(println))

  graph.run

  awaitTermination()
}

Stream processing graph

+-------+  Event      +-----------------------+
| Kafka +-------------+ Sink.foreach(println) |
+-------+             +-----------------------+

Load Testing using the Akka HTTP websocket clients

val wsClient: Flow[Message, Message, Future[WebsocketUpgradeResponse]] = 
  Http().websocketClientFlow(WebsocketRequest(Uri(s"ws://localhost:$port/ws")))

(full code omitted, check github.com/pkinsky/ws_to_kafka for full load test)