A dull, resilient stream processor in Go

Is "dull" a technical term or does it literally just mean boring here?

Hey everyone, author here. We use Benthos as a general stream swiss army knife for all the dull tasks, but you can also use it as a framework for writing your own stream processors in Go.

It doesn't provide any tools for idempotent calculations yet, just at-least-once delivery guarantees. But when you use it as a framework you get the benefit of all the configurable processors in your new service.

Interesting! We use NSQ at work to collect logs/messages from various systems and forward them to various endpoints (Kafka, ELK, archives, Prometheus). This looks like a more advanced/flexible system, where as we just use a handful of single purpose daemons writen in go (nsq2kafka, nsq2es, nsqarchive, nsqstream-metrics).

I gave a quick intro to nsq talk at our local go meetup recently. If you haven't used NSQ before I highly recommend giving it a try.

https://docs.google.com/presentation/d/1e9yIm-0aNba_H1gX_u7D...

Talk about timing!

I'm in the process of rejigging our live telemetry pipeline (dealing with petabytes) to improve flexibility and resiliency and have been looking for something to replace our similar systems that have grown organically. This fits the bill exactly; it's designed exactly how I imagined our new system should look.

I'm wondering if HN has their personal favorite for these sorts of problems.

Supported Sources & Sinks: Amazon (S3, SQS), Elasticsearch (output only), File, HTTP(S), Kafka, MQTT, Nanomsg, NATS, NATS Streaming, NSQ, RabbitMQ (AMQP 0.91), Redis, Stdin/Stdout, Websocket, ZMQ4.

Looks like this could be useful for webhooks delivery, a problem our team is working on at the moment.

We have 100's of tenants, each of which could get their own stream with some delivery guarantees set. If one tenant's endpoint is down or another tenant fills the stream with 1M messages, it should not affect delivery rates of other tenants. Seems to fit the bill.

I see the HTTP output mentions some retries, but I guess these run as part of the delivery step, blocking this goroutine as opposed to rescheduling the message? Sometimes it takes hours for clients to restore their receiving systems and it would be great if messages for past N hours would still be delivered..

I must be missing something. It doesn't look dull.

This seems pretty similar to Wallaroo but with more focus on connecting to everything. And of course golang vs Pony

Super exciting, I can imagine using this instead of Kafka for more simple use cases. Wonder if they will add a full logging system so we can dump Kafka completely

This seems quite a bit like Heka. Very excited.