There’s no instruction manual for stream processing. With established technology, there are volumes of information about how to do something better — books, blogs, user groups, reference architectures. But with an emerging technology as complex and varied as data stream processing, it’s a brave new world.
That’s why Quix is sponsoring the first community dedicated to figuring this all out. We’re calling it The Stream. Whether you’ve been processing data on a message broker like Kafka for a while, or you’re simply stream-curious, you’re welcome. You’ll find friends and helpful allies here.
Our community connects on Slack and in person at meetups. Our first two live events are in Berlin this month. You should be a part of the community if you:
Work with data — whether as a software developer, data scientist, or engineer of another stripe. We’re having great conversations with mechanical and electrical engineers who focus on building IoT devices and need the data stream processing backbone to make them go.
Write code in Python — while many data-oriented technologies have centered on SQL, and streaming data has roots in Java and Scala for software engineering, we’re focusing on Python because it’s easy to use (making streaming accessible to newcomers), extensive (there’s a huge ecosystem of packages, like Pandas, that do much of the work for you) and incredibly flexible (making streaming useful to anyone from any industry).
Are interested in real-time data — you recognize the power and potential of streaming data, migrating workloads from batch to stream processing, or realizing the benefits of it such as lower latency, more efficient use of compute resources, and real-time automation or ML.
I believe passionately in streaming data. It will be the defining technology of this era. The 2010s focus on big data is giving way to faster architectures that enable you to build new product categories.
It’s early days, and innovation can be complicated. I believe this community will be coming together to better understand not only how to use stream processing, but also when not to. We’re not about technology for technology’s sake. We’re about finding new and better ways to handle the massive volume and velocity of data that’s streaming in from IoT devices and digital experiences every day.
Our community’s goals are learning, supporting each other, and helping to develop connected devices and data-driven products faster, easier and more efficiently.
You might use Quix for that. You might not. Either way, you’re welcome and we hope to make the community a valuable forum and connection for you.
Meet The Stream in person
Can better data help policy makers and communities build resilience?
A glossary for stream processing
- Data Council offers a “100% no bullsh*t guarantee” event about data science, engineering and analytics. You’ll want to join for the easy networking and the abundance of speakers across six learning tracks. There’s something for everyone here.
- The European Union has proposed an addition to its digital rulebook. Read about it on TechCrunch.
- Niyi Odumosu, Associate Solutions Architect at Confluent, uses the Metrics API, Docker, Prometheus, Grafana, Splunk and Datadog to put together a full monitoring solution for your Confluent Cloud deployment.
- To settle some confusion about the terms real time and stream processing, check out How stream processing goes beyond real-time.