Announcing Quix’s major product release

by Tomas Neubauer
| 2 Dec, 2021

Accelerate time to market for live data-driven products

In Formula One, teams spend Monday through Friday fine-tuning a car’s performance. Then the weekend comes, and they can finally test it on the track.

Does it work in the real world? Do these improvements deliver results?

If projects are built in a dev environment, it’s entirely possible that code won’t work. That’s because a fabricated test environment can’t replicate the messiness of the real world.

Let me give you an example, drawn from the experience my co-founders and I had while working at McLaren. If you work with sample testing data, you’re essentially testing code against a best-case scenario. In the real world, live data has faults, time delays, and it can come in the wrong order.

If you haven’t practiced on real-world data, the results are chaos.

That’s why testing any code against real, live data is essential. It gives you the confidence that when you show up for race day, your code will work.

That’s also the thinking the Quix team brought to the problem of developing with live data in any industry. How can we help developers write and test code on a real-world data set, without breaking the production environment that companies hold so dear?

In this article, we’ll cover the new features of Quix, and the thinking behind them:

 

Build, test and deploy from an online IDE

It’s widely understood that working with a streaming data message broker such as Kafka is hard. As part of our effort to bring this complex technology to all developers and data scientists, today we’re announcing a major advancement in the Quix platform: an integrated development environment (IDE), complete with connections to live data sources.

This solves many problems a user has when writing code that interacts with Kafka. For example, developers must:

  • Configure code to send and receive messages with Kafka
  • Install client libraries on their local machines, which requires time to install and debug
  • Set up and configure security and encryption between Kafka and the local machine.

You have to go through all of these steps to even start writing code. Then, when you get to coding, you’re working in a distributed architecture that requires additional integration testing. When you find an error, you must pull back code into your local IDE to find and solve the problem.

From the beginning, Quix has been on a mission to democratize live data by reducing the complexity and hassle associated with stream processing. Our latest release now takes care of configuration, and the online IDE is ready to run the steps above handled for you. This enables you to seamlessly develop, test and run code, without having to deploy it to your company’s production environment.

The new Quix online IDE

 

Why streaming data isn’t more accessible

Message brokers are gaining rapid adoption in companies across every industry, thanks in large part to the work done by Confluent, which has accelerated the adoption of Kafka. It’s emerging as an essential component of the modern data stack, so more people need more access.

But IT creates firewalls around this data, in part because streaming data is hard to handle. Some of the world’s largest organizations, like Audi and BMW, use Kafka in their enterprise IT architecture, but their automotive engineers don’t have access to streaming data resources.

Quix is doing for the Kafka message broker what DBT did for the data warehouse. The Quix platform creates a real-time mirror of data streams to provide a safe, bolt-on environment that anyone with Python skills can use. It doesn’t require users to have their own CI/CD infrastructure, IDE, Git or Kafka. It’s a very simple interface into that complex world.

“Quix is doing for the Kafka message broker what DBT did for the data warehouse.”

This enables people with technical skills in product lines and business units to access live data, build applications, and automate actions.

And it’s not just data scientists and data engineers who benefit. Applications in mobility solutions and audio processing enable people with skills in mechanical, electrical and electronic engineering. Even chemists and biologists who’ve never heard of Kafka have used Quix to access streaming data technology.

 

Why CI/CD development takes so long

The traditional CI/CD paradigm for application development requires a long, iterative process:

  1. Develop code locally in your IDE, build and run unit tests, iterate
  2. Connect your code to fake data, run an integration test, iterate
  3. Deploy your code to a dev environment, test with real people, iterate
  4. Deploy code to a staging environment, run automated tests, iterate
  5. Release code to production, run environment and acceptance tests, deploy

This paradigm is more difficult for developers writing microservices on data streaming infrastructure — such as Kafka — because of the decentralized nature of their applications.

It’s even harder for developers of real-time data products. In the current approach, developers have to ask IT for access to production systems. A data engineer has to prepare a table of data.The data scientist can then use Jupyter Notebooks to explore and develop a model, but they cannot test it.

So next, they have to ask DevOps to create a test topic, build and deploy features, and deploy their model artifact as a microservice. After a few lengthy iterations, and some tough meetings with legal and compliance, the team can finally release their product to production.

Each step requires significant cross-functional collaboration, enablement and support.

The difficult path to real-time development

 

Quix’s new paradigm safely connects users to the production broker, where they can self-serve all the resources they need. With one connection, a data scientist can explore data in Jupyter, then build, test and deploy features and models, all without any support from external teams. When legal and compliance are happy with the product, it can be hooked up to production in seconds with our API and SDK.

Build faster on streaming data with Quix

 

Our new online IDE is enabling access to Kafka for new types of developers — let’s see how.

 

How the Quix IDE works

The Quix online IDE connects your code directly to live data input topics and runs it against live data or an authentic replay of a historic live stream.

You can compare output variables against input variables, and check that everything performs as intended on real data — with gaps, bursts, data streamed out of order, and other common faults. Once you’re happy with your code, deploy it with a click of a button.

Rapid development with the Quix IDE

 

We’ve found through our Formula One experience, and through the experiences of our customers, that building with real data from day one produces a higher quality product, faster. It eliminates the risk and hassle of CI/CD development cycles to release new applications to production faster.

 

Prebuilt connectors to live data streams

We’re particularly excited about Quix’s new connectors — a collection of live data sources provided out of the box in Quix. This includes financial currency markets, Twitter, Amazon Kinesis, Google pub/sub, Confluent Kafka, and historical data sets that you can use to train your model on real-world data.

Quix connectors

 

You can access these connectors directly from our Topics view:

Add a data source connector from the Topics view

 

Then, with just a few settings, add that data source to your project.

Simple data source configuration

 

Visualize and query data

We’ve also introduced an improved data explorer to help you visualize your data in waveform and table format. This layout should feel more familiar to people accustomed to SQL queries. There is a code generator to help developers load data into external systems like Jupyter Notebooks.

View and query data streams, parameters and events in waveform view

 

See the same query in tabular format

 

Copy and paste the query to load data into external systems like Jupyter Notebooks

 

Live preview for real-time data

We introduced a new feature, live preview, which offers a real-time view of your data using a WebSocket protocol. This allows developers to observe data flowing through the platform without delay, and without requiring you to enable data persistence.

 

Improved queries, filtering and speed

Under the hood, Quix is faster on all counts, including dramatically improved startup times for multiple deployments, and deployments spend less time in build queues.

Our Telemetry Query was also improved with these capabilities:

  • Filter streams by the events they contain, not just a single parameter such as “speed”
  • Filter data by tag
  • Query parameter data and event data in the same call
  • Create queries across various streams without first having to select the stream.

 

Getting started is easier than ever

For new Quix users, social sign on is now available when you sign up to try Quix, so it’s one-click easy to access your workspace. (Goodbye, captcha.) We also launched further enhancements to our docs and guides including a new mobility tutorial to predict real-time fleet availability based on ever-changing demand, weather, traffic and other influences.

We’ve wrapped up all of these new features with developer tools including a status page and a changelog to give you the details on each major and minor release.

If all of these new features get you excited to try Quix, sign up for a free Quix account, complete with free credits to launch your first project.

by Tomas Neubauer

Tomas Neubauer is cofounder and CTO at Quix, responsible for the direction of the company across the full technical stack, and working as a technical authority for the engineering team. He was previously technical lead at McLaren, where he led architecture uplift for Formula One racing real-time telemetry acquisition. He later led platform development outside motorsport, reusing the knowhow he gained from racing.

Related content