Real time data streaming enhancements from DiffusionData

DiffusionData announced the release of Diffusion 6.10. The latest developer-centric enhancements to the framework aim to free up resources, speed and simplify development, and reduce operational costs.

“We spend a lot of time speaking to developers and architects about how we can improve our framework, what they need that we’re not providing and what can we do better. This latest release is mainly based on the feedback we’ve received. The enhancements address architectural and technical challenges many of our customers face on a daily basis. We are very grateful for the community and client input we get which helps us to further strengthen our market position as a global leader in real-time data streaming,” said Riaz Mohammed, CTO at DiffusionData.

Enhancements in Diffusion 6.10 include: 

Floating Point Support in Topic Views

Many businesses use decimal or floating point numbers in their data, Diffusion’s Topic Views feature now supports floating point numbers in all clauses. This particularly applies to the ‘process’ transformation which now has support for setting floating point numbers and using them in calculations. This includes the ability to specify the scale (the number of places after the decimal point) and the rounding policy to use in calculations.

Conditional update and OR operator constraints

Using Diffusion it is possible to specify constraints when updating a topic, either directly or using an update stream. One of these constraints is checking the current value of the topic before applying the update. Until now, the only constraint possible on a value was an exact binary equality comparison with the current value. Following a client request, you can now update topics from many different sources and still maintain the order.

6.10 has been extended to support relational comparisons, such as ‘greater than’, ‘less than’ etc. The values specified do not have to exactly match the type. For example, you can specify that the update is only applied if the current value is less than five and this would work with integral or floating point values. It would also work against a string value containing a number.

Previously it was possible to compose update constraints using an and method so that the update would only be applied if a lock was held and the value was equal to a specified value. In 6.10, you can also use an or method to compose constraints along with and so that an update is only applied if there is no topic, the topic has no value or the value is less than five.

Ability to retrieve metrics using the API

Metrics provide valuable insights into how the system is performing and allow users to tune their installation accordingly. Previously, metrics have been available through Prometheus or viewable via the Diffusion Console. In 6.10 you can also retrieve metrics via the API.

This is achieved using a new metricsRequest method in the metrics feature which returns an object which you can configure and then issue a fetch request to return the required metrics. By default, it will return all metrics for all servers in a cluster, but it is possible to request only metrics for the currently connected server or for specified servers. Users can also filter metrics so that the result only contains the metrics they are interested in. Providing metrics access in the API allows users to build metric monitoring into their own applications.

Update stream recovery

Clustered server architectures are ideal for availability but present issues when the shape of the cluster changes (such as adding a new server). This feature allows applications to seamlessly recover from blips in the cluster. Previously, they would have to code applications to be resilient to these issues. In essence, less code for the user to write and maintain.

If a Diffusion cluster repartitioned whilst clients were updating, a transient error would occur and the update stream would be invalidated. The updating application would need to be coded to recover from such an eventuality. In 6.10, it is possible to specify a retry strategy, and a new RecoverableUpdateStream is created. This stream keeps in-flight updates and is able to reapply them when a transient cluster error occurs, reducing the complexity of the calling application.

REST API to manage gateway framework

Gateway Framework allows simple integration with countless external systems. The previous gateway applications were managed in real time using the Diffusion console. The addition of REST API management means that such applications can be managed programmatically from within the user’s own applications. A REST API user authenticates using a Logon call and is returned a token that it can use to make subsequent REST calls to manage the Gateway.