Skip to main content

​
Overview

The Data Utility is a no-code stream processing utility that enables you to modify streaming events without writing custom code. It allows you to create new fields, update existing values, remove unnecessary attributes, and restructure JSON payloads as data flows through your Kafka pipeline. Unlike batch data processing, the Data Utility performs transformations on every event as it arrives, allowing downstream applications to consume clean, enriched, and standardized data in real time. The Data Utility is ideal for preparing data before it reaches analytics systems, databases, APIs, or downstream applications.

​
Key Features

​
Real-Time Data Transformation

Transform streaming events as they flow through the pipeline without introducing additional processing stages.

​
No-Code Configuration

Configure transformations through the Condense UI without writing application code.

​
Field Manipulation

Create, update, rename, or remove fields from incoming JSON events.

​
Data Standardization

Normalize incoming data into a consistent structure before it reaches downstream systems.

​
Pipeline Integration

Deploy directly into existing Condense pipelines alongside connectors, transforms, and other utilities.

​
Live Deployment

Update transformation rules and redeploy without rebuilding applications.

​
How to Use the Data Utility

​
Prerequisites

Before configuring the Data Utility, ensure that:
  • You have Admin, Maintainer, or Developer access.
  • The input Kafka topic already exists.
  • Incoming messages are valid JSON.
  • The output topic has been created (or will be created during deployment).

​
Step 1. Access the Data Utility

  1. Log in to Condense.
  2. Open your Workspace.
  3. Navigate to Transforms.
  4. Select Data Utility.
Image Placeholder

​
Step 2. Select the Input Schema

The utility needs the input schema before transformations can be configured.

​
Option A (Recommended): Read from Topic

  1. Click Choose Topic.
  2. Select the Kafka topic.
  3. Condense automatically reads the JSON schema.
Image Placeholder

​
Option B: Upload JSON

If the topic is unavailable:
  1. Click Upload JSON.
  2. Upload a sample JSON payload.
  3. Validate the schema.
The uploaded JSON must match the structure of the incoming events.
Image Placeholder

​
Step 3. Configure the Utility

Provide the following information.

​
Step 4. Configure Transformations

Create one or more transformation rules. Supported operations include:
  • Add a new field
  • Update an existing field
  • Rename a field
  • Remove a field
  • Copy values between fields
  • Apply default values
  • Build nested JSON structures
As rules are added, Condense displays a live preview of the resulting payload.
Image Placeholder

​
Step 5. Validate

Before deployment:
  • Verify the generated schema.
  • Review the output preview.
  • Ensure the output topic is correct.
Resolve any validation errors before deployment.

​
Step 6. Deploy

Click Deploy Utility. After deployment:
  • The Data Utility becomes part of the active pipeline.
  • Incoming events are transformed in real time.
  • The transformed events are published to the configured output topic.
The deployed utility appears in Pipelines for monitoring.
Image Placeholder

​
Editing a Data Utility

  1. Open Pipelines.
  2. Select the deployed Data Utility.
  3. Click Edit Configuration.
  4. Modify the required transformation rules.
  5. Click Deploy Utility.
The updated configuration replaces the previous deployment.

​
Deleting a Data Utility

  1. Navigate to Pipelines.
  2. Select the deployed Data Utility.
  3. Click Delete Utility.
  4. Confirm the deletion.
Deleting the utility removes it from the pipeline but does not delete Kafka topics.

​
Best Practices

  • Keep transformations focused on a single responsibility.
  • Use descriptive names for utilities.
  • Validate sample payloads before deployment.
  • Avoid unnecessary field duplication.
  • Standardize payload structures before downstream processing.
  • Test transformations in a non-production environment first.

​
Common Pitfalls

​
Invalid JSON Schema

Ensure uploaded JSON accurately represents incoming events.

​
Output Topic Conflicts

Do not use the same topic for both input and output unless explicitly supported.

​
Excessive Transformations

Avoid combining unrelated transformations into one utility. Multiple smaller utilities are easier to maintain.

​
Schema Changes

If the input schema changes, review and update transformation rules before redeploying.

​
Frequently Asked Questions

​
Can the Data Utility modify incoming events?

Yes. It can create, update, rename, remove, and restructure fields before publishing the transformed event.

​
Does the original message change?

No. The original event remains unchanged in the input topic. The transformed event is written to the configured output topic.

​
Can multiple transformations be configured?

Yes. Multiple transformation rules can be applied to a single event during processing.

​
Is coding required?

No. The Data Utility is completely configuration-driven and requires no programming.

​
Can I edit a deployed Data Utility?

Yes. Existing configurations can be updated and redeployed from the Pipelines page.
  • Transforms
  • Pipelines
  • Applications
  • Alert Utility
  • Split Utility