Skip to main content
A Pipeline in Condense is the visual and functional representation of data flow between deployed connectors, transforms, and utilities inside a workspace. It is automatically materialized when a connector or transform is deployed. No manual creation needed.
Before you begin, make sure you have:
  • A workspace created and configured
  • At least one environment linked to the workspace
  • Workspace Admin or Maintainer role

​
What You’ll Build

By the end of this guide, you will have a live pipeline ingesting data from a source, processing it through a transform, and delivering it to a destination, all inside your Condense workspace.

​
Step 1 - Deploy an Input Connector

An Input Connector receives data from an external system and injects it into your pipeline.
  1. Navigate to your Workspace → click Connectors in the left sidebar
  2. Click + Add Connector
  3. Select Input as the connector role
  4. Choose your connector type (e.g. Kafka, HTTP, Teltonika, Pub/Sub)
  5. Fill in the required configuration fields:
    • Title - a unique name for this connector
    • Output Topic - the Kafka topic this connector will publish data to
    • Environment Variables - credentials, endpoints, or protocol settings
  6. Click Deploy
Once deployed, your Input Connector will appear as a block on the Pipeline canvas and begin publishing data to the configured output topic.

​
Step 2 - Add a Transform

A Transform processes messages as they flow between your Input and Output connectors.

​
Option A - Prebuilt Transform

  1. On the Pipeline canvas, click + Add Transform
  2. Select Prebuilt
  3. Choose a transform type:
    • Geofence - location-based event triggering
    • Alert Utility - trigger alerts when a condition is met
    • Split Utility - route messages based on rules
  4. Configure the required parameters:
    • Input Topic - must match the Output Topic from your Input Connector
    • Output Topic - the topic the transform will publish results to
  5. Click Deploy

​
Option B - Custom Transform

  1. Go to Applications → create or select an existing application
  2. Build, test, and publish the application
  3. Back on the Pipeline canvas, click + Add Transform → Custom
  4. Select your published application and version
  5. Configure input/output topics and environment variables
  6. Click Deploy

​
Step 3 - Deploy an Output Connector

An Output Connector takes the processed data and delivers it to an external destination.
  1. Navigate to Connectors → click + Add Connector
  2. Select Output as the connector role
  3. Choose your connector type (e.g. PostgreSQL, Big Query, S3, Kafka)
  4. Fill in the required configuration fields:
    • Title - a unique name for this connector
    • Input Topic - must match the Output Topic from your Transform
    • Environment Variables - destination credentials and settings
  5. Click Deploy
Condense automatically draws a connecting line between blocks that share matching topics. Your pipeline is now live.

​
Step 4 - View Your Pipeline

Click Pipeline in the left sidebar. You will see all deployed blocks connected by lines:
Buildyourpipeline1

Pipeline Canvas Showing Connected Blocks

Clicking any block opens its Component Detail Panel showing the connector name, category, type, status, and Kafka topic information:
Buildyourpipeline2

Node Details Panel

Pipelines are automatically materialized. They appear automatically as soon as connectors and transforms are deployed. You do not manually create a pipeline.

​
Step 5 - Monitor Your Pipeline

​
Real-Time Metrics

Each block displays live CPU and memory utilization:
Buildyourpipeline3

Real-Time Resource Utilization Metrics

Colour-coded status indicators let you instantly identify Running, Stopped, or Error states:
Buildyourpipeline4

Colour-Coded Pipeline Status Indicators

​
Logs and Debugging

Access detailed real-time logs for each block with customizable streaming intervals (5 seconds by default):
Buildyourpipeline5

Logs and Debugging Panel

​
Editing Your Pipeline

If you need to fix a misconfiguration after deployment, use Edit Configurations:
Buildyourpipeline6

Edit Configurations Panel

  1. Click the block you want to edit on the Pipeline canvas
  2. Click Edit Configuration
  3. Update the required fields
  4. Click Save Changes
Fields with a lock icon cannot be edited after deployment. To change locked fields, delete and redeploy the component.

​
Roles and Permissions

​
Best Practices

✓ Use clear topic naming to make the pipeline canvas self-explanatory ✓ Group related connectors and transforms logically to simplify understanding ✓ Avoid unused topic links to reduce visual clutter ✓ Regularly check component logs to catch issues early ✓ Document the pipeline purpose in the workspace description

​
Common Pitfalls

  • Misaligned Topic Names - Double-check that Output Topic of one block exactly matches the Input Topic of the next block
  • Unused Components Left Running - Remove or disable unused connectors to prevent unnecessary resource usage
  • Overcrowded Canvas - Use multiple workspaces when a single pipeline becomes too dense