> ## Documentation Index
> Fetch the complete documentation index at: https://docs.condense.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Build your First Pipeline

A **Pipeline** in Condense is the visual and functional representation of data flow between deployed connectors, transforms, and utilities inside a workspace. It is automatically materialized when a connector or transform is deployed. No manual creation needed.

> [!NOTE]
>
> **Before you begin**, make sure you have:
>
> * A workspace created and configured
> * At least one environment linked to the workspace
> * Workspace Admin or Maintainer role

## What You'll Build

```text theme={null}
[ Input Connector ]  →  [ Kafka Topic ]  →  [ Transform ]  →  [ Output Connector ]
   (Data Source)            (Stream)          (Process)          (Destination)
```

By the end of this guide, you will have a live pipeline ingesting data from a source, processing it through a transform, and delivering it to a destination, all inside your Condense workspace.

***

## Step 1 - Deploy an Input Connector

An **Input Connector** receives data from an external system and injects it into your pipeline.

1. Navigate to your **Workspace** → click **Connectors** in the left sidebar
2. Click **+ Add Connector**
3. Select **Input** as the connector role
4. Choose your connector type (e.g. Kafka, HTTP, Teltonika, Pub/Sub)
5. Fill in the required configuration fields:
   * **Title** - a unique name for this connector
   * **Output Topic** - the Kafka topic this connector will publish data to
   * **Environment Variables** - credentials, endpoints, or protocol settings
6. Click **Deploy**

> [!TIP]
>
> Once deployed, your Input Connector will appear as a block on the Pipeline canvas and begin publishing data to the configured output topic.

***

## Step 2 - Add a Transform

A **Transform** processes messages as they flow between your Input and Output connectors.

#### Option A - Prebuilt Transform

1. On the **Pipeline canvas**, click **+ Add Transform**
2. Select **Prebuilt**
3. Choose a transform type:
   * **Geofence** - location-based event triggering
   * **Alert Utility** - trigger alerts when a condition is met
   * **Split Utility** - route messages based on rules
4. Configure the required parameters:
   * **Input Topic** - must match the Output Topic from your Input Connector
   * **Output Topic** - the topic the transform will publish results to
5. Click **Deploy**

#### Option B - Custom Transform

1. Go to **Applications** → create or select an existing application
2. Build, test, and publish the application
3. Back on the Pipeline canvas, click **+ Add Transform** → **Custom**
4. Select your published application and version
5. Configure input/output topics and environment variables
6. Click **Deploy**

***

## Step 3 - Deploy an Output Connector

An **Output Connector** takes the processed data and delivers it to an external destination.

1. Navigate to **Connectors** → click **+ Add Connector**
2. Select **Output** as the connector role
3. Choose your connector type (e.g. PostgreSQL, Big Query, S3, Kafka)
4. Fill in the required configuration fields:
   * **Title** - a unique name for this connector
   * **Input Topic** - must match the Output Topic from your Transform
   * **Environment Variables** - destination credentials and settings
5. Click **Deploy**

> [!TIP]
>
> Condense automatically draws a connecting line between blocks that share matching topics. Your pipeline is now live.

***

## Step 4 - View Your Pipeline

Click **Pipeline** in the left sidebar. You will see all deployed blocks connected by lines:

![Buildyourpipeline1](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline1.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=33445e7f65da3e7f107245750956b2e0)

Pipeline Canvas Showing Connected Blocks

Clicking any block opens its **Component Detail Panel** showing the connector name, category, type, status, and Kafka topic information:

![Buildyourpipeline2](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline2.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=9ff94ce222acc6049206105c6ce49c04)

Node Details Panel

> [!NOTE]
>
> Pipelines are automatically materialized. They appear automatically as soon as connectors and transforms are deployed. You do not manually create a pipeline.

***

## Step 5 - Monitor Your Pipeline

#### Real-Time Metrics

Each block displays live CPU and memory utilization:

![Buildyourpipeline3](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline3.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=21f01381d2879f1307d20cd6675e4b57)

Real-Time Resource Utilization Metrics

Colour-coded status indicators let you instantly identify Running, Stopped, or Error states:

![Buildyourpipeline4](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline4.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=3d4e5bd90cde6ff6fbc44da3ab1aaced)

Colour-Coded Pipeline Status Indicators

#### Logs and Debugging

Access detailed real-time logs for each block with customizable streaming intervals (5 seconds by default):

![Buildyourpipeline5](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline5.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=350cc95624efd8d33fea214784df2d31)

Logs and Debugging Panel

### Editing Your Pipeline

If you need to fix a misconfiguration after deployment, use **Edit Configurations**:

![Buildyourpipeline6](https://mintcdn.com/zeliot/VW94n5Up548LL-x8/images/buildyourpipeline6.png?fit=max\&auto=format\&n=VW94n5Up548LL-x8\&q=85\&s=2ce93b5796f26a237db248657b96ce93)

Edit Configurations Panel

1. Click the block you want to edit on the Pipeline canvas
2. Click **Edit Configuration**
3. Update the required fields
4. Click **Save Changes**

> [!WARNING]
>
> Fields with a **lock icon** cannot be edited after deployment. To change locked fields, delete and redeploy the component.

### Roles and Permissions

| **Operation** | **Admin** | **Maintainer** | **Developer** | **Viewer** |
| - | :-: | :-: | :-: | :-: |
| Deploy pre-built connectors | **✓** | **✓** | **✕** | **✕** |
| Deploy custom connectors/transforms | **✓** | **✓** | **✕** | **✕** |
| Configure deployed components | **✓** | **✓** | **✕** | **✕** |
| Delete deployed components | **✓** | **✓** | **✕** | **✕** |
| View pipelines and connections | **✓** | **✓** | **✓** | **✓** |
| View component logs/configuration | **✓** | **✓** | **✓** | **✓** |

***

## Best Practices

**✓**  Use **clear topic naming** to make the pipeline canvas self-explanatory

**✓  Group related connectors and transforms** logically to simplify understanding

**✓  Avoid unused topic links** to reduce visual clutter

**✓  Regularly check component logs** to catch issues early

**✓**  Document the pipeline purpose in the workspace description

***

## Common Pitfalls

* **Misaligned Topic Names** - Double-check that Output Topic of one block exactly matches the Input Topic of the next block
* **Unused Components Left Running** - Remove or disable unused connectors to prevent unnecessary resource usage
* **Overcrowded Canvas** - Use multiple workspaces when a single pipeline becomes too dense


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.