Google Cloud Pub/Sub (Output, Stream)

Connector Type: Output Connector Description Publishes processed data to Google Cloud Pub/Sub topics for downstream cloud applications. Brief Description The Google Cloud Pub/Sub Stream Output Connector allows Condense to publish real-time messages from Kafka-native topics into a Google Cloud Pub/Sub topic via a configured subscription. It is designed for teams that rely on GCP’s native messaging infrastructure and want to extend their Condense pipeline outputs into Google-native services, APIs, and event-driven systems. This connector bridges Kafka-native output topics within Condense to GCP Pub/Sub, enabling seamless integration with Cloud Functions, BigQuery, App Engine, IoT Core, and other GCP-native subscribers. Links to Relevant Documents

Google Cloud Pub/Sub Overview: https://cloud.google.com/pubsub/docs/overview
Topic and Subscription Management: https://cloud.google.com/pubsub/docs/admin
IAM Roles and Permissions (Publisher): https://cloud.google.com/pubsub/docs/access-control
Service Account Key JSON Management: https://cloud.google.com/iam/docs/creating-managing-service-account-keys
Publishing Messages to Pub/Sub: https://cloud.google.com/pubsub/docs/publisher

Connector Overview This connector acts as a publisher to Google Cloud Pub/Sub. It listens to events from a configured Kafka-native input topic in Condense, formats each message as a Pub/Sub-compliant payload, and publishes them to a target topic within GCP. Key scenarios include: A. Forwarding streaming output to GCP-hosted downstream systems B. Triggering GCP Cloud Functions or Cloud Run apps from Kafka-native logic C. Connecting Condense pipelines with GCP’s data lake, ML, or analytics services D. Hybrid deployments across Condense (BYOC) and Google Cloud Core Functionality

Kafka-to-GCP publishing from Condense topics to Pub/Sub topics
Service Account Authentication for secure GCP access
Schema-less JSON Publishing with automatic encoding
High Throughput async message publishing with retries
Backpressure-aware publishing that scales with Condense topic load

How It Works in Condense

The connector consumes events from a configured input topic in Condense.
Messages are serialized to a Pub/Sub-compatible JSON format.
Using the Service Account Key, the connector authenticates and publishes to Pub/Sub.
The configured subscription receives the messages immediately in GCP.
Any downstream system subscribed to the GCP Pub/Sub topic receives the data.

Note: The connector does not create topics or subscriptions. These must exist and be accessible beforehand. Configuration

CATEGORY	FIELD NAME	DESCRIPTION	REQUIRED
General	Title	Unique name for the connector instance within Condense	Yes
Pub/Sub	Subscription Name	The name of the GCP Pub/Sub subscription linked to the topic	Yes
Pub/Sub	Project ID	The Google Cloud Project containing the Pub/Sub topic	Yes
Pub/Sub	Service Account Key	JSON credentials of a service account with Pub/Sub Publisher role	Yes
Condense	Topic (Input)	Kafka-native Condense topic to read from	Yes

Field-by-Field Explanation 1. Title Purpose: Internally identifies the connector in the Condense UI and logs. Format: Simple identifier. Must be unique within the workspace. Example: pubsub-outbound-connector, analytics-to-gcp-stream 2. Subscription Name Definition: Name of the GCP Pub/Sub subscription attached to the topic that should receive published data. Important: Ensure this subscription exists and is actively connected to the correct topic. Example: raw-alerts-sub, iot-stream-analytics-sub 3. Project ID Definition: The unique GCP project identifier that owns the topic. Purpose: Used to route the publishing API calls correctly. Example: condense-platform-io, data-streamer-dev-19233 4. Service Account Key Type: Full JSON object of the service account credentials. Required IAM Role: Must have roles/pubsub.publisher access on the destination topic. Security: Never expose this key in logs or UI. Use environment variable mapping or vault reference if available. How to Obtain: A. Go to IAM > Service Accounts in the GCP Console. B. Create a new service account or use an existing one. C. Grant Pub/Sub Publisher role. D. Generate and download a JSON key. 5. Topic (Input) Definition: The name of the Kafka-native Condense topic that supplies data for this connector. Role: Acts as the source stream for all outgoing messages. Example: sensor-enriched-events, ml-detections, fraud-detection-results Sample Configuration Example

{
  "title": "gcp-pubsub-output",
  "subscriptionName": "stream-app-subscription",
  "projectId": "iot-insights-88271",
  "serviceAccountKey": "{... full JSON key ...}",
  "inputTopic": "enriched_gps_data"
}

Troubleshooting and Common Issues 1. Permission Denied A. Ensure the service account has roles/pubsub.publisher. B. Validate the service account’s access to the correct project. 2. Topic or Subscription Does Not Exist A. This connector does not create Pub/Sub resources. B. Pre-create the topic and attach the subscription before use. 3. Failed Message Delivery A. Messages that fail during publishing will be retried automatically. B. Persistent failures may indicate a network or quota issue. 4. Serialization Errors A. Ensure message payloads are JSON serializable. B. Avoid binary blobs or malformed UTF-8 strings. 5. API Rate Limits A. Google Cloud Pub/Sub enforces rate limits. B. Distribute load or increase quota if required. Advanced Considerations 1. Message Format A. Condense publishes messages as raw JSON. B. Metadata fields (e.g., event timestamp, source, topic ID) can optionally be included via transforms before output. 2. Pub/Sub Topic Retention A. Messages published to a topic persist for a configurable retention window (default: 7 days). B. Consider retention vs downstream consumer acknowledgment timing. 3. Pub/Sub Delivery Guarantees A. Delivery is at least once from Condense to Pub/Sub. B. Consumers must be idempotent if downstream duplication matters. 4. Large Payloads A. Google Pub/Sub enforces a 10MB limit per message. B. Split large payloads into multiple messages in the upstream transform if necessary. Best Practices

Use dedicated Pub/Sub topics for Condense outputs to avoid noisy data mixing.
Rotate service account keys regularly and store them securely.
Monitor delivery metrics via Condense logs and GCP Pub/Sub metrics.
Integrate Pub/Sub outputs with GCP-native consumers like Cloud Functions, Dataflow, or BigQuery for scalable processing.

Spotlight

Understanding Condense

Build Pipeline

Vapr AI

Security and Compliance

Help and Support

AI Agents

Fully Managed Kafka

Google Cloud Pub/Sub (Output, Stream)