Overview
The HTTP Input Connector enables Condense to ingest data from external HTTP or HTTPS endpoints and publish it to Apache Kafka topics for real-time stream processing. The connector supports both Polling and Webhook modes, allowing applications to either periodically fetch data from REST APIs or receive events instantly through HTTP requests. It also supports multiple authentication methods, SSL/TLS security, configurable polling intervals, and flexible data formats, making it suitable for integrating external systems with streaming pipelines.Key Features
The HTTP Input Connector provides the following capabilities:- Supports Polling and Webhook modes
- Ingests data from HTTP and HTTPS endpoints
- Publishes incoming events to Kafka topics
- Supports multiple authentication mechanisms
- Supports One-Way and Two-Way SSL (Mutual TLS)
- Supports JSON, String, Avro, and Binary payloads
- Configurable polling intervals
- Configurable batch processing
- Retry and error-handling strategies
- No-code configuration through the Condense UI
Supported Input Modes
Supported Authentication
The connector supports multiple authentication methods depending on the target HTTP service.SSL/TLS Support
The connector supports secure communication with HTTPS endpoints.Supported Payload Formats
The connector supports the following data formats:- JSON
- String
- Avro
- Binary
Typical Use Cases
The HTTP Input Connector is commonly used for:- Polling REST APIs
- Receiving webhook notifications
- Collecting IoT events
- Integrating third-party SaaS applications
- Importing business events into Kafka
- Event-driven microservices
- Real-time data ingestion
Prerequisites
Before configuring the connector, ensure:- The HTTP or HTTPS endpoint is accessible.
- Required authentication credentials are available.
- SSL certificates are available if HTTPS is used.
- A Kafka topic exists to receive incoming events.
- Appropriate permissions are available to deploy connectors.
How to Configure the HTTP Input Connector
Step 1. Navigate to Connectors
- Log in to Condense.
- Navigate to Connectors.
- Click Create Connector.
- Select HTTP Input Connector.
Add Screenshot: Connector Selection
Step 2. Configure Basic Information
Provide the following information:- Connector Name
- Description (optional)
- Target Workspace
Step 3. Configure the HTTP Endpoint
Enter:- Endpoint URL
- HTTP Method (if applicable)
- Request Headers
- Query Parameters (optional)
Add Screenshot: Endpoint Configuration
Step 4. Select Input Mode
Choose one of the following:Polling Mode
Configure:- Polling Interval
- Request Timeout
- Batch Size
Webhook Mode
Configure:- Webhook Endpoint
- Listener Configuration
Add Screenshot: Mode Configuration
Step 5. Configure Authentication
Select the required authentication type. Supported options include:- None
- Basic Authentication
- JWT
- API Key
- OAuth2
- Digest Authentication
Add Screenshot: Authentication Settings
Step 6. Configure SSL
If HTTPS is used, configure:- One-Way SSL
- Mutual TLS (Two-Way SSL)
Add Screenshot: SSL Configuration
Step 7. Configure Kafka Settings
Specify:- Output Kafka Topic
- Message Format
- Key Configuration (optional)
Step 8. Configure Error Handling
Select one of the available strategies:
Configure retry interval and maximum retries if Retry is selected.
Step 9. Deploy the Connector
- Review all configurations.
- Click Deploy Connector.
- Monitor deployment from the Pipelines page.
Editing the HTTP Input Connector
- Navigate to Pipelines.
- Select the deployed HTTP Input Connector.
- Click Edit Configurations.
- Update the required settings.
- Click Deploy Connector to apply the changes.
Add Screenshot: Edit Configuration
Deleting the HTTP Input Connector
- Navigate to Pipelines.
- Select the deployed connector.
- Click Delete Connector.
- Confirm the deletion.
Best Practices
- Prefer HTTPS for production deployments.
- Use Mutual TLS when connecting to secure enterprise services.
- Configure polling intervals based on API rate limits.
- Avoid overly aggressive polling frequencies.
- Validate authentication credentials before deployment.
- Use Retry for transient network failures.
- Monitor connector health after deployment.
- Validate payload structure before publishing to Kafka.
Limitations
- Each connector monitors a single endpoint.
- Polling frequency depends on endpoint availability and response time.
- Authentication requirements depend on the external service.
- SSL certificates must remain valid for secure communication.
Related Documentation
- HTTP Output Connector
- Apache Kafka Input Connector
- MQTT Input Connector
- Security
- SSL Configuration
Frequently Asked Questions (FAQs)
What is the HTTP Input Connector used for?
It ingests data from HTTP or HTTPS endpoints and publishes it to Kafka topics for downstream processing.Which input modes are supported?
The connector supports:- Polling
- Webhook
Which authentication methods are supported?
The connector supports:- None
- Basic Authentication
- JWT
- API Key
- OAuth2
- Digest Authentication
Can I connect to HTTPS endpoints?
Yes. The connector supports both HTTP and HTTPS endpoints.Does the connector support Mutual TLS?
Yes. Both One-Way SSL and Two-Way SSL (Mutual TLS) are supported.Which payload formats are supported?
Supported formats include:- JSON
- String
- Avro
- Binary