Snowflake (Output, Store)

• https://docs.snowflake.com/en/user-guide/data-load-overview
• https://docs.snowflake.com/en/user-guide/jdbc-configure
• https://docs.snowflake.com/en/sql-reference/sql

• Real-time analytics
• Business intelligence
• Operational reporting
• Compliance archiving
• Data warehousing
• Machine learning feature stores

• Streams Kafka events into Snowflake tables.
• Supports parameterized SQL execution.
• Supports INSERT, UPDATE, and MERGE operations.
• Authenticates using Snowflake credentials.
• Supports secure encrypted connections.
• Provides retry and operational monitoring.

1. The connector subscribes to the configured Kafka topic.
2. Incoming events are mapped to SQL parameters.
3. The connector establishes a secure connection to Snowflake.
4. SQL statements are executed against the configured database.
5. Execution status and metrics are exposed through Condense observability.

​

Title

• Description: Friendly name for the connector.
• Example: Snowflake_Streaming_Orders

​

Instance URL

• Description: Fully qualified Snowflake account URL.
• Example:

xy12345.ap-south-1.snowflakecomputing.com

• Requirements:
  • Must match your Snowflake account endpoint.
  • Must be reachable from the Condense deployment.

​

Username

• Description: Snowflake user used for authentication.
• Requirements:
  • Must have permissions to connect.
  • Must have privileges to execute the configured SQL.

​

Password

• Description: Password associated with the Snowflake user.
• Security Recommendation: Store credentials securely and rotate them regularly.

​

Database Name

• Description: Target Snowflake database.
• Example: STREAMING_DB
• Requirements:
  • Database must already exist.
  • User must have access permissions.

​

Query

• Description: SQL statement executed for each incoming record or batch.
• Supported Statements:
  • INSERT
  • UPDATE
  • MERGE
• Example:

INSERT INTO sensor_readings
(device_id, event_time, temperature)
VALUES (?, ?, ?)

• Recommendation: Use parameterized queries to improve security and performance.

​

Input Topic

• Description: Kafka topic inside Condense that supplies events.
• Example: sensor.events.processed

{
  "title": "Snowflake_Streaming_Orders",
  "instanceUrl": "xy12345.ap-south-1.snowflakecomputing.com",
  "username": "condense_writer",
  "password": "********",
  "databaseName": "STREAMING_DB",
  "query": "INSERT INTO sensor_readings (device_id, event_time, temperature) VALUES (?, ?, ?)",
  "inputTopic": "sensor.events.processed"
}

​

Authentication Failed

• Incorrect username or password.
• Invalid Snowflake account URL.

• Verify account endpoint.
• Confirm credentials using the Snowflake web interface or SQL client.

​

SQL Execution Failed

• Invalid SQL syntax.
• Missing table or columns.
• Parameter mismatch.

• Validate the SQL statement directly in Snowflake.
• Ensure the number of placeholders matches the mapped event fields.

​

No Data Written

• No events in the input topic.
• Incorrect topic configuration.

• Verify the input topic is receiving events.
• Review connector logs for processing errors.

​

Connection Timeout

• Network restrictions.
• Firewall rules.
• Incorrect endpoint.

• Verify outbound connectivity.
• Confirm the Snowflake endpoint is accessible.

​

Permission Denied

• Insufficient database privileges.

• Grant the required INSERT, UPDATE, or MERGE privileges to the configured user.

​

Batch Processing

​

Parameterized Queries

​

Semi-Structured Data

​

Schema Evolution

​

Performance Optimization

• Use appropriately sized Snowflake warehouses.
• Batch writes whenever possible.
• Partition large tables appropriately.

1. Use dedicated service accounts with the minimum required privileges.
2. Always use parameterized SQL statements.
3. Validate SQL queries before deploying the connector.
4. Store credentials securely using secret management solutions.
5. Monitor connector throughput, latency, and failures using Condense observability and Snowflake monitoring.