Amazon S3 (Output, Store)

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html
• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html
• https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html

• Data lake ingestion
• Long-term archival
• Regulatory compliance
• Machine learning datasets
• Batch analytics
• Disaster recovery backups

• Streams Kafka events into Amazon S3.
• Buffers events before upload for efficient storage.
• Supports JSON, NDJSON, and CSV output formats.
• Authenticates using AWS IAM credentials.
• Automatically retries failed uploads.
• Supports region-specific S3 buckets.

1. The connector subscribes to the configured Kafka topic.
2. Incoming events are buffered in memory.
3. Files are generated according to configured batching policies.
4. Generated files are uploaded to the specified Amazon S3 bucket.
5. Upload status and operational metrics are available through Condense observability.

​

Title

• Description: Friendly name for the connector.
• Example: S3_Event_Archive_Prod

​

Bucket Name

• Description: Target Amazon S3 bucket where files will be uploaded.
• Requirements:
  • Bucket must already exist.
  • IAM credentials must have write access.
• Example: fleet-event-archive

​

Region

• Description: AWS Region where the bucket is hosted.
• Examples:
  • ap-south-1
  • us-east-1
  • eu-west-1

​

Access Key

• Description: AWS IAM Access Key ID used to authenticate with Amazon S3.
• Permissions Required:
  • s3:PutObject
  • s3:ListBucket
  • Optionally s3:GetObject

​

Secret Key

• Description: AWS Secret Access Key associated with the Access Key.
• Security Recommendation: Store credentials securely and rotate them regularly.

​

Input Topic

• Description: Kafka topic inside Condense from which events are consumed.
• Example: vehicle.events.processed

{
  "title": "S3_Event_Archive_Prod",
  "bucketName": "fleet-event-archive",
  "region": "ap-south-1",
  "accessKey": "AKIA****************",
  "secretKey": "********************************",
  "inputTopic": "vehicle.events.processed"
}

​

Access Denied

• Missing IAM permissions.
• Incorrect AWS credentials.

• Verify the IAM user has s3:PutObject permission.
• Validate the configured Access Key and Secret Key.

​

Bucket Not Found

• Bucket name is incorrect.
• Incorrect AWS region.

• Verify the bucket exists.
• Ensure the configured region matches the bucket region.

​

Upload Failures

• Network connectivity issues.
• Temporary AWS service interruptions.

• Review connector logs.
• Retry after connectivity is restored.

​

Files Not Visible

• Objects uploaded under a different prefix.

• Verify object prefixes within the bucket.
• Check the generated object key structure.

​

Object Naming

condense/<topic>/year=2026/month=07/day=13/hour=15/<uuid>.json

​

Supported Output Formats

• JSON
• NDJSON
• CSV

​

Compression

​

Downstream AWS Integrations

• Amazon Athena
• AWS Glue
• Amazon EMR
• Amazon Redshift Spectrum
• AWS Lambda
• Amazon QuickSight

1. Enable bucket versioning for recovery from accidental overwrites.
2. Configure lifecycle policies to archive or delete old objects automatically.
3. Use dedicated IAM users with least-privilege permissions.
4. Organize data using date-based folder structures for efficient analytics.
5. Monitor upload success, latency, and failures using Amazon CloudWatch and Condense observability.