> ## Documentation Index > Fetch the complete documentation index at: https://docs.condense.io/llms.txt > Use this file to discover all available pages before exploring further. # Amazon MSK ##### **Condense and Amazon MSK both offer managed Kafka on AWS, but they solve different problems. Amazon MSK is an infrastructure-service that simplifies the setup of Kafka brokers and storage; however, it leaves the "Application Gap" to the user, who must still manage separate AWS services like Lambda, Flink (via Amazon MSK Connect), or EKS to process data. Condense is a unified streaming platform that integrates high-performance brokers with a native execution layer. Because the platform runs directly in a private VPC, it scales both the messaging and the business logic automatically as one, removing the need to stitch together multiple AWS services.** ## **How does Condense compare against Amazon MSK?** | **System Design** | **Condense** | **Amazon MSK** | | :---------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Single Binary | A unified environment that bundles Kafka, Kubernetes, and the App Runtime into one integrated stack. | A managed infrastructure design that automates the deployment of standard Apache Kafka on AWS EC2 instances, utilizing Amazon EBS for storage and integrated IAM/VPC for security. | | Platform Model | Managed App Runtime: Custom transforms run in a managed, serverless-style environment. Condense handles containerization, state management, and orchestration. | External Execution: Business logic requires external services like AWS Lambda, Glue Streaming, or Flink. | | Scaling Scope | Full-Stack Autonomy: Automatically scales brokers, connectors, and custom-code transforms (Java/Python/Go) based on real-time consumer lag. | Infrastructure-Centric: Scales broker nodes and storage, but scaling the processing code (Lambda/Flink) is a separate task. | | Cloud Storage | Native Object Storage: Direct offloading to S3/GCS/Azure Blob; data stays in your buckets for infinite retention. | Tiered Storage: Available, but uses AWS-managed storage layers that are often opaque and separate from your buckets. | | Enterprise BYOC | 100% BYOC Native: Specifically engineered to run in your VPC to eliminate "SaaS" networking taxes and maintain 100% data sovereignty. | AWS-Native: Fully managed within AWS, but often requires complex VPC peering or PrivateLink for external access. | | Performance | Yes (Serverless Scale and Zero-Hop and ): Running GBps+ enterprise data workload and autonomous scale for spikes and variations. Application logic runs "local" to the broker, eliminating the network latency of external processors. | Standard Kafka: Reliable but adds 10–50ms latency when piping data to external Lambda or Flink clusters. | | **Feature** | **Condense** | **Amazon MSK** | | :----------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------ | | Industry Readiness | Dual-Layer Ecosystem: Access a vast library of generic connectors alongside industry specific domain-aware transforms for Mobility, IoT, and FinTech (e.g., VIN parsing, Geofencing) out of the box. | Horizontal Generalist: Raw streaming infrastructure. All industry-specific logic must be engineered from scratch. | | Apache Kafka API | Yes (100% Native): Built on upstream Kafka 4.0/KRaft. | Standard: The industry-standard managed Kafka implementation on AWS. | | Dev Experience | Integrated AI IDE: Build, test, and deploy Java or Python logic natively with built-in AI for debugging and code generation. | AWS Console/CLI: Focuses on cluster configuration. No integrated environment for developing or testing stream logic. | | Flink Integration | Native Runtime Alt: Replaces Flink for most cases with a native, event-driven microservice runner. | Extra Service: Integrates with Amazon Managed Service for Apache Flink (billed as a separate service). | | Observability | Inbuilt Full-Stack: Trace an event from ingestion through custom code to the final destination in one unified dashboard. | Siloed Monitoring: Detailed CloudWatch metrics for brokers, but application tracing requires X-Ray or external APM tools. | | Operational Effort | Zero-Ops Application: Eliminates "Microservice Sprawl." You simply write the logic; the platform autonomously runs, scales, and monitors the entire application pipeline. | Low-Ops Infrastructure: Removes Kafka complexity but requires manual wiring of AWS services to build an end-to-end app. | | **Feature** | **Condense** | **Amazon MSK** | | :----------------- | :--------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------- | | Access Experts | Direct Engineering: Built in support widget to raise ticket. Condense offers support priority based TAT for resolution | Ticket-Based: Traditional support; technical depth depends on your AWS Support Plan level. | | Cloud Availability | AWS, GCP, Azure (Customer Subscription) | AWS Only | | Security | Own Cloud -Sovereign: Inherits your IAM and security groups; 100% data localization. | Hardened: Deep integration with AWS IAM, KMS, and VPC security policies. | | SLAs | 99.95% Full-Stack: Covers the brokers, the processing logic, and the connectors under one SLA. | 99.9% Broker: Strong broker-level SLA, but excludes the uptime of your custom processing code. | | TCO | Predictable BYOC: Pay for compute (vCPU/hr) inside your VPC. No throughput or egress fees; uses your existing cloud credits (EDP). | Layered Fees: Charges for broker hours, storage, partition count, and cross-AZ data transfer. | ## **Why Switch to Condense?** Amazon MSK is a solid hosting service for Kafka brokers, but it only solves the transport problem. To actually process data, teams must architect, secure, and manage separate AWS services like Lambda, Flink (Kinesis Data Analytics), or EKS. Condense merges these layers. It provides a single environment that hosts both the high-performance brokers and the business logic. This allows developers to deploy production code directly on the stream, removing the complexity of "stitching" together multiple AWS services to build a single application. With Amazon MSK, scaling is fragmented. You can scale brokers, but you must independently manage the scaling of your processing layer whether that’s tuning Lambda concurrency or managing Flink TaskManagers. If these layers don't stay in sync during a spike, data lag occurs. Condense merges these layers. It provides a single environment that hosts both the high-performance brokers and the business logic. This allows developers to deploy production code directly on the stream, removing the complexity of "stitching" together multiple AWS services to build a single application.
 Amazon MSK is a horizontal tool designed for any use case, meaning all industry-specific logic must be engineered from scratch. Condense provides a Verticalized Ecosystem. It includes pre-built, domain-aware transforms for industries like Mobility, IoT, and FinTech. Instead of writing custom code to parse telematics or decode financial protocols, teams can use production-ready assets, moving from prototype to production months faster than on a general-purpose AWS stack. Amazon MSK billing is layered and complex, often involving charges for broker instances, storage, and significant fees for cross-AZ data transfer and NAT Gateways. Condense runs natively within a private VPC and is engineered to eliminate these "cloud taxes." By unifying storage and compute, it removes the need to move data between different AWS services (like MSK to Lambda), which cuts down on cross-AZ transfer costs and eliminates the markup typically associated with managed SaaS data movement. In an AWS-native architecture, logs and metrics are scattered across CloudWatch, X-Ray, and various service consoles. Debugging a laggy pipeline requires jumping between infrastructure and application logs. Condense embeds a Full-Code IDE and AI-Assisted Layer directly into the platform. Developers can write, test, and deploy transforms with built-in version control and state management. The unified dashboard provides full-stack observability, allowing an event to be traced from ingestion through custom code to the final sink in a single view, significantly reducing "Mean Time to Recovery" (MTTR). ## **Why is Condense the Best Way to Enable Agentic AI and Real-Time Data** | **WITHOUT CONDENSE** | **THE CONDENSE WAY** | | :----------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Coding Connectors Requires specialized Java/Scala skills to write and maintain industry specific connectors | Universal & Industry-Ready Connectors Deploy universal or specialized connectors (e.g., Telematics for Mobility) that come with built-in parsing for complex schemas | | Complex Management Development and maintenance of ever changing industry connectors becomes difficult | Configurable Output Sinks Configure and deploy pre-built sink/source connectors and through UI into the data pipeline | | Maintenance & Scalability Managing scale and failover of connectors become a challenge as the load increases | | | **WITHOUT CONDENSE** | **THE CONDENSE WAY** | | :---------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Research & Glue Code Weeks spent finding libraries and writing boilerplate just to connect components | In-Built AI IDE & Git Sync Use purpose built AI agents to create, test, and build your custom transforms with GIT support. Publish them to be used in data pipeline directly | | Maintenance & Scalability Multiple workflows and automations are required to manage availability and failover of stream processing applications | Native Stream Processing No external engine needed. Deploy your custom logic as reusable connectors or transforms that runs as containerized services | | Disjointed Lifecycle Constant context switching between IDE, Git, Cloud Console, and CI/CD tools | Management & Scalability Complete lifecycle, versioning and scalability of services managed by Custom Transform Framework | | **WITHOUT CONDENSE** | **THE CONDENSE WAY** | | :--------------------------------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- | | Absence of Insights Creation of observability layer based on disjointed CLI tools, multiple available monitoring stacks and manual log aggregation | Native Dashboard Built-in visual pipeline view to see data moving in real-time, Check and act based on states of services, logs and configurations | | Manual Tracking Manually monitoring usage, over-

provisioning, and under-utilization risks that impacts availability and increases operational cost | Comprehensive Observability Seamless integration with external tools industry accepted observability stacks for platform and cloud Ops | | |
Purpose Built AI Agents

Autonomously checks system to generate actionable insights | | **WITHOUT CONDENSE** | **THE CONDENSE WAY** | | :------------------------------------------------------------------------------------------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Complex Setup** Manual provisioning of Cloud compute resources and networking. Deployment of platform architecture for streaming usecases | Automated Provisioning Automated deployment of cloud resources and platform tailor-made for data streaming on your cloud subscription | | Maintenance Nightmare Difficult to manage uptime between Infra and other system upgrades and cross dependencies | Fully Managed Maintenance All upgrades, patches, and downtime recovery are handled by the Condense team. User stays on a stable interface with 99.95% availability | | Security Considerations Maintaining custom build governance workflows, cloud security and compliances becomes difficult over time | Security & Compliance Enterprise grade governance, audits, Information security and compliance certifications out-of-the-box |