Anchor met with op Coinbase behulp Van AWS decks we are making data stream

By dan Moore, Eric Sung, LV Lu, Xinyu Liu
Tl; dr: Coinbase uses AWS Managed streaming (Dec) for Kafka for ultra-low latency, seamless cross-service communication, data ETLS, and database modification data collection (CDC). The engineers from our Data Processing Platform team will present this work at the AWS Re: Invent conference in November 2021.
Summary Coinbase, we consume billions of activities every day from users, applications and crypto resources in our products. Clickstream data is collected through web and mobile clients and written to Kafka using Ruby and Golang’s own SDK package. In addition, change data collection (CDC) streams from different databases are fed through Kafka Connect. A key customer of these Kafka messages is our data ETL pipeline, which sends data to our data warehouse (Snowflake) for further analysis by our teams of data processing experts and data analysts. In October, the company’s internal services (such as our core brokerage and real-time inventory management products) rely on our Kafka cluster to perform critical applications with low latency (less than 10 Ms).
With the help of AWS-managed Kafka (CEC), our team has reduced Kafka’s daily operating costs associated with maintaining and repairing the property, allowing us to focus our engineering time on important business needs. We found that scaling / deleting Kafka clusters and updating brokers to the latest version of Kafka is easy and secure with MSK. This article describes our basic architecture and complete vehicle ecosystem that we developed around CEC.
MSKConfig configuration and benefits: TLS-validated, full A-Z outageMulti cluster support for 30 broker node clusters and~17tb / broker99 storage in multiple access zones to protect from the Outagemulti cluster. Up to 9% of monthly SLA uptime from awsbenefits: since CEC manages AWS, one of the biggest advantages is that we can prohibit in-house engineers from actively supporting zookeeper / broker nodes. This includes all broker security patch October updates, node repairs, and AWS