Posthumous event: 19 May 2021

Bryant Khau and Leonardo Zizzamia Summary Between 05:50 and 07:38 on Wednesday, May 19, there were connectivity problems with, Coinbase mobile apps and Coinbase Pro. During this time, many users, including features such as buying, selling, and trading, encountered slow loading times and errors while trying to access Coinbase. This article provides a detailed look at the error, explains its cause, and explains the changes we've made to avoid similar errors. The outage Before this event, there was a massive increase in traffic (ETH fell 20%, BTC 25%) as many users reacted to a sudden drop in prices in the crypto market. A group of oncall technicians gathered after being called about high failure rates for various departments. Related services were as follows: Disconnected Web Servers: This was causing users who were not logged in to an error page while visiting Graphical SQL Service: This caused parts of the mobile app to load very slowly and Coinbase Pro API errors of around 10%: This made Coinbase Pro partially inaccessible. Tried to buy crypto with card, denied in different groups to examine each issue in parallel and prioritize follow-up actions. Root Cause Analysis In the days following the interruption, we recreated a clear picture of what happened from the first minute. The disconnected pages were largely inaccessible as the samples started to fail and took more than 40 minutes to return to a healthy state. The rapid increase in requests eventually reached the maximum threshold of manually boosted Nginx router connections during the crash. This ultimately resolved the bottleneck in NodeJS HTML Response2. We've seen timeouts and increased latency in our GraphQL service, which collects data from core services. The delays were due to GraphQL's very slow autoscaling. Aut

Join the Discussion

  • BrokerEUR/USD
    Core Spreads 0.6pips (variable) margin: 3.33%
    SpreadEX 0.6pips (variable) margin: 3.33%
  • Back to top