Network Latency

This incident has been resolved as of 9:25 AM PST.
Our cloud platform provider (GCP) has notified us that the network issues should no longer be happening.

We have verified that all products have recovered. There is minimal data backup, and we will be closely monitoring data pipelines in the next few hours.

A full incident post-mortem will be published soon.

We are currently investigating a network latency issue. This is caused by downtime at one of our cloud providers, specifically Google Cloud Platform, which is affecting their global customers.


Here is a summary of what is being investigated on their end:
https://status.cloud.google.com/incidents/FRpfyvgG3MUeSTTuX1Dx
https://status.cloudamqp.com/

During this time, the following services are affected:
- All Parade applications will see slowdowns in terms of network requests
- Parade data processing will be delayed (this includes load and carrier data processing from the TMS)

According to their site, this problem is intermittent, and seems to be causing intermittent timeouts and network packet loss.

Pending their investigation, we are planning workarounds on our end to bring our cluster back to full health.