Vero

Vero Status
Official status page

All Systems Operational
12/07
13/07
14/07
15/07
16/07
17/07
18/07
19/07
20/07
21/07
22/07
23/07
24/07
25/07
26/07
27/07
28/07
29/07
30/07
31/07
01/08
02/08
03/08
04/08
05/08
06/08
07/08
08/08
09/08
10/08
11/08

Latest Incidents

Resolved Issues calculating segments and sending newsletters

We are currently investigating an issue affecting the calculation of segments which will mean your newsletter may experience delays if attempting to send.

about 2 months ago Official incident report

Resolved Email template discrepancies

We're investigating reports that some email templates are rendering differently than expected. If this is the case you should see these emails being discarded instead of sent. This ensure's that users will not receive incorrect content. Please email [email protected] if you have any questions regarding your account or emails.

Resolved API connections issues

We have been alerted that there are reports of connection issues to our API and we are investigating.

Resolved Delays in email processing and statistics

On Wednesday 27th May at approximately 10am UTC we observed intermittent delays in API processing and, from approximately 12pm, delays in workflow and behavioural email processing. These delays were not fully resoled until 4pm UTC. From that time, the queues were processing normally. We have made several changes in response to the delays seen on the 26th and 27th May and are confident that delays in email or API processing should not occur again. We monitored these queues through Thursday 27 May and will continue to do so. Both API and email status components were updated to reflect the delays as they occurred. If you have any questions, please email us at [email protected]

Resolved Delays in email processing and statistics

We are sharing this post mortem to provide an easy-to-navigate record of our prior component updates this week. On Tuesday 26th May at around 12pm UTC we began observing intermittent delays in email processing and statistics collection. Upon investigation, this appeared to be related to some infrastructure changes we had recently deployed. Our Operations Team was able to manage many of these delays as they occurred, but message queues for **some** customers were delayed on and off between 12pm UTC and 6pm UTC. From 6pm processing had returned to real time. In response to these delays, we reviewed and adjusted the previously deployed infrastructure changes. We have updated the status of all relevant components in real time throughout the week. If you have any questions, please email us at [email protected]

Resolved Increased API error rates

At 17:18 UTC 18 May 2020, one of the databases backing our API began experiencing issues. Between 17:18 and 17:21 a small number of API requests were dropped (approximately 800 total). We responded to the issue and between 17:21 and 17:40 we were able to rectify the core problem. By 17:47 we were processing at normal capacity. It took approximately one hour to work through the backlog of API requests that had accumulated. Triggered messages were delayed during this period. No data was lost. Aside from the ~800 dropped requests, no data or emails were lost. This issue was caused by a regression introduced with a recent infrastructure change. We are reviewing our tests and QA moving forward. Thanks. If you have any questions, please write to us at [email protected] — we're happy to answer them.

Resolved Delayed email processing

At 17:32 UTC yesterday (29th April) we observed degrading performance in processing emails and updated the email status component whilst investigating. We identified an increasing number of errors related to an associated database, which we modified and rebooted. We were able to quickly regain access to the affected database and begin processing the backlog of emails by 18:20 UTC. Email processing and statistics were delayed for a total 48 minutes and are now processing real time. All API processing and segment queries remained realtime throughout and were not affected. If you have any questions about your account please contact [email protected]

Resolved Issues calculating segments and sending newsletters

We are currently experiencing issues calculating segments and sending newsletters. This is a result of issues with our primary segmentation data store. We are working with the service provider responsible for this service to understand the root cause. We will provide further updates as soon as they become available. Note that automated messages (Workflows, Behavioural and Transactional campaigns) and data collection (updating user properties and tracking events) remain **unaffected** and are **operating as normal**.

Resolved Logs and Customer Profiles pages not responding

At 12:30 UTC yesterday, several of the nodes in the search cluster powering the Logs and Customer Profile pages in Vero went offline. We were alerted to this issue and, by 13:20 UTC, had begun to restore the affected nodes. Due to recovery time the UI was returned to normal at 16:30 UTC. During the intervening period the UI was unavailable. Once restored, there were delays in processing logs into the UI as the accumulated backlog caught up. No other processes were affected. We have investigated and understood the root cause of this issue. Fortunately, we have been working toward replacing this component of our architecture over the last few months as part of planned infrastructure changes. We will be finalising this work within the new two weeks and are confident that this issue should not arise again. If you have any questions please email us at [email protected]

Resolved Sendgrid incorrectly encoding non-roman characters

Based on testing we have conducted, Sendgrid appears to be incorrectly encoding non-roman characters in outgoing emails. This is resulting in end users receiving content that replaces the non-roman characters with random encoded characters. This will only affect Vero customers who use Sendgrid accounts to deliver their emails. We have raised these examples with Sendgrid and will keep you updated. Please refer to this post for more publicly available information: https://news.ycombinator.com/item?id=21453178 We wanted to make you aware of this occurrence. If you want to discuss options such as switching to an internal Vero email provider, please email us at [email protected]

Stats

0 incidents in the last 7 days

0 incidents in the last 30 days

Follow Vero and other services to get alerts about incidents in realtime.

Get started now
14 days of trial / No credit card required

Don't miss another incident in Vero!

Ctatus aggregates status page from services so you don't have to. Follow Vero and hundreds of services and be the first to know when something is wrong.

Get started now
14 days of trial / No credit card required