Preliminary Root Cause:
SimpliGov utilizes Azure Service Bus as it’s message queuing system for things like submissions, dashboard updates etc. Azure Government support informed SimpliGov that they performed updates to Azure Service Bus infrastructure and between 9:00 and 10:00 PST on 21-10-2021, SimpliGov was identified as a customer using Service Bus in USGov Arizona which experienced increased error rates for service bus.
SimpliGov received “Connection reset by peer” error messages from the Azure Service bus service which resulted in partial connection loss to the service. Azure support states that the primary cause of the issue on their side was that a subset of backend instances experienced an unexpected high utilization due to a platform upgrade in US Gov Arizona. The Azure Government product engineering group allocated more bandwidth for the host nodes which brough the instances back to a healthy state. From the SimpliGov side, this mitigating action allowed our services to connect to Azure Service bus as expected.
During this period of time, most intake submissions should have worked as expected but some users would have experienced slower than expected dashboard synchronization times and potential duplicates depending on their workflow configuration as submissions were retried when initial connections to service bus failed.
Mitigation:
In the immediate term, SimpliGov support employed manual dashboard synchronization processes to ensure that any records coming in were being reflected as soon as possible. Azure support resolved the Service Bus issue on their end and this allowed SimpliGov processing and dashboard synchronization services to function as expected.
Going forward, SimpliGov has implemented several new features within our upcoming Dec 2021/Jan 2022 Production release allowing for better failover and fault tolerance in such scenarios. SimpliGov will be hosted on Azure Kubernetes Services instead of Azure Service Fabric, split service bus queue architecture will be enabled and additional retry policies have been added to handle scenarios whereby Azure Services return transient errors. These 3 particular items, in addition to smaller individual fixes and improvements should reduce the propencity of such incidents going forward.
Next Steps:
We apologize for the impact to affected customers. SimpliGov will continue to monitor the situation going forward with azure service bus and configure specific alerts relating to Service bus and “connection reset by peer' messages. We will be deploying additional updates / architectural changes as part of our upcoming production release scheduled for Jan 2022 to also improve future fault tolerance.
All customer records being processed throughout the incident period should be consistent with their expected statuses. Customers do not need to take any reconciliatory actions in their production tenants unless you are directly notified to do so by the SimpliGov team.
Additional RCA from Microsoft Azure Government support:
Incident Summary:
Between 10/21 17:07 UTC and 10/21 19:14 UTC, Event Hub, Service Bus, and Relay service customers may have intermittently experienced an increase in latency or timeouts on runtime operations.
Root Cause:
On 10/21/21 at 17:07 UTC, an upgrade was performed on clusters servicing Service Bus, Event Hub, and Relay services. During the upgrade, as TCP connections were disconnected from each gateway machine, there was a noticeable increase in TCP connection requests as existing TCP connections were terminated resulting in high CPU utilization by the LSASS process that handles TLS/SSL handshakes. As the LSASS process hit the CPU threshold set by the service teams, processing of TLS/SSL handshakes were slowed causing increased latency/delay in serving send and receive requests made by client applications.
The issue was mitigated at 10/21/21 at 19:14 UTC after the team scaled out the cluster and added machines to allocate more resources to process incoming requests.
Next Steps:
Microsoft apologizes for the impact to affected customers. We have scaled out our gateway machines to increase capacity and are taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):