Degraded performance due to resource contention
Incident Report for Fenwick
Postmortem

We recently were made aware about slower than usual peak performance within Fenwick Cloud apps. We apologise for any delays this may have caused with processing due to lower than normal peak performance.

A long running user process was started at 11:30am and continued to consume resources which ended in a temporary exhaustion of apportioned resources. This means processes were running slower than normal, peaking around 2:30pm. We identified the process and stopped its execution which returned peak performance levels to normal.

Our “fair-use” controls took effect which limited the impact of this issue.

Future Mitigation

We’re looking at our options to improve telemetry and reporting to be more quickly notified about potential runaway processes so they don’t reach a point of any noticeable disruption amongst all our users.

We routinely rebalance workloads to provide the best peak performance based on trends and user habits and will review our fair-use controls to provide the best balance of peak performance and sustained performance.

Posted Jan 18, 2024 - 09:29 AEDT

Resolved
This incident has been resolved.
Posted Jan 16, 2024 - 15:45 AEDT
Monitoring
Resource pressure has been relieved and metrics are returning to normal.
Posted Jan 16, 2024 - 15:31 AEDT
Identified
We've had some reports of slower than usual performance on our cloud platform.

After initial investigation we've identified the cause of the resource.contention and are reviewing options for remediation.
Posted Jan 16, 2024 - 15:16 AEDT
This incident affected: Fenwick Cloud (Hosted Platform).