tag:gorillastack.statuspage.io,2005:/historyGorillaStack Status - Incident History2024-03-28T13:51:56+11:00GorillaStacktag:gorillastack.statuspage.io,2005:Incident/189749122023-11-01T16:57:57+11:002023-11-01T16:57:57+11:00Webapp intermittent availability issue<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>16:57</var> AEDT</small><br><strong>Resolved</strong> - We are pleased to inform you that the recent issue causing intermittent service disruptions has been resolved, and we have not observed any further occurrences.<br /><br />Our technical team successfully addressed the problem, and the fix has been implemented.<br /><br />After a period of monitoring, we are delighted to report that the issue has not recurred. Our platform is stable and running smoothly.<br /><br />Thank you for your understanding and patience during this time. We appreciate your continued support.<br /><br />For any questions or concerns, please don't hesitate to reach out to our support team.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>12:09</var> AEDT</small><br><strong>Monitoring</strong> - We'd like to provide an update on the recent incident. Our technical team has identified and resolved the issue, which caused intermittent service disruptions.<br /><br />We are now closely monitoring our platform to ensure the stability of our services.<br /><br />Thank you for your patience and understanding during this time.<br /><br />If you have any questions or concerns, don't hesitate to reach out to our support team.<br /><br />We appreciate your continued support.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>14:53</var> AEDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of web app access issues and are currently looking into the issue.<br />At 9:50 AEDT today, we experienced an incident that temporarily affected the availability of our Webapp. We understand the importance of your access to our platform, and we sincerely apologize for any inconvenience this may have caused.<br /><br />Resolution: Our technical team took immediate action to restore service by redeployment, which resolved the issue temporarily.<br /><br />Our team is actively investigating the root cause of this incident to prevent its recurrence. We are committed to ensuring the stability and reliability of our platform.<br /><br />- We will provide regular updates on our progress.<br />- Rest assured, your data and security are of utmost importance, and we are taking all necessary precautions.<br /><br />We appreciate your understanding and patience during this investigation. Your feedback is invaluable to us, and we are dedicated to keeping you informed.</p>tag:gorillastack.statuspage.io,2005:Incident/182683202023-08-26T20:33:59+10:002023-08-26T22:43:58+10:00Error in Event Processing<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>20:33</var> AEST</small><br><strong>Resolved</strong> - We have identified a fix and deployed it to production. <br />The issue is related to a breaking change introduced in a patch version of our runtime dependencies. <br />https://nodejs.org/en/blog/release/v18.17.0 in particular the breaking change in URL parser.<br /><br />We have monitored the fix and are satisfied that it has resolved the situation, thanks to all for your patience.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>15:19</var> AEST</small><br><strong>Investigating</strong> - We have monitored the fix and are now investigating a different avenue</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>14:01</var> AEST</small><br><strong>Monitoring</strong> - We have identified the issue and have deployed a fix and we are monitoring said fix now.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>13:45</var> AEST</small><br><strong>Investigating</strong> - We are currently investigating an issue where event processing is currently impacted. This may result in inaccurate reporting on events and rule executions. Scheduled executions are still working but there is a degradation in reporting coming from those rule executions.</p>tag:gorillastack.statuspage.io,2005:Incident/180728322023-08-08T09:29:02+10:002023-08-08T09:29:02+10:00Scheduler Errors<p><small>Aug <var data-var='date'> 8</var>, <var data-var='time'>09:29</var> AEST</small><br><strong>Resolved</strong> - We are pleased to inform you that the issue regarding the random scheduler failures has been successfully resolved. We apologize for any inconvenience this incident may have caused and greatly appreciate your patience throughout this process.<br /><br />After thorough investigation, we identified that recent network setting updates inadvertently led to rule execution issues. These network changes disrupted the expected execution flow of scheduled rules, resulting in the sporadic failures observed.<br /><br />The resolution process has successfully addressed the issues, and all scheduled rules and executions are now working as intended.<br /><br />We appreciate your understanding and collaboration throughout this incident. Your feedback and insights have been instrumental in achieving a swift resolution. If you have any further questions or concerns, please feel free to reach out to our support team at help@gorillastack.com.<br /><br />Thank you for choosing our services and for your continued trust in us.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>18:08</var> AEST</small><br><strong>Identified</strong> - We are currently investigating elevated level of scheduler failures from late last week.<br /><br />The scheduler failures have led to missed scheduled rule executions affecting multiple customers. We apologize for any inconvenience this may have caused and assure you that we are working diligently to resolve the issue.<br /><br />We will provide regular updates as we progress in our investigation and resolution efforts. Your patience and understanding during this time are greatly appreciated.<br /><br />For any questions or concerns, please reach out to our support team at help@gorillastack.com.</p>tag:gorillastack.statuspage.io,2005:Incident/174272402023-05-30T06:30:00+10:002023-05-31T11:52:31+10:00Resrouces preview in tag groups may not be up to date<p><small>May <var data-var='date'>30</var>, <var data-var='time'>06:30</var> AEST</small><br><strong>Resolved</strong> - We would like to inform our valued customers about an incident that occurred over the past month, where the resource previews in tag groups may not have been up-to-date. <br /><br />We want to assure you that although the resource preview information was not accurate during this period, the actual rule execution was not impacted with correct resources targeted. <br /><br />We have successfully resolved the issue, and the resource previews for tag groups are now displaying the most recent and accurate matching resources.<br /><br />Resolution:<br />Our technical team worked diligently to investigate and address the underlying cause of this incident. We have implemented the necessary fixes to ensure that the resource previews in tag groups are now updated, providing customers with accurate and up-to-date information.<br /><br />We sincerely apologize for any inconvenience caused during this period. We understand the importance of accurate and timely information, and we have taken measures to prevent similar incidents from occurring in the future. Thank you for your understanding and ongoing support.<br /><br />If you have any further questions or concerns, please do not hesitate to contact us.</p>tag:gorillastack.statuspage.io,2005:Incident/167687012023-04-06T18:45:00+10:002023-04-06T20:40:54+10:00Scheduled rules execution failed<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>18:45</var> AEST</small><br><strong>Resolved</strong> - We would like to inform you that we encountered an incident with the scheduled rules on our platform on April 6, 2023. <br />From approximately 11:00AM AEST, we identified that the scheduled rules had stopped working. Our team promptly investigated the issue, and we were able to identify and fix the root cause of the problem by 18:37 AEST.<br /><br />We deeply apologize for any disruption this may have caused. During the affected timeframe, scheduled rule executions may have resulted in failed execution or no execution. <br />However, we want to assure you that any rules that were missed during this time will be automatically executed on their next scheduled run. <br />Additionally, you may perform manual execution in the meantime to ensure that the scheduled rules are executed properly.<br /><br />We appreciate your patience and understanding as we worked through this issue. Our commitment to providing a reliable and seamless experience for our customers remains unwavering. We will continue to closely monitor the situation to prevent any recurrence of this issue.<br /><br />If you have any further concerns or questions, please do not hesitate to contact our support team. Thank you for your understanding and ongoing support.</p>tag:gorillastack.statuspage.io,2005:Incident/118235312022-10-11T21:00:00+11:002022-10-11T21:15:16+11:00Web Application Unavailable<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>21:00</var> AEDT</small><br><strong>Resolved</strong> - We were able to identify the issue and resolve it. The affected web application has been down for about 4 hours.<br /><br />A recent code change and build change cause the latest deployment to fail.<br /><br />I sincerely apologise for the outage. We're launching a thorough post-mortem and to prevent this in the future.<br /><br />Regards,<br />Ray</p>tag:gorillastack.statuspage.io,2005:Incident/77254152021-08-10T10:30:00+10:002021-08-10T11:52:27+10:00Issues assuming customer roles<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>10:30</var> AEST</small><br><strong>Resolved</strong> - In making some configuration changes, we lost the ability to assume customer's roles which we do for aggregating resources and performing actions.<br /><br />We have backed out the changes and have reached out to customers affected.</p>tag:gorillastack.statuspage.io,2005:Incident/76960782021-08-06T15:11:55+10:002021-08-06T15:11:55+10:00Issues assuming customer roles<p><small>Aug <var data-var='date'> 6</var>, <var data-var='time'>15:11</var> AEST</small><br><strong>Resolved</strong> - In making some configuration changes, we lost the ability to assume customer's roles which we do for aggregating resources and performing actions.<br /><br />We have identified this issue in a deployment and have rolled back the config change.</p>tag:gorillastack.statuspage.io,2005:Incident/64502012021-03-04T17:55:02+11:002021-03-04T17:55:02+11:00Disruption to delivery of notifications<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>17:55</var> AEDT</small><br><strong>Resolved</strong> - We have fixed the underlying issue and will continue to monitor closely</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>16:37</var> AEDT</small><br><strong>Investigating</strong> - We are unable to deliver notifications and are investigating. Rule execution, API and WebApp are all fully operational</p>tag:gorillastack.statuspage.io,2005:Incident/41069652020-05-14T16:10:29+10:002020-05-14T16:10:29+10:00Rule Execution Failing<p><small>May <var data-var='date'>14</var>, <var data-var='time'>16:10</var> AEST</small><br><strong>Resolved</strong> - The incident has been resolved. We apologise again for the inconvenience and thank you for your patience.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>15:04</var> AEST</small><br><strong>Monitoring</strong> - We have rolled back and are monitoring the results. Initial smoke tests indicate that the issue has been resolved. We'll continue to monitor the situation and reach out to users before resolving this issue.<br /><br />We recommend that customers manually execute any rules that failed during this period.<br /><br />We apologise for the inconvenience and will be performing further investigation and a post-mortem.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>14:39</var> AEST</small><br><strong>Update</strong> - We have identified a change that is likely at fault and are rolling back. Another update coming shortly.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>14:27</var> AEST</small><br><strong>Investigating</strong> - We are investigating an issue causing Rule Execution to fail. More to come</p>tag:gorillastack.statuspage.io,2005:Incident/22770172019-03-22T10:15:15+11:002019-03-22T10:15:15+11:00Potential DNS Issues<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>10:15</var> AEDT</small><br><strong>Resolved</strong> - It appears that this was only an issue for customers without HTTPS redirect by default enabled. We implemented a redirect to https for all customers attempting to reach GorillaStack over port 80.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>09:38</var> AEDT</small><br><strong>Investigating</strong> - Some (but not all) users are unable to resolve their way to v2.gorillastack.com<br /><br />We're investigating this issue.<br /><br />All core scheduling and event management are working and are unaffected</p>tag:gorillastack.statuspage.io,2005:Incident/22550812019-03-11T12:30:32+11:002019-03-12T14:33:43+11:00Some scheduling impacted<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>12:30</var> AEDT</small><br><strong>Resolved</strong> - Over the weekend, GorillaStack upgraded some core components of our architecture. Unfortunately, after this update, a bug caused us to miss some scheduled rule executions.
<br />
<br />All affected users were contacted directly with a list of their rules impacted and with information options to run any missing rules manually.</p>tag:gorillastack.statuspage.io,2005:Incident/19920742018-10-25T11:47:44+11:002018-10-25T11:48:47+11:00Web Application Unavailable<p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:47</var> AEDT</small><br><strong>Resolved</strong> - We were able to identify the issue and resolve it.<br /><br />One of our databases, had its disk fill up. Once identified, we addressed the issue.<br /><br />We initially reported that only the Webapp was affected. Now that the cause is known, it is clear that in fact, our API, Scheduling and Rule Execution were affected too.<br /><br />I sincerely apologise for the outage. We're launching a thorough post-mortem and will report back with our plans to prevent this in the future.<br /><br />Regards,<br />Elliott</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>11:09</var> AEDT</small><br><strong>Investigating</strong> - We appear to have issues with the availability of our web application.<br /><br />All rules and scheduling remain unaffected.<br /><br />More to come</p>tag:gorillastack.statuspage.io,2005:Incident/19776852018-10-18T18:51:57+11:002018-10-19T14:44:42+11:00Schedule Evaluation<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:51</var> AEDT</small><br><strong>Resolved</strong> - We're continuing to monitor and evaluate options around more proactive scheduler health checks, but the scheduler is fully operational and continuing to mark it as having "degraded performance" is inaccurate.<br /><br />We apologise for any impact!</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>16:10</var> AEDT</small><br><strong>Monitoring</strong> - The scheduler has been reloaded, resolving the current instance of this issue. We are now doing deeper root cause analysis to understand what caused the scheduler to evaluate incorrectly for a handful of rules.</p>tag:gorillastack.statuspage.io,2005:Incident/18790742018-08-27T16:30:36+10:002018-08-27T16:30:36+10:00Web Application Partial Outage<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>16:30</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved. We have finished root cause analysis and this will be resolved with the next major update (which we are shipping later this week).<br /><br />To re-iterate: there was no impact to scheduling, trigger detection, action execution, notifications or any other systems. It was isolated to our web application.<br /><br />Thank you for your patience.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>15:40</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the situation.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>15:24</var> AEST</small><br><strong>Identified</strong> - GorillaStack is experiencing some downtime on the web application.<br /><br />All scheduling and other components are working normally.<br /><br />We'll keep you up to date with what's going on.</p>tag:gorillastack.statuspage.io,2005:Incident/18024992018-07-06T19:42:48+10:002018-07-07T00:20:03+10:00Issues with Web Application Load Balancing<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>19:42</var> AEST</small><br><strong>Resolved</strong> - We were able to find the issue with a database lock that affected some requests to the web application.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>19:06</var> AEST</small><br><strong>Investigating</strong> - Some requests to the web application are resulting in 500s. We're investigating and will be back with more information shortly</p>