tag:status.reelcrafter.com,2005:/historyReelCrafter Status - Incident History2024-03-29T02:51:56-05:00ReelCraftertag:status.reelcrafter.com,2005:Incident/200678262024-02-25T01:00:57-06:002024-02-25T01:00:57-06:00Scheduled database upgrade - full outage<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>01:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>18:21</var> CST</small><br><strong>Scheduled</strong> - We'll be performing a required database upgrade on February 24th at 10PM Pacific Time. We expect to be down for about 45-60 minutes. Both the ReelCrafter app and reels will be inaccessible during this time.</p>tag:status.reelcrafter.com,2005:Incident/182622462023-08-25T12:54:27-05:002023-08-25T12:54:27-05:00Audio transcoding/processing not working<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>12:54</var> CDT</small><br><strong>Resolved</strong> - The audio transcode feature has been fixed. Thank you for your patience!</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>11:51</var> CDT</small><br><strong>Identified</strong> - We've identified the issue and are working on a fix. Stay tuned.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>11:40</var> CDT</small><br><strong>Investigating</strong> - We're aware that processing uploaded audio files is not currently working. Hang tight, we're working on identifying the issue and working toward a fix ASAP.</p>tag:status.reelcrafter.com,2005:Incident/177336312023-07-01T23:00:39-05:002023-07-01T23:00:39-05:00Reels and embeds maintenance downtime<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>23:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>22:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:34</var> CDT</small><br><strong>Scheduled</strong> - On July 1st at 10pm Central / 8pm Pacific (July 2nd at 5am Central European Time), ReelCrafter reels and embedded players will be down for no more than an hour, for a maintenance update. We actually expect the downtime to last no more than 15-20 minutes, but are allotting the full hour in case of unforeseen issues. The ReelCrafter application will not be affected.<br /><br />You can subscribe to updates at https://status.reelcrafter.com to keep up to date on this update, as well as any future outages.</p>tag:status.reelcrafter.com,2005:Incident/176558002023-06-24T21:05:19-05:002023-06-24T21:05:19-05:00Reels and embeds maintenance downtime<p><small>Jun <var data-var='date'>24</var>, <var data-var='time'>21:05</var> CDT</small><br><strong>Completed</strong> - We've abandoned tonight's scheduled maintenance and will reschedule for next weekend. We discovered some last-minute bugs that prevented us from moving forward.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>12:04</var> CDT</small><br><strong>Scheduled</strong> - On June 24th at 10pm Central / 8pm Pacific (June 25th at 5am Central European Time), ReelCrafter reels and embedded players will be down for no more than an hour, for a maintenance update. We actually expect the downtime to last no more than 15-20 minutes, but are allotting the full hour in case of unforeseen issues. The ReelCrafter application will not be affected.<br /><br />You can subscribe to updates at https://status.reelcrafter.com to keep up to date on this update, as well as any future outages.</p>tag:status.reelcrafter.com,2005:Incident/175649562023-06-13T16:34:25-05:002023-06-13T16:34:25-05:00Tracking is non-functional<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:34</var> CDT</small><br><strong>Resolved</strong> - Tracking should be working again! Any events that were not appearing will have been processed and will show up on your dashboard.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>16:03</var> CDT</small><br><strong>Monitoring</strong> - In the aftermath of the recent Amazon Web Services outage, it looks like our event tracking is not working properly. Tracking events *are* being captured in our queue, which is good news (they won't be lost), but they're not being processed, and so they won't show up on your dashboard just yet. To spare your inbox from updates, we'll send one more update once the "clog" has cleared and tracking events are flowing in again.</p>tag:status.reelcrafter.com,2005:Incident/175626142023-06-13T15:45:39-05:002023-06-13T15:45:39-05:00All services down<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:45</var> CDT</small><br><strong>Resolved</strong> - We're in the clear! We haven't heard official word from Amazon Web Services as far as status goes, but it looks like reels and embeds are loading consistently now. We apologize for the frustrating experience. We'll continue to keep a close eye on things to make sure they're stable, and update you if things change.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:42</var> CDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>15:39</var> CDT</small><br><strong>Update</strong> - As of 1:38pm Pacific Time, Amazon Web Services announced that they're starting to see improvement in their services. We actually were able to load up a few test reels, but then they were down again. Hang tight, it looks like we're nearing the end of the outage soon.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:32</var> CDT</small><br><strong>Monitoring</strong> - Amazon Web Services has identified the root cause of the issue and are working on a fix. There's no ETA for that yet, but we're monitoring the situation closely and will keep you posted.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>14:01</var> CDT</small><br><strong>Identified</strong> - Amazon Web Services (our service provider) is experiencing a major outage, which is affecting the ReelCrafter app, and all reels and embedded players. We will keep you posted on updates/progress as Amazon investigates and works on resolving its issues.</p>tag:status.reelcrafter.com,2005:Incident/172431402023-05-14T00:55:36-05:002023-05-14T00:55:36-05:00Database maintenance - full app downtime<p><small>May <var data-var='date'>14</var>, <var data-var='time'>00:55</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>00:36</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>14</var>, <var data-var='time'>00:33</var> CDT</small><br><strong>Scheduled</strong> - We will be performing a database upgrade. It shouldn't take more than one hour to complete.</p>tag:status.reelcrafter.com,2005:Incident/163502622023-03-05T23:13:15-06:002023-03-05T23:13:15-06:00Server upgrade failure<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>23:13</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>23:10</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>23:03</var> CST</small><br><strong>Identified</strong> - We've had a failure occur while upgrading our server. We're doing a rollback at the moment to restore functionality again.</p>tag:status.reelcrafter.com,2005:Incident/163001132023-02-27T19:56:09-06:002023-02-27T19:56:09-06:00Web application non-functional<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>19:56</var> CST</small><br><strong>Resolved</strong> - The issue's been resolved. We apologize for the hiccup!</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>19:47</var> CST</small><br><strong>Identified</strong> - The web application is currently down (this does NOT affect reels - your reels are fully functional). We've identified the issue and are applying a fix that should be live in the next 10 minutes or so.</p>tag:status.reelcrafter.com,2005:Incident/162699782023-02-24T13:45:41-06:002023-02-24T13:45:41-06:00Media uploading/transcoding inoperable<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>13:45</var> CST</small><br><strong>Resolved</strong> - The issue has been resolved. Thank you for your patience, and we apologize for the hassle.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:55</var> CST</small><br><strong>Identified</strong> - We've identified the source of the transcode failure issue and are working on a solution.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:09</var> CST</small><br><strong>Investigating</strong> - Uploading media will cause it to be stuck transcoding indefinitely. We're looking into the issue right now to determine the root cause, and will provide updates when we know more.</p>tag:status.reelcrafter.com,2005:Incident/127189292022-11-04T13:14:12-05:002022-11-04T13:14:12-05:00Intermittent downtime with reels and embeds<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>13:14</var> CDT</small><br><strong>Resolved</strong> - The issue seems to have resolved itself. We will keep a close eye on this today to make sure the problem doesn't return.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>13:00</var> CDT</small><br><strong>Investigating</strong> - Reels and embedded players are experiencing intermittent downtime. This is due to an issue with our service provider (Amazon), and we're investigating it right now. This is our top priority.</p>tag:status.reelcrafter.com,2005:Incident/110903692022-09-09T12:52:13-05:002022-09-09T12:52:13-05:00Sporadic errors in ReelCrafter app<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>12:52</var> CDT</small><br><strong>Resolved</strong> - It looks like our cloud provider has solved their server issues, and we're fully operational again.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>12:20</var> CDT</small><br><strong>Update</strong> - The last update from our cloud provider was at 9:50am PDT. They're working on their server issue but don't have an ETA at this time. We'll continue to keep tabs on this and update you again in the next couple of hours or so. To reiterate: ReelCrafter seems to be mostly working, with occasional errors.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>11:30</var> CDT</small><br><strong>Monitoring</strong> - Our cloud provider is experiencing a partial outage, which is affecting the ReelCrafter app sporadically. The app may function mostly ok for you, but you may run into the occasional error. To our knowledge, reels are not affected.<br /><br />We're keeping an eye on updates from our cloud provider for updates.</p>tag:status.reelcrafter.com,2005:Incident/104524922022-07-01T14:52:27-05:002022-07-01T14:52:27-05:00Reel downloads not working<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>14:52</var> CDT</small><br><strong>Resolved</strong> - We've identified and resolved the reel download issue.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>14:31</var> CDT</small><br><strong>Investigating</strong> - Reel downloads are currently not working. This is a result of attempting to roll out an update that would fix extremely long reels from not downloading at all.<br /><br />We're investigating and hope to have things up and running again shortly.</p>tag:status.reelcrafter.com,2005:Incident/97427892022-04-08T23:30:37-05:002022-04-08T23:30:39-05:00Updating server components<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>23:30</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>22:31</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>18:10</var> CDT</small><br><strong>Scheduled</strong> - During this brief maintenance period, we will be updating some under-the-hood components to their latest versions. This is just a routine task to ensure the software that powers ReelCrafter remains current and serviceable. We don't expect any outages, but some very brief downtime is possible.</p>tag:status.reelcrafter.com,2005:Incident/96537472022-03-28T21:49:07-05:002022-03-28T21:49:07-05:00Editing media in a reel not working<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>21:49</var> CDT</small><br><strong>Resolved</strong> - The media block issue has been resolved.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>21:40</var> CDT</small><br><strong>Identified</strong> - We've identified the root cause and are working on a fix.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>20:46</var> CDT</small><br><strong>Investigating</strong> - We're looking into an issue where editing a reel appears to clear out all the playlists and media inside it. Your reels are NOT empty, it's just not loading up the media properly. IMPORTANT NOTE: do not touch the media bock when editing your reel, otherwise when you publish it, the media *will* be gone.<br /><br />If you happen to lose the media in your reel, please contact support and we can revert to an earlier version of your reel.<br /><br />Again, in summary: your reels are fine, editing them is ok too, just DO NOT touch the media block until this issue is resolved.</p>tag:status.reelcrafter.com,2005:Incident/91324112022-01-20T19:30:30-06:002022-01-20T19:30:30-06:00Server infrastructure changes<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>19:30</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>18:46</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>17:52</var> CST</small><br><strong>Scheduled</strong> - ReelCrafter v2 will be undergoing some minor infrastructure changes. We expect the maintenance to last no more than 30 minutes and will not likely impact service. This was scheduled for late tonight, but we've had to move it forward to address some issues with migration from v1.</p>tag:status.reelcrafter.com,2005:Incident/90217312022-01-06T20:04:12-06:002022-01-06T20:04:12-06:00[v2] Audio not loading on reels<p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>20:04</var> CST</small><br><strong>Resolved</strong> - The issue has been fixed. You can resume sending out reels. Apologies for the inconvenience!</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>19:50</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we're rolling it out now.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>19:47</var> CST</small><br><strong>Investigating</strong> - In v2, audio is currently not playing on reels. Please refrain from sending reels until this incident is resolved.</p>tag:status.reelcrafter.com,2005:Incident/87967572021-12-08T19:28:38-06:002021-12-08T19:28:39-06:00Full outage<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:28</var> CST</small><br><strong>Resolved</strong> - We're back up and running again - sorry for the inconvenience.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>18:24</var> CST</small><br><strong>Investigating</strong> - We're experiencing a major outage and believe it might be an aftershock of the recent Amazon Web Services outage. We're investigating now and will have an update soon.</p>tag:status.reelcrafter.com,2005:Incident/87846492021-12-07T17:53:18-06:002021-12-07T17:53:19-06:00v1 Web Embeds are down<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>17:53</var> CST</small><br><strong>Resolved</strong> - Web embeds appear to be working normally. If you notice any further issues, please let us know!</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:09</var> CST</small><br><strong>Update</strong> - It looks like web embeds are intermittently functional. Over the past few hours, they're up and down, so it appears things may slowly be coming back online on Amazon's end. We'll continue to monitor the situation and provide updates.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:45</var> CST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>12:45</var> CST</small><br><strong>Monitoring</strong> - Due to an Amazon Web Services outage, ReelCrafter v1 web embeds are non-functional. We'll keep monitoring the situation as things develop, and update this incident. We apologize for the inconvenience.</p>tag:status.reelcrafter.com,2005:Incident/87814782021-12-07T16:20:15-06:002021-12-07T16:20:18-06:00Major Amazon outage affecting ReelCrafter v2<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>16:20</var> CST</small><br><strong>Resolved</strong> - ReelCrafter v2 beta appears to be in full working order. If anyone experiences issues again, please let us know via support.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:44</var> CST</small><br><strong>Update</strong> - The v2 web application appears to be fully operational.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:35</var> CST</small><br><strong>Update</strong> - ReelCrafter v2 beta is mostly back online, though reels and embedded players are down and real-time functionality is not working.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>09:57</var> CST</small><br><strong>Monitoring</strong> - There appears to be a major outage with AWS (Amazon Web Services), which is affecting ReelCrafter v2. ReelCrafter v1 is apparently unaffected so far.<br /><br />We're keeping tabs on updates from Amazon and will keep everyone posted.</p>tag:status.reelcrafter.com,2005:Incident/87816102021-12-07T14:08:50-06:002021-12-07T14:08:51-06:00Track uploads failing<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:08</var> CST</small><br><strong>Resolved</strong> - Audio transcoding appears to be working, so feel free to upload tracks! Web embeds are still down, and we're monitoring that situation closely.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:03</var> CST</small><br><strong>Monitoring</strong> - In ReelCrafter v1, track uploads are getting stuck and won't transcode. This is due to a major Amazon Web Services outage that has affected a large portion of the Internet.<br /><br />We're keeping tabs on this and will update our status page as we know more, or when the issue is resolved. We're sorry, and thank you for your patience!</p>