• Replies 59 replies
  • Subscribers 343 subscribers
  • Views 2127 views
  • Users 0 members are here
Related

Contingency Plans for RAMP Customers?

Brandi Sellers
$organization

As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime.  We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare. 

Parents
  • $organization
    If you “realize that any system, hosted locally or remotely, will go down and cause downtime”, then you don’t really expect 100% uptime.  You can’t have it both ways.
     
    I think I understand what you’re saying though.  100% uptime is the goal.  Even if we acknowledge that it’s not actually achievable, it’s still the goal we shoot for.
     
    However, shooting for an unachievable goal isn’t a wise business choice.  Instead, business leaders should set aside the time and resources necessary to determine how much downtime is actually acceptable.  In order to do that you must also determine how much that downtime will cost. 
     
    I know it’s hard to do.  We’ve had previous senior managers (no longer here!) at CSO who have repeatedly stated that NO downtime is acceptable.  Yet, they also refused to support expending additional resources to help build necessary redundancy to lower downtime.
     
    Reality is what it is.  Know how much it costs to be down.  Know how much you can actually afford to lose to downtime on an annualized basis.  Put that on a rolling timeline, and then see if your service provider is falling within those guidelines.   Then judge.
     
    Having been self-hosted for a long time I understand how frustrating it can be to have a single outage that trashes that goal.  It’s especially frustrating when it hits one of those single points of failure where redundancy was simply not cost justified. 
     
    I’m I have users here who would say that they would love to have our systems hosted by someone else because of the unplanned downtime we’ve experienced.  Most years we experience less than 4 hours of both planned and unplanned downtime.  But… there have been times when we’ve experienced the 12 hour outage.  One of those rare events that is not likely to happen again for another 15 years.  Is it worth the hundreds of thousands of dollars it would cost over that time to build around such an event?  Nope.  It just isn’t. 
     
    Down time is a fact of life.  Don’t fear it.  Accept it.  But, plan for it.  Measure it.  And hold your providers accountable if they fail to meet their uptime goals as stated in their SLA.
     
    Since we’re not a RAMP client, I am curious.  Are the RAMP SLA’s being met?
     
     
  • $organization in reply to Dan Spees

    Our TNMP site is up and would be nice if we can easily redirect TNEW to TNMP ... for situations like today. 

  • $organization in reply to Sabina Spilkin

    That's ironic.  Of course, TNMP is going away soon...

Reply Children