• Replies 59 replies
  • Subscribers 343 subscribers
  • Views 2127 views
  • Users 0 members are here
Related

Contingency Plans for RAMP Customers?

Brandi Sellers
$organization

As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime.  We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare. 

Parents
  • $organization
    If you “realize that any system, hosted locally or remotely, will go down and cause downtime”, then you don’t really expect 100% uptime.  You can’t have it both ways.
     
    I think I understand what you’re saying though.  100% uptime is the goal.  Even if we acknowledge that it’s not actually achievable, it’s still the goal we shoot for.
     
    However, shooting for an unachievable goal isn’t a wise business choice.  Instead, business leaders should set aside the time and resources necessary to determine how much downtime is actually acceptable.  In order to do that you must also determine how much that downtime will cost. 
     
    I know it’s hard to do.  We’ve had previous senior managers (no longer here!) at CSO who have repeatedly stated that NO downtime is acceptable.  Yet, they also refused to support expending additional resources to help build necessary redundancy to lower downtime.
     
    Reality is what it is.  Know how much it costs to be down.  Know how much you can actually afford to lose to downtime on an annualized basis.  Put that on a rolling timeline, and then see if your service provider is falling within those guidelines.   Then judge.
     
    Having been self-hosted for a long time I understand how frustrating it can be to have a single outage that trashes that goal.  It’s especially frustrating when it hits one of those single points of failure where redundancy was simply not cost justified. 
     
    I’m I have users here who would say that they would love to have our systems hosted by someone else because of the unplanned downtime we’ve experienced.  Most years we experience less than 4 hours of both planned and unplanned downtime.  But… there have been times when we’ve experienced the 12 hour outage.  One of those rare events that is not likely to happen again for another 15 years.  Is it worth the hundreds of thousands of dollars it would cost over that time to build around such an event?  Nope.  It just isn’t. 
     
    Down time is a fact of life.  Don’t fear it.  Accept it.  But, plan for it.  Measure it.  And hold your providers accountable if they fail to meet their uptime goals as stated in their SLA.
     
    Since we’re not a RAMP client, I am curious.  Are the RAMP SLA’s being met?
     
     
  • $organization in reply to Dan Spees

    I want to jump in here following Dan's lead. We have been on Tessitura since v3 (which I installed on local SQL servers 13 years ago). We moved to RAMP about 7 years ago. Yes, there has been downtime, and yes, it has generally occurred at the worst possible moment. Any technology will have outages. Every client, RAMP or not, must have documented and practiced plans to continue operations while systems are down.

  • $organization in reply to Randall Mitchell

    In addition, we might want to take into account historical context and the pledge that has been stated to improve going forward.  Our organization has also been on RAMP for 7+ years (in fact, since our inception).  There was some downtime in the first year or so, but following that, RAMP up-time was fairly impeccable (for us at least) for about 5 straight years, and only in this last year has it started to become more of an issue again.

    Yes, that does not solve any issues today and can still grab us at some inconvenient moments, but it does help to give hope to and retain confidence in the hosted service.  Prior to this last year, I had no complaints about the RAMP service, and again, while it does nothing for today's issues, the fact that these issues are being taken seriously gives me confidence.

    And of course, we shall continue to plan for any eventualities that we can.  And that is why we all love our Box Offices so much.

  • $organization in reply to John A. Moskal II

    But I ask you John...how can we as RAMP member plan for continued service if RAMP prohibits us from say, having a local instance of Tessitura to bounce to should we go down? RAMP will not allow any outside connections (we tried to connect Azure for better cloud reporting and were flat out denied that option). Tessitura is forcing us to rely solely on them and their word and their IT team. You know I'm a huge Tessitura advocate. I've been using for years. I've been with RAMP and on-prem. I speak with absolutely certainty, I NEVER experienced this level of issues when with on-prem organizations. Our downtime was minimal. I intend to hold Tessitura accountable to their word on improving our experience. Two outages in 30 days is unacceptable....especially as a newly implemented organization.

Reply
  • $organization in reply to John A. Moskal II

    But I ask you John...how can we as RAMP member plan for continued service if RAMP prohibits us from say, having a local instance of Tessitura to bounce to should we go down? RAMP will not allow any outside connections (we tried to connect Azure for better cloud reporting and were flat out denied that option). Tessitura is forcing us to rely solely on them and their word and their IT team. You know I'm a huge Tessitura advocate. I've been using for years. I've been with RAMP and on-prem. I speak with absolutely certainty, I NEVER experienced this level of issues when with on-prem organizations. Our downtime was minimal. I intend to hold Tessitura accountable to their word on improving our experience. Two outages in 30 days is unacceptable....especially as a newly implemented organization.

Children
No Data