• Replies 59 replies
  • Subscribers 343 subscribers
  • Views 2111 views
  • Users 0 members are here
Related

Contingency Plans for RAMP Customers?

Brandi Sellers
$organization

As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime.  We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare. 

Parents
  • $organization
    If you “realize that any system, hosted locally or remotely, will go down and cause downtime”, then you don’t really expect 100% uptime.  You can’t have it both ways.
     
    I think I understand what you’re saying though.  100% uptime is the goal.  Even if we acknowledge that it’s not actually achievable, it’s still the goal we shoot for.
     
    However, shooting for an unachievable goal isn’t a wise business choice.  Instead, business leaders should set aside the time and resources necessary to determine how much downtime is actually acceptable.  In order to do that you must also determine how much that downtime will cost. 
     
    I know it’s hard to do.  We’ve had previous senior managers (no longer here!) at CSO who have repeatedly stated that NO downtime is acceptable.  Yet, they also refused to support expending additional resources to help build necessary redundancy to lower downtime.
     
    Reality is what it is.  Know how much it costs to be down.  Know how much you can actually afford to lose to downtime on an annualized basis.  Put that on a rolling timeline, and then see if your service provider is falling within those guidelines.   Then judge.
     
    Having been self-hosted for a long time I understand how frustrating it can be to have a single outage that trashes that goal.  It’s especially frustrating when it hits one of those single points of failure where redundancy was simply not cost justified. 
     
    I’m I have users here who would say that they would love to have our systems hosted by someone else because of the unplanned downtime we’ve experienced.  Most years we experience less than 4 hours of both planned and unplanned downtime.  But… there have been times when we’ve experienced the 12 hour outage.  One of those rare events that is not likely to happen again for another 15 years.  Is it worth the hundreds of thousands of dollars it would cost over that time to build around such an event?  Nope.  It just isn’t. 
     
    Down time is a fact of life.  Don’t fear it.  Accept it.  But, plan for it.  Measure it.  And hold your providers accountable if they fail to meet their uptime goals as stated in their SLA.
     
    Since we’re not a RAMP client, I am curious.  Are the RAMP SLA’s being met?
     
     
Reply
  • $organization
    If you “realize that any system, hosted locally or remotely, will go down and cause downtime”, then you don’t really expect 100% uptime.  You can’t have it both ways.
     
    I think I understand what you’re saying though.  100% uptime is the goal.  Even if we acknowledge that it’s not actually achievable, it’s still the goal we shoot for.
     
    However, shooting for an unachievable goal isn’t a wise business choice.  Instead, business leaders should set aside the time and resources necessary to determine how much downtime is actually acceptable.  In order to do that you must also determine how much that downtime will cost. 
     
    I know it’s hard to do.  We’ve had previous senior managers (no longer here!) at CSO who have repeatedly stated that NO downtime is acceptable.  Yet, they also refused to support expending additional resources to help build necessary redundancy to lower downtime.
     
    Reality is what it is.  Know how much it costs to be down.  Know how much you can actually afford to lose to downtime on an annualized basis.  Put that on a rolling timeline, and then see if your service provider is falling within those guidelines.   Then judge.
     
    Having been self-hosted for a long time I understand how frustrating it can be to have a single outage that trashes that goal.  It’s especially frustrating when it hits one of those single points of failure where redundancy was simply not cost justified. 
     
    I’m I have users here who would say that they would love to have our systems hosted by someone else because of the unplanned downtime we’ve experienced.  Most years we experience less than 4 hours of both planned and unplanned downtime.  But… there have been times when we’ve experienced the 12 hour outage.  One of those rare events that is not likely to happen again for another 15 years.  Is it worth the hundreds of thousands of dollars it would cost over that time to build around such an event?  Nope.  It just isn’t. 
     
    Down time is a fact of life.  Don’t fear it.  Accept it.  But, plan for it.  Measure it.  And hold your providers accountable if they fail to meet their uptime goals as stated in their SLA.
     
    Since we’re not a RAMP client, I am curious.  Are the RAMP SLA’s being met?
     
     
Children
  • $organization in reply to Dan Spees

    I want to jump in here following Dan's lead. We have been on Tessitura since v3 (which I installed on local SQL servers 13 years ago). We moved to RAMP about 7 years ago. Yes, there has been downtime, and yes, it has generally occurred at the worst possible moment. Any technology will have outages. Every client, RAMP or not, must have documented and practiced plans to continue operations while systems are down.

  • $organization in reply to Randall Mitchell

    In addition, we might want to take into account historical context and the pledge that has been stated to improve going forward.  Our organization has also been on RAMP for 7+ years (in fact, since our inception).  There was some downtime in the first year or so, but following that, RAMP up-time was fairly impeccable (for us at least) for about 5 straight years, and only in this last year has it started to become more of an issue again.

    Yes, that does not solve any issues today and can still grab us at some inconvenient moments, but it does help to give hope to and retain confidence in the hosted service.  Prior to this last year, I had no complaints about the RAMP service, and again, while it does nothing for today's issues, the fact that these issues are being taken seriously gives me confidence.

    And of course, we shall continue to plan for any eventualities that we can.  And that is why we all love our Box Offices so much.

  • Former Member
    Former Member $organization in reply to Dan Spees

    Being an attraction and a museum our experience with these frequent outages are much different. We have a mobile (not fixed location) box office where people buy admission when they arrive. We more often than not have a line of people waiting to enter. 85% of our tickets are still sold onsite. Tessitura going down cripples our entire entrance experience. It requires our staff to be tethered to a wired credit card machine in a fixed location. We have to manually track tickets sold to enter in the system later creating more work for box office staff. It creates issues for the finance team to reconcile payments after the fact as they can not be run through tessitura after the fact. Etc. When we have a busy summer day with eight thousand people visiting us, and our ticketing system goes down for an hour, any contingency plan is a lot of extra work no matter how you look at it.

  • $organization in reply to Dan Spees

    Disruptions in service are always frustrating, not only for the network and our individual organizations, but, most importantly, for our patrons.  And, of course, they never occur at "convenient" times.  At the Kimbell Art Museum, we have implemented two back-up systems (wired and wireless) which enable us to continue selling on-site admissions, audios, and memberships during outages.  

    We appreciate the network's continued efforts to resolve and minimize these outages, but we certainly share everyone's concerns about coping with the realities of service disruptions. 

  • $organization in reply to Dan Spees

    Our TNMP site is up and would be nice if we can easily redirect TNEW to TNMP ... for situations like today. 

  • $organization in reply to Sabina Spilkin

    That's ironic.  Of course, TNMP is going away soon...

  • $organization in reply to John A. Moskal II

    But I ask you John...how can we as RAMP member plan for continued service if RAMP prohibits us from say, having a local instance of Tessitura to bounce to should we go down? RAMP will not allow any outside connections (we tried to connect Azure for better cloud reporting and were flat out denied that option). Tessitura is forcing us to rely solely on them and their word and their IT team. You know I'm a huge Tessitura advocate. I've been using for years. I've been with RAMP and on-prem. I speak with absolutely certainty, I NEVER experienced this level of issues when with on-prem organizations. Our downtime was minimal. I intend to hold Tessitura accountable to their word on improving our experience. Two outages in 30 days is unacceptable....especially as a newly implemented organization.

  • $organization in reply to Christopher Cuhel

    Yes and no.  Soon is probably hyperbole, and TNMP apps are expected to continue, but in this case Sabina was referring to the separate TNMP mobile web site, and that will become obsolete when TNEW v7 finishes rolling out, although I don't know off the top of my head what the schedule for support is after that.