• Replies 59 replies
  • Subscribers 343 subscribers
  • Views 2127 views
  • Users 0 members are here
Related

Contingency Plans for RAMP Customers?

Brandi Sellers
$organization

As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime.  We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare. 

Parents
  • $organization
    If you “realize that any system, hosted locally or remotely, will go down and cause downtime”, then you don’t really expect 100% uptime.  You can’t have it both ways.
     
    I think I understand what you’re saying though.  100% uptime is the goal.  Even if we acknowledge that it’s not actually achievable, it’s still the goal we shoot for.
     
    However, shooting for an unachievable goal isn’t a wise business choice.  Instead, business leaders should set aside the time and resources necessary to determine how much downtime is actually acceptable.  In order to do that you must also determine how much that downtime will cost. 
     
    I know it’s hard to do.  We’ve had previous senior managers (no longer here!) at CSO who have repeatedly stated that NO downtime is acceptable.  Yet, they also refused to support expending additional resources to help build necessary redundancy to lower downtime.
     
    Reality is what it is.  Know how much it costs to be down.  Know how much you can actually afford to lose to downtime on an annualized basis.  Put that on a rolling timeline, and then see if your service provider is falling within those guidelines.   Then judge.
     
    Having been self-hosted for a long time I understand how frustrating it can be to have a single outage that trashes that goal.  It’s especially frustrating when it hits one of those single points of failure where redundancy was simply not cost justified. 
     
    I’m I have users here who would say that they would love to have our systems hosted by someone else because of the unplanned downtime we’ve experienced.  Most years we experience less than 4 hours of both planned and unplanned downtime.  But… there have been times when we’ve experienced the 12 hour outage.  One of those rare events that is not likely to happen again for another 15 years.  Is it worth the hundreds of thousands of dollars it would cost over that time to build around such an event?  Nope.  It just isn’t. 
     
    Down time is a fact of life.  Don’t fear it.  Accept it.  But, plan for it.  Measure it.  And hold your providers accountable if they fail to meet their uptime goals as stated in their SLA.
     
    Since we’re not a RAMP client, I am curious.  Are the RAMP SLA’s being met?
     
     
  • Former Member
    Former Member $organization in reply to Dan Spees

    Being an attraction and a museum our experience with these frequent outages are much different. We have a mobile (not fixed location) box office where people buy admission when they arrive. We more often than not have a line of people waiting to enter. 85% of our tickets are still sold onsite. Tessitura going down cripples our entire entrance experience. It requires our staff to be tethered to a wired credit card machine in a fixed location. We have to manually track tickets sold to enter in the system later creating more work for box office staff. It creates issues for the finance team to reconcile payments after the fact as they can not be run through tessitura after the fact. Etc. When we have a busy summer day with eight thousand people visiting us, and our ticketing system goes down for an hour, any contingency plan is a lot of extra work no matter how you look at it.

Reply
  • Former Member
    Former Member $organization in reply to Dan Spees

    Being an attraction and a museum our experience with these frequent outages are much different. We have a mobile (not fixed location) box office where people buy admission when they arrive. We more often than not have a line of people waiting to enter. 85% of our tickets are still sold onsite. Tessitura going down cripples our entire entrance experience. It requires our staff to be tethered to a wired credit card machine in a fixed location. We have to manually track tickets sold to enter in the system later creating more work for box office staff. It creates issues for the finance team to reconcile payments after the fact as they can not be run through tessitura after the fact. Etc. When we have a busy summer day with eight thousand people visiting us, and our ticketing system goes down for an hour, any contingency plan is a lot of extra work no matter how you look at it.

Children
No Data