As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime. We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare.
I echo this sentiment. RAMP prohibits us, the clients, from having ANY sort of contingency for when RAMP is down. We are NOT allowed to say, host a backup locally should RAMP go down. We are VERY limited by RAMP to our access (even VERY seasoned users). The silence and lack of response from the RAMP team is simply unacceptable on this front. We are losing money because of this. And with Tessitura going to a completely web based product, how is Tessitura going to ensure that RAMP and TNEW clients don't experience this level of service disruption on a regular basis. I recently saw a posting from Jack about customer service. We are the customers here and the level of service we are receiving is sub-par at best.
Christopher Cuhel said:And with Tessitura going to a completely web based product, how is Tessitura going to ensure that RAMP and TNEW clients don't experience this level of service disruption on a regular basis
My sense is that many of the outages (and performance issues) revolve around the complex Jenga-like stack of services RAMP has to maintain to support an application that still uses PowerBuilder for core elements as a cloud, so a proper web services should involve a radical simplification of architecture and fewer of these "issue to host machines connect to storage devices" type failures.
I sure hope so Gawain. As I've said....on-prem organizations rarely, if ever, experience this level of outage.
I'd like to jump in and echo what Gawain has said.
Having self hosted our Tessitura environment for many years, both on premise, and in colocation environments, we've had plenty of opportunities to sit down with engineers and discuss what it would take to eliminate all single points of failure and build in redundancy wherever possible.
At the end of the day it's hard to accomplish these goals at costs that are justifiable because of the complexity of the end-to-end Tessitura environment. Building truly redundant transactional data environments involves much more than backup and restore.
I can't imagine that simplification of the overall Tessitura environment is not a top level goal for the Network. Weather they talk much about it or not.
In the meantime, I like having the power to take action and do what we feel needs to be done to keep our operation running 24x7. But despite all the planning and work we do, things happen that are out of our control. We have our single points of failure and sometimes they fail. Over the course of a year we typically run around 3 nines for our uptime. Being up 99.9% of the time means that we'll experience unplanned outages for a total duration of about 9 hours a year. That sounds pretty reasonable until 4 of those hours happen in the 3 hours before and the 1 hour after the start of a huge on sale event.
No one likes to deal with outages such as this. But, failing to plan for them is the same as planning to fail. You will have unexpected downtime where all or part of your ability to conduct transactions unavailable. It doesn't matter if you use a Managed Service such as RAMP or self host. It will happen. Plan for it.
I completely hear what you are saying and push back with this simple statement: what is a small organization with a limited staff and person thrown into an IT role who is not an IT person supposed to do? How are they supposed to plan for fail over when they are just trying to keep their organization running? They rely on uptime and Tessitura and RAMP to ensure their up time. Do I expect 100% uptime all the time...yes. Do I realize that any system, hosted locally or remotely, will go down and cause downtime? Yes. I have worked for both styles of hosting for Tessitura. RAMP hosting has, by far, been the MOST frustrating.
Christopher Cuhel said:on-prem organizations rarely, if ever, experience this level of outage
I don't have hard numbers, but I feel like RAMP outages are more frequent than we experienced self hosting. We also get a lot of connectivity "blips" where we'll lose connection as an organization for a few minutes before we start reconnecting. But on the other hand, the outages are usually much shorter. When our server room flooded, for instance, we weren't back online in an hour...
But I wouldn't be surprised if the RAMP outages were more frequent than someone with a standard installation because of all the additional "cloud" and virtualization pieces. And I haven't been keeping track, but I feel like nine times out of ten, those are the fail points when there is an outage, rather than hard drive failures or routers dying.