As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime. We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare.
The amount of RAMP outages recently is making our administration question the decision to move to Tessitura. I am incredibly curious what the Network is doing on the back end to decrease the number of times that these types of issues occur?
There is no more important priority to the Tessitura Network than the consistent and reliable uptime and access to our services. The disruptions are unacceptable and we have put all available resources toward the continued hardening of our hosted environments. We greatly regret the disruptions. The data is secure and despite the number of outages, the technology is robust and top tier.
As far as what we are doing and continuing to do to improve our uptime: We have a great deal of redundancy in the environment and are adding redundancy and capacity. We are continuing to add engineers and management to better coordinate all of our activities in this team. We are working very closely with our Product Development team to streamline the Tessitura backend technology in order to reduce the complexity in the hosted environment. We are refining and adding to our communications to provide as much information as we can if there is a disruption in service. We have increased the size of our customer relations and support team around hosting services in order to better serve our hosted organizations. We can and will go into further detail on any or all of these topics.
I will reiterate - As a company we pride ourselves on our excellent technology, services and community. At all levels we recognize that we are falling short in our hosting services and there is no more critical priority for us at this time. Jack Rubin, CEO, myself as President and all senior management are involved in returning our hosting services to excellent up-time and are glad to speak with any individuals to discuss further.
Andrew,
We appreciate the response clarifying your best intentions, but our CEO has asked me to make an additional request for some information. I have opened a support ticket for this request, but am copying it in below to gauge the interest of fellow community contacts on a proposal like this.
We would like an update on the issue and estimated time for resolution. We believe the support and communication in particular in these instances is below average and would like to have emailed updates every 15 minutes to Primary, Business, and IT contacts in our organization. I suspect other organizations will echo this. The priority is resolution, but we can plan our day and those ahead immediately if we have initial ballpark estimates and timely updates. I don't think this is too much to ask for. We are in season renewals and have a Broadway production in residence, one in which the booking agent happens to represent the show that was in residence during our last incident. I have opened a support ticket (11479,15) and placed calls to emergency support this morning and have been directed to the service status page on the website which has updated twice in three hours. This is not good enough. Please let me know the latest as soon as possible so I may pass the information on to our CEO and everyone in the organization who is impacted right now.
Regards,ChrisBass Performance Hall
Adding some more things that need to be done by Tessitura in outages like this:
For TNEW affected clients - Tessitura needs to provide a popup page that says the website is experiencing difficulties
Tessitura NEEDS to create a redundant failover system so if the primary servers go down, the backups automatically start up and we barely notice an issue OR allow licensees to create their own failover when RAMP goes down
There needs to be FAR better communication about what is happening and where the status of the outage is
What you are describing Andrew are great plans for the future of RAMP but I would like to know what Tessitura is intending to do now to fix this major problem. This is completely unacceptable and our patrons don't care that the database we are using is planning all this stuff. They just want to be able to call or login and buy their tickets, make their donations, or just interact with our organizations. When Tessitura goes down, Tessitura doesn't look bad, we, the licensees, look bad.
I understand what you're saying Andrew, but the reality is that if we can't sell tickets due to your system being unreliable/down you are crippling not just our organization but also impacting our visitor experience. Ticketing is the first place people go when they walk in the door, and when our Box Office staff are selling wristbands with a tally on a piece of aper and unable to process credit cards we are giving a poor first impression that we are not the world class destination we claim to be.
We trusted the RAMP hosted product and the reality is that we can not do that anymore. We have been hearing for over a year the same information you posted above. There is a point where we need to make a change when we hear that our other neighbors also using tessitura are not experiencing the frequent issues we are.
Heidi Quicksilver
Vice President - Technology
Rock & Roll Hall of Fame