As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime. We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare.
Just to be sure: TNEW (at least ours) is still available in this outage, but I'm guessing a RAMP API dependent custom site would be down, which is an added wrinkle for us to consider as we extend our customizations and look into shared session pages.
Yes, our TNEW pages are RAMP hosted and affected by this outage.
Huh, I misread: "This issue is isolated to the North American region both application and TNEW websites for RAMP hosted sites. Locally hosted TNEW sites are not affected." That's weird. So I guess the API servers are not affected. I thought we were up, but maybe I was looking at cached pages? We're down now.
The amount of RAMP outages recently is making our administration question the decision to move to Tessitura. I am incredibly curious what the Network is doing on the back end to decrease the number of times that these types of issues occur?
I might at minimum look into hosting our own TNEW pages.
I echo this sentiment. RAMP prohibits us, the clients, from having ANY sort of contingency for when RAMP is down. We are NOT allowed to say, host a backup locally should RAMP go down. We are VERY limited by RAMP to our access (even VERY seasoned users). The silence and lack of response from the RAMP team is simply unacceptable on this front. We are losing money because of this. And with Tessitura going to a completely web based product, how is Tessitura going to ensure that RAMP and TNEW clients don't experience this level of service disruption on a regular basis. I recently saw a posting from Jack about customer service. We are the customers here and the level of service we are receiving is sub-par at best.
This being the second outage in the last month for us has been a pain. Our museum opens at 10AM EST so we were down for an hour. Our TNEW site worked during this time and our online purchasers had no interference.
Our back up plan is using Verifone machines to take credit card payments and take cash payments as usual at the Box Office. Our team marks each price type that they sold on a sheet of paper using the hash system. When the system comes back up they sell the tickets in Tessitura as cash and our Finance department adjusts for the actual credit cards we applied through the Verifone. It works, but it is a pain. Especially since someone could easily forget to mark the tickets they sold down on a transaction or two, but that is what math is for.
I too echo Christopher Cuhel's statement. The number of outages are increasing. What is RAMP planning to do for the future?
They are down again right now.
Jason Song
Scottsdale Arts
IT Manager
T: 480-425-5340 C: 480-529-4653
JasonS@ScottsdaleArts.org
What's incredibly frustrating, I've worked for RAMP and on-prem organizations. Only with RAMP have I experienced this level of down time and they seem to not do anything about it and never answer our concerns. They recommend RAMP to new clients and the product is just not stable. And with the ban on us being able to create a local fail-over, we cannot make disaster plans. This needs to change quickly.
Christopher Cuhel said:And with Tessitura going to a completely web based product, how is Tessitura going to ensure that RAMP and TNEW clients don't experience this level of service disruption on a regular basis
My sense is that many of the outages (and performance issues) revolve around the complex Jenga-like stack of services RAMP has to maintain to support an application that still uses PowerBuilder for core elements as a cloud, so a proper web services should involve a radical simplification of architecture and fewer of these "issue to host machines connect to storage devices" type failures.
There is no more important priority to the Tessitura Network than the consistent and reliable uptime and access to our services. The disruptions are unacceptable and we have put all available resources toward the continued hardening of our hosted environments. We greatly regret the disruptions. The data is secure and despite the number of outages, the technology is robust and top tier.
As far as what we are doing and continuing to do to improve our uptime: We have a great deal of redundancy in the environment and are adding redundancy and capacity. We are continuing to add engineers and management to better coordinate all of our activities in this team. We are working very closely with our Product Development team to streamline the Tessitura backend technology in order to reduce the complexity in the hosted environment. We are refining and adding to our communications to provide as much information as we can if there is a disruption in service. We have increased the size of our customer relations and support team around hosting services in order to better serve our hosted organizations. We can and will go into further detail on any or all of these topics.
I will reiterate - As a company we pride ourselves on our excellent technology, services and community. At all levels we recognize that we are falling short in our hosting services and there is no more critical priority for us at this time. Jack Rubin, CEO, myself as President and all senior management are involved in returning our hosting services to excellent up-time and are glad to speak with any individuals to discuss further.