As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime. We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare.
Luke McKenzie (Past Member) said:Of course it only works when TNEW is working at least a little bit
That's the rub, isn't it? But also, surely TNEW is generally in the best position to make determinations of when to display such messages, rather than waiting for someone to notice a customer email, and then call our web admin at 11 at night to log in to Tessitura from wherever they are to change a system table, again assuming some part of RAMP is functioning.
I may not be supposed to be sharing this (surely this is a safe forum for it?) but rumor has it that a very relevant session will be taking place at TLCC on Sunday. Remember to pack your torches and pitchforks!
That's where the TTL values come into play. If you have a DNS record for www.myorganization.org that points to your website, and that record has a TTL of 15 minutes any client that has that record cached will do a DNS lookup the next time the client attempts to access the site if more than 15 minutes has transpired since the last lookup.
It's true that other DNS servers may cache the records as well. But... they should be respecting the TTL also.
There was a time when there were quite a few poorly configured and managed DNS servers out there which could cause problems. However, DNS hacks and other issues over the years have done a lot to clean this up. We've found that a 15 minute TTL will be very well propagated within that time frame.
And I agree... managing though DNS change is ugly. But... it could be better than a site that seems up but isn't.
Yuck!
We encountered a problem sending the confirmation to the email address provided, but your order was successful. Please contact the box office for an order confirmation.
I'd rather our site not be available at all until everything is confirmed to be working.
I agree completely, though to play devil's advocate for a moment – in the case of a variable outage like this, how do they know whose sites are the "most" down? What about the collateral damage of shutting down sites that were working relatively well, if any? </devilsadvocate>
In the absence of a TNEW automatic shutdown notice, I'm wondering about setting up our own go-between service (similar to Dan Spees' idea below) that fields requests to our tickets URL, is smart enough to monitor TNEW and redirect to it when it's working, and don't when it's not. But also as Dan says, there may be better uses of time and resources to deal with outages effectively (including holding Tess responsible where they should be).
Luke McKenzie (Past Member) said:in the case of a variable outage like this, how do they know whose sites are the "most" down? What about the collateral damage of shutting down sites that were working relatively well, if any
My most recent experiences are of services coming up partially, then going down again. If it's variable enough that it's moving around like that for everyone, then it probably would be safer to just take everyone down rather than hope that everyone who happens to be up now will remain up.
And if it is fundamentally linked to different organizations, then it would be good if they had a mechanism to assess who is up and who is down (even if it's just calling people, or manually trying to buy tickets on their sites).
We could do more: our pre-commerce site is our own, and we should have something posted there, but I don't think I'm getting through to our web admin, so single point of failure there.
This is my main concern. I don't have the ability to get to the shut off switch to update the messaging that Luke referred to above, so what should the set up look like to redirect the DNS to a generic message page is my next problem to solve.
The intent of this post was not a pile on to the issues; we are genuinely trying to outline a solid contingency plan to ensure we have an MVS operation should the system be down for an extended period of time. Right now, here is our preliminary outline:
- Update website to alert visitors attempting transactions of issue (define messaging for Tess down versus TNEW and next steps depending on timing of festival)
- Notify all staff of issue and move Box Office team into down mode
- During high volume times, begin automatic distribution of seating reports three times a day to ensure we can give reasonable information on ticket availability over the phone
- Secure 1-2 functional CC terminals that can be sent to high volume walk up shows at remote box offices (we don't own any of our venues and open a remote box office before each performance)
- Complete training and testing for home box office agents on manually processing payments through web payment portal
- Establish editable pdf forms that can be completed and submitted for processing later for use during down time for tickets, contributions, and/or special events that are standardized and collect the appropriate information.
Obviously this isn't completely vetted and we have details to work through, but we are working to get this done prior to our Festival. Other opinions and/or ideas are definitely appreciated.
But I ask you John...how can we as RAMP member plan for continued service if RAMP prohibits us from say, having a local instance of Tessitura to bounce to should we go down? RAMP will not allow any outside connections (we tried to connect Azure for better cloud reporting and were flat out denied that option). Tessitura is forcing us to rely solely on them and their word and their IT team. You know I'm a huge Tessitura advocate. I've been using for years. I've been with RAMP and on-prem. I speak with absolutely certainty, I NEVER experienced this level of issues when with on-prem organizations. Our downtime was minimal. I intend to hold Tessitura accountable to their word on improving our experience. Two outages in 30 days is unacceptable....especially as a newly implemented organization.
It is?
Yes and no. Soon is probably hyperbole, and TNMP apps are expected to continue, but in this case Sabina was referring to the separate TNMP mobile web site, and that will become obsolete when TNEW v7 finishes rolling out, although I don't know off the top of my head what the schedule for support is after that.
Update on Tessitura Hosting Services/RAMP Webinar
We wanted to provide an update on our RAMP hosting service, including our ongoing work to continually improve our technology, our plans for Tessitura version upgrades, and changes and improvements to our communications. This webinar will include a presentation and ample time to answer your questions and get your feedback. While this will likely be of most interest to users of our hosting service (RAMP), it is open to any members of our community.
Register here:https://www.tessituranetwork.com/en/Items/Events/webinars/2018/Update-on-Tessitura-Hosting-Services-RAMP---Mar-2018
Exactly. But we can't forget this is poor planning on Tessitura's part and we're starting to see the real issues come from it.
What's worse is the performance is terrible. The choices are all over the place and never make sense.
Now we are seeing Angular pop up in the client as if TN is going to be able to keep up with it's turnover. TN is going to get stuck supporting Angular 4 or 5 on older machines. It's as if they've learned nothing from their software rot over the last few years.
Tessitura is a horrible choice for small organization. It's strong focus marketing features (Dashboards? Tessitura OTG?) and Technical dependence on older software commonly baked into the service (Powerbuilder, Silverlight) over actual technological development (support for cloud virtualization, decentralization of services) is all horrible news for any customer. I commonly find myself playing goalie to updates just because some department wanted to implement something they saw as a feature.
Now Tessitura has jumped on the bandwagon of "Agile" without knowing how to properly handle it's legacy products it's just no place for any organization, small or large.
This exactly. It is extremely hard to architect a highly fault-tolerant application with all the legacy software and requirements wrapped around Tessitura. Until a move is made to switch to a more micro-services approach, this won't change. The REST API is a move in the right direction here but from what I've heard, it's performance doesn't make that switch so appealing.