As Tessitura RAMP access is down for the second time this month and fourth time since we put tickets on sale in January, I'd like to hear what if any contingency plans others have set up for system downtime. We've only had one instance of being down during our actual Festival, but we did have a few hours during our Donor Presale that we were down and it was a nightmare.
Gawain,
Regarding partially down web services...
We experience this from time to time too. There are so many pieces involved. Front end, load balanced server clusters; content management systems; back end connections to Tessitura WebAPI servers with multiple web applications running on them; connections to databases in different security zones... Even self-hosted environments have to deal with these things.
One potential solution to the issue of partial site down situations is to host a simple web server somewhere else with a rather obscure name. This could be a server hosted by your DNS provider, many of which provide basic web server hosting as part of your DNS hosting package. When you find it necessary, you can change your DNS records and point your Tessitura enabled web server to this other server which can display customer service information about your outage. As long as you keep your TTL's on your Tessitura web server DNS records fairly short, you can effect a switch within a reasonable period of time.
Dan Spees said:When you find it necessary, you can change your DNS records and point your Tessitura enabled web server to this other server which can display customer service information about your outage.
That can get pretty ugly, though, right? In my experience DNS changes can often take hours to propagate. So by the time all clients are seeing the right page the problem might be fixed, and now it'll be an hour before they can get back to the site again. IANANE (I am not a network engineer).