Hi folks
We're looking at migrating our API to a load-balanced (and virtualised) pair of servers.
Is anybody else running load-balanced / virtualised API servers? Any issues we should know about? Any tips or tricks or traps for young players?
(...and why have so many people not got their photos on their shiny new Profiles? Be Brave, mes enfants.....)
Ken
Hi Ken,
We don't load-balance our API server, but we are load-balancing our Webservers (on two separate boxes) and our 'Waiting Room' servers (again on two separate boxes). We have seen definate improvements in doing this and have been able to increase our concurrent bookers.
I'm curious as to why you have decided to load-balance your API - you may risk creating a bottle-neck to the TT Database. Also your choice of using virtualised servers (presumably set up on the same box) negates any advantage that may be gained (such as covering the risk of hardware failure) if they were set up on separate boxes.
I've attached a PDF showing the (basic) setup of our load-balanced web servers (I can add one showing the Waiting Room servers later, but as so few people use this facility it may not be of much use).
Hope this has been of some use.
Cheers Keith
Ken,
If you have two physical virtual servers, using one virtual instance on each server will work just fine. We are the Pittsburgh Cultural Trust are thinking in the same lines for our load balanced/redundant business continuity initiatives but are still in the planning phase.
I don’t see Keiths concern with bottlenecking when you are simply creating a “virtual host” that load balances the calls against two API boxes, that then in return forwards those calls to the database. The API itself does not create that much traffic that it will cause any IO issues on any of the connected devices, like motherboard, cpu, network cards etc, etc… but needless you want to do a little CYA and spread that load on two different physical boxes, simply for the reason of hardware failure. Even on our biggest on sales the API box is hardly jogging, 5-10% CPU usage and minimal disk IO, so that shouldn’t be of any concern. The problem is the back-end, the SQL database server… if it can’t keep up with the amount of calls being made to it when you combine seat, creditcard, box office usage, report writing, development and all the other parts that are tying into the database then you will see failures all across the board. The system is only as strong as the weakest link.
Ken, I think it would be of great use for the rest of the organizations to hear about your progress on this project… I’m sure we all want to be able to serve as many people as we possibly can so we all can provide better solutions as a community and to our patrons and constituents.
Thanks
Doug
From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Keith Bursnall Sent: Wednesday, May 06, 2009 11:02 AM To: Diamond, Douglas Subject: Re: [Tessitura Technical Forum] Load-balancing the API
From: Ken McSwain <bounce-kenmcswain5454@tessituranetwork.com> Sent: 5/5/2009 6:31:05 PM
This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!
Also, if you go down the ESXi route, this offers load balancing, providing you have a server with 2 physical network cards connected to your switches.
Our load testing on the virtual application environment and real DB servers sold many thousands of tickets in around 15 minutes (I forget the exact numbers). The only errors we got were genuine errors where we'd ran out of seats to sell. I'd guess other systems would fall over such as credit card authorisation or our public facing internet line if it were to happen IRL.
Thanks guys
We're really just following a general principle of trying to build redundancy and load-balancing into critical systems wherever we can.
Our proposed setup is two ESX servers, with one virtual API server in effect hosted on each. (Although the hosts will theoretically be spread across both physical servers, the API servers will be locked to individual boxes).
We do something similar already with our web site , which lives on four load-balanced virtualised servers spread across four physical boxen. (sort of like a little Cloud, is how i think of it - one of those fluffy white ones that float around on a summer day)
The load-balancing is done by separate hardware (F5) , outside the server farm. That works fine so far.
I was mostly wondering whether anyone had experienced any subtle issues actually running load-balanced API servers. As far as I could see, it should be fine, but you never know. Sounds like we should be ok.
If we do have any issues with it, I'll pass them on.