Hi,
We're finding huge latency issues (32ms vs. 900ms) for many API calls when we move our website from the same server as the API server to an external web host.
To troubleshoot, I've tried turning off encryption of the traffic (on our test site!), and setting up a test website that shows up how long each call takes. Populating a dropdown with performance dates based on availability on a MOS is proving to the most cumbersome.
Any tips?
Cheers
John Chabot, IT Manager
Young Centre for the Performing Arts
Do you see any issues running tracert between the webserver and api box? Any bad hops?
Also, are you sure that you've isolated the timing between the http request/response and the xml parsing?
You want to eliminate any causes at the web server level.
Hi Mark,
Thanks for your quick reply.
Pings come back at 92ms. Traceroutes are clean. We only have 800kbit/sec upstream for our organization, so I'm guessing it's just not robust enough to handle API traffic + our normal day-to-day.
I had our programmer/analyst write a script which times all the API calls on our splash page. Internally, they come back ~20ms each. Externally they take over 600ms each, which leads me to believe it's our upstream. I turned on verbose logging in IIS for the API, which is bytes sent, and time to serve, and it corresponds with the the API response times that we debugged.
I tried turning off SSL encryption (in case the minimal overhead or certificate handshake or whatnot was slowing it down), and being methodical about matching up the internal and external web.configs so I was comparing apples to apples, but still no dice.
We think we're using the API efficiently, which means no excessive calls to get the job done, and since the site works well when it's internally hosted, I'm concluding it's the upstream speed. ADSL, 6mbit/800kbit up.
Any other thoughts?
Cheers,
John Chabot
Hi John,
Humm, so we've got API calls through the same pipe as your organizations traffic? If that is the cause you should see wildly different latentcies depending upon your users.
But if the latency is consistant, then it's something systemic. One thing that can trip you up is firewall rules, you can have situations where you're forcing multiple authentications per request.
Usually in these cases it's something simple, you just have to keep segmentating the problem.
Mark
Also, how are you addressing the API, are you using DNS or IP? If it's DNS, is your server caching the lookup?
DNS can really eat up ms.
Do you have multiple network cards active on the API box? We have had similar problems with a dual-homed api system where both addresses to the IIS server was being reported and updated on DNS. This basically meant that we infact had DNS redunancy configured but no way to get to the secondary nic as it was a locked DMZ, only accessible from the outside and this caused MAJOR lag intermittenly. After some research the solution was simple, remove DNS registering on the NIC connected to DMZ and then clear cache and reload all DNS servers you use.
DNS resolution does cause a longer MS response initially but if you use ping against IP address without name resolution, you should be able to identify if it's DNS related or not. Also thing you might want to consider doing is simply to have another IIS box spun up with a default page, then set your firewall rules up against it, then run similar tests to see if you are getting same response fluctuation or not.
Trying to configure web.config to use static ip addressing against SEAT server and DATABASE could be a route to try, to see if it resolves the issue.
M2c
Doug