We periodically run into issues where the Tessitura Report Server is in an "decided to stop process scheduled reports" state. When you log into the server, you will see that the Report Server is still running (not frozen or anything like that), but it has stopped processing scheduled jobs due to any number of possible issues which trigger a pop-up window describing the actual issue. Off the top of my head, a couple of problems I've found that will cause this state are deadlocks; dropped connection to Tessitura services, which for some reason the Report Server will not attempt to reconnect to; and a report having an invalid parameter. But there are more.
We've been struggling with identifying when the Report Server application is in this state. Sometimes it is days before someone realizes their reports are not running and by then it is too late in some cases...
Does anyone know of a good way to monitor the state of the Report Server application?
Ultimately I'd like to set up more intelligent alerting so we know when reports are not processing. One idea is to simply have a report scheduled to run every morning and if IT does not receive this, then we know there was an issue. The problem with this is that we have to remember to look for this email every morning and if it is missing. Also it is not a perfect solution as a time-sensative report may still be missed. I also saw a post referencing the following app: http://jockersoft.com/english/appmonitor_index.php I am unsure if this will help though because technically the Report Server is still running.
We have SO many problems with the report server. Every day it is stuck, reports are not run at critical times, and at the end of the month I have to make sure to log in at midnight to run the reports. We have worked with support on this before but there wasn’t anything that solved the problem.
One thing I’ve noticed is there is a process that gets stuck. It’s called gswin32c which is a ghostscript process. If I end that process the reports will finish, but again this messes up our need to have reports run at specific times and some of the PDF’s cannot be opened because they are damaged files.
Please Tessitura, can’t you rewrite this application?
From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Katie Lachance-Duffy Sent: Thursday, January 24, 2013 7:09 PM To: Gloria Ormsby Subject: RE: [Tessitura Technical Forum] Monitoring Tessitura Report Server state
Thanks for this idea, David! We've been struggling with this issue for years, and I just implemented your solution.
I just wanted to check in and see if you have any suggestions or words of advice now that you've been doing this for several months.
Thanks,
~Katie Lachance-Duffy
Application Systems Analyst/DBA/Tessitura Program Director
Philadelphia Regional Arts Consortium
From: David Frederick <bounce-davidfrederick9045@tessituranetwork.com> Sent: 9/28/2012 12:04:29 PM
Hi Jason,
We have similar issues where Report Server stops working and no one really knows until scheduled jobs are missed. I am not aware of a good way to monitor the actual state of the report server application itself; however, I am currently planning on creating a scheduled job (likely via SQL Server Agent) that essentially checks the scheduled queue table – which I believe is gooesoft_schedule_queue:
select COUNT(*)
from gooesoft_schedule_queue a
where a.status = 's'
and DATEADD(HH, 1, a.computed_date) < GETDATE();
If this query returns anything other than 0, then the report server is likely hung and my job would send an email out to administrators. I am padding this by an hour to take into account delays in starting jobs, as well as the fact that some jobs may not run exactly when scheduled due to the fact that multiple jobs are scheduled at around the same time. I’ll probably adjust how much time I pad this by based on how many false positives come out once I first implement.
David
_______________________________________________________ David Frederick Database Applications Analyst Segerstrom Center for the Arts 600 Town Center Drive, Costa Mesa, CA 92626 T (714) 556-2122 x 4067 E DFrederick@SCFTA.org
From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Jason Kaiser Sent: Friday, September 28, 2012 9:30 AM To: David Frederick Subject: [Tessitura Technical Forum] Monitoring Tessitura Report Server state
This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!