Report Requests not Running - Stalled on QueueStatus "s"

We've been working on integrating with the Reporting features on the Tessitura API for one of our clients lately, and we've recently ran into some issues. When I first started working on this (around 5/12/21), we were able to generate Report Schedules and direct Report Request calls that would successfully execute on the Test server (it never worked on the Prod server). Around 5/28/21, the reports requests stopped executing on the Test server as well, and are now stalled in queue with a QueueStatus of "s". We've been trying to debug this without much success, so we have a few questions to try and debug it.

What User, User Group, Resource, and Security permissions are required to generate a functioning Report Schedule as well as do direct Report Requests calls? We want to ensure that our users are configured correctly. Currently, we're attempting to generate them using both an Admin and Non-Admin account.

Parents
  • Hi, Tom,
     
    While we haven’t gone deep enough under the hood to diagnose symptoms at the level you have, we have been having what I assume is a similar problem pretty much since we went live on AWS.
     
    We have had a ticket open for over a month on this with very little progress, and it is causing major operational issues for us. Scheduled reports simply fail to show up sometimes, and on-demand reports intermittently don’t work. We were initially told it had something to do with the barcode font we were using in our primary ticketing report, so we reworked the report to remove the barcode (really annoying the people who relied on that to quickly tally program attendance), and while this reduced the problem a little for that report (presumably by reducing the rendering time overall), the underlying issue is still there for all reports. We are dealing with some pretty negative perceptions about the reliability of the system because of this. The most recent update we got from support was that “the system admin team is still investigating this” on July 8.
     
    I think we were early on the AWS migration path – we went live in mid-May.
     
    -David
     
     
    ____
     
    David Dwiggins, Chief Information Officer
    Historic New England
    (he/him/his)
    Eustis Estate
    1424 Canton Ave
    Milton, MA 02186-2418
    617-994-5948 (office)
     
     
  • ,

    It's good to know that I'm not alone.  We were compelled to move to AWS mid-June 2021.  We have had a support ticket open for about a week now.

    I'm wondering if anyone else out there is having the same problems.

    I'm also wondering what might be similar between your organization and mine.

  • Hi Tom and David,

    Boann here with the Support Escalation Team. I have reviewed your open support tickets on this issue and based on what we are seeing in our troubleshooting thus far, it does not seem like the issues are related. We are continuing to investigate both issues, and understand how frustrating it can be to have a report run slowly or not complete. We will continue to keep you updated on our progress within each support ticket.

    Thank you,

    Boann Petersen

    Support Escalation Manager

  • Hey  we have also had some strange things happen since the AWS cutover. We were, as you say, compelled to do it about two weeks after you. We've had support tickets open since then. A few things have happened but the absolutely most frustrating has been issues with report delivery out of Analytics. 

    First, we were receiving reports from dashboards that had been culled or re-named post-cutover - we learned that this was because our "previous" instance of Analytics had not been switched off yet. Even though the client had been cut over, I was still able to log into the old Analytics through Tessitura on the Go, and that instance of Sisense was sending out reports that no longer existed in the actual live instance. There was a lot of confusion around this and it took the network a couple of days to confirm that this is what had happened, which was concerning given how confidently the cutover had been insisted upon. 

    Once that "ghost" environment was switched off, hell broke loose. Now we're not receiving 90% of our scheduled reports out of Analytics (for dashboards and schedules created prior to the cutover), and dashboards created after the cutover are inconsistently erroring out when we try to use the "send me a report now" functionality, and schedules for those new dashboards aren't working at all. We've been in pre-sales for two weeks so this has not exactly helped with the stress of resuming business after almost a year and a half. 

Reply
  • Hey  we have also had some strange things happen since the AWS cutover. We were, as you say, compelled to do it about two weeks after you. We've had support tickets open since then. A few things have happened but the absolutely most frustrating has been issues with report delivery out of Analytics. 

    First, we were receiving reports from dashboards that had been culled or re-named post-cutover - we learned that this was because our "previous" instance of Analytics had not been switched off yet. Even though the client had been cut over, I was still able to log into the old Analytics through Tessitura on the Go, and that instance of Sisense was sending out reports that no longer existed in the actual live instance. There was a lot of confusion around this and it took the network a couple of days to confirm that this is what had happened, which was concerning given how confidently the cutover had been insisted upon. 

    Once that "ghost" environment was switched off, hell broke loose. Now we're not receiving 90% of our scheduled reports out of Analytics (for dashboards and schedules created prior to the cutover), and dashboards created after the cutover are inconsistently erroring out when we try to use the "send me a report now" functionality, and schedules for those new dashboards aren't working at all. We've been in pre-sales for two weeks so this has not exactly helped with the stress of resuming business after almost a year and a half. 

Children