Vantiv connectivity

We're on Payment Express, and while double charges usually rare for us, we've had five in the last week.  We're on 14.1/v6.

I'm trying to run down the circumstances, but it sounds like sometimes the cashier sees card declined and sometimes Tessitura crashes when payment information is submitted.

We are on Vantiv, and were getting lots of these last summer, but after v14.1.4 (we are on 15.0.4 at the moment), these have almost vanished for us, e.g. two since 1/1/2019.

Hey Kathleen,

We started experiencing this a few months ago (Payment Express). We are V15.0.3 and TNEW6. I opened a TASK ticket about this and this was the response:

"I wanted to let you know that the v15.0.3 Service Pack has been released and includes a fix that extends the timeout for the SOAP API while waiting for a payment to process to five minutes.
 
While this is not a silver bullet that will resolve 100% of the instances of payments recorded in the bank but not in Tessitura, it will address many of the issues we have seen as a result of the latency in the RAMP environment recently. The situations where we would still expect to see a double payment are if the Tessitura Application crashes between a payment getting authorized and saved, if Citrix closes Tessitura between a payment getting authorized and saved (due to a user walking away from their workstation without saving and the timeout settings causing Citrix to close Tessitura before Tessitura can close itself), and finally in some very few extreme cases of latency where a payment was delayed by more than five minutes. In all of my research of the double payments related to the latency issues I would guess that ~10% or so were delayed by more than five minutes.
 
The Hosting Services team is continuing to test and implement fixes to resolve the latency issues, but we wanted to take a moment now to see if you would like to start the process of getting 15.0.3 applied. If you are interested let me know and we can get your test environment scheduled to start."

I'm not certain if our upgrade to V15.0.3 fixed the problem. I haven't heard anything from our Finance department so I'm guessing the problem has resolved. I would suggest opening a TASK ticket though so that Tessitura can continue to track this issue.

I just checked with Finance and our double charge issue appears to have ended with the V15.0.3 service pack.

- Chris

We are locally hosted, btw.

I'm not sure if this is the same issue, but our web vendor let us know that they have noticed a number of SOAP requests to Vantiv resulting in a sporadic connection reset. We've seen it happen most often in Test, but a few times in Live.

We followed up with Vantiv, and this was their response:

We are aware of this issue and we have several projects inflight to stabilize our Express certification environment.  Some work has already been completed so hopefully they are seeing some improvement.

It feels like there are many issues happening with payment processing. We're now seeing phantom charges/authorizations appearing on customer cards that have no history in Tessitura, but it's clear the charge started in Tessitura. I'm still researching this new issue. I hope the developer team is reading this string and seeing that there might be a growing problem with payment processing that needs attention. It would be better to be proactive now, before it becomes an even bigger problem.

Hello Kathleen and all,

As you have likely found, the process of authorizing a credit card is complicated; there are many entities touching a single transaction as it is passed along from your website or the Tessitura application to the card holder’s bank and back. There are many points at which communication can break down. The Support Team is always happy to help investigate double-payment issues, and Kathleen, I have opened a ticket for you so we can start to look at the specific instances in your environment.

With that being said, there are several common patterns we see around double payments:

1. The authorization is initiated and then communication between Tessitura and the payment processor or the payment processor and the bank is disrupted. In some cases the authorization is disrupted, but in some cases the payment is authorized at the bank but the authorization does not make it back to Tessitura. To help resolve some of these instances Tessitura tries to reverse any authorization that returns a communication error three times.
   
2. The payment is authorized, but is not saved before the Tessitura application closes ungracefully. We see this when connecting to Tessitura through Citrix when the Citrix session time out closes Tessitura before the Tessitura application timeout closes Tessitura. We recommend saving an order or contribution with an authorized payment before walking away from the application. In addition, we recommend setting the application timeout to 15 minutes (900 seconds in T_DEFAULTS) so that Tessitura can close and cancel any unsaved payments properly. We also see the application close ungracefully sometimes due to defects causing the application to crash. We work to resolve these defects as they are identified, and encourage you to report to Support when you see the application crash before a payment can be saved.
   
   
3. With hosted payments online, both in TNEW and custom integrated websites, we see payments recorded with the processor that do not make it into Tessitura for a variety of reasons, from users clicking away too soon to communication breakdowns between the payments window and the website. In TNEW we have added a variety of checks and safety procedures in the coding around the payment window to prevent as many stranded payments as we can. When stranded hosted payments occur reach to Support or your web developer for custom integrated websites.

In all cases, we recommend reconciling your batches in the bank regularly, so that these issues can be identified and corrected quickly. When you find payments recorded in the bank and not in Tessitura contact Support and we can help to identify the underlying issues.

Thanks,

Boann Petersen

Support Escalation Manager