Hi all!
Would love to pick your brains on this. We moved from V14.0 to 15.1 this past June. During our time on 14.0, we got pretty familiar with the bug DEVOLD-6256 in the Defects list:
"When using Vantiv, credit card tokens were not being updated when a transaction was processed through the API and the expiration date entered online did not match the expiration date stored with the token resulting in a declined charge. This has been corrected so that the stored token will be updated and the credit card will not be declined due to the expiration date mismatch."
We spent about 2 years on 14.0 before moving to a version that had this bug fix this past summer - this means that in our system, we accumulated a good bit of bad data where, if a patron tried to use a card that was already in our system but had expired, the card expiration date in T_ACCOUNT_DATA got updated successfully but the expiration date in Vantiv was not updated, causing their card to be repeatedly declined unless someone went into their account and manually deleted the card, or the card got purged through regular purging. We were excited to see the bug fixed in V15.1, so that now new instances of this mismatch between T_ACCOUNT_DATA expiration date and Vantiv expiration date won't get created. However, we had the realization after the upgrade that this doesn't really do anything about the existing cards in the system with this mismatch, so we have patrons still encountering the same card decline issue despite their card working fine anywhere else.
We worked with Tessitura Support on this and talked a while about trying to work with Vantiv to do some kind of bulk update or get our card expir date issue, but we were quoted a couple thousand dollars from Worldpay to even get a list of all of our tokens from their system, so we've abandoned that route. Beyond more aggressively purging old cards in our system (which we are also doing), the only other path we're looking at is as follows - trying to somehow query any card in T_PAYMENT_GATEWAY_ACTIVITY that had two separate expiration dates appear at any point up until the upgrade (ie, any card whose expiration date was once one thing and then changed to a new thing, which was presumably not updated with Vantiv). We'd then have a list of act_ids, and then we'd probably go in and temporarily modify the out of the box CC purge utility to accept an inclusion list of account ids instead of an exclusion list of customer nos, and then purge the bad cards. This feels a bit convoluted, so we wanted to check in and see if anyone else has come up with a slicker way of cleaning up the bad data.
Hi McKenzie,
I have been thinking about your token data this morning and how best to go about cleaning it up. Most importantly, we must recommend that you do not alter the Credit Card Tokenization and Deletion Utility in any way. Tessitura does not support changes to any standard procedures, or customizations of the utilities that interact with the Payment Gateway (any of the credit card billing utilities, and creating or deleting tokens).
I would recommend running the Credit Card Tokenization and Deletion Utility to delete all tokens that are not currently tied to a billing schedule. While this may seem drastic, I believe that it is the cleanest way to resolve the issue, short of having someone manually delete individual cards from records.
I am happy to reopen our support ticket on this topic if you would like to discuss this further.
Thank you,
Boann Petersen
Support Escalation Manager
Hey Boann! Thanks for all your work helping us think through this.
I see where you're coming from, but I don't think purging all cards in our system is feasible. First, we store card info for a wide range of customer service related issues, and it would be massively detrimental for our customer-facing teams to nuke all the cards. Because we can only filter by an exclusion list on the constituent record, it means that the constituent records of many of our most active patrons (donors, subscribers, etc) get pulled into an exclusion list because of one lone card we don't want to delete (we store a credit card for charging lounge drinks on their account, for example), but then all the rest of their cards get stuck on their account forever and don't get purged as well.
However, on a more practical level, we can only purge about 3k cards in one night before the deletion utility fails. As you know from our (lengthy :) ) support ticket, we're working through a big backlog of cards to purge currently, and we've found that we can only do it in batches of 3k cards at a time. We have about 500k cards stored in our system... if we decided to delete everything, it would take almost half a year to do that.
I've run a couple of super preliminary queries to try and get at how much of these bad cards we're talking about, and it's probably around 15-20k cards. This doesn't seem huge in the grand scheme of things, but for a patron to have been struck by this card, they would have had to purchase tickets two times with the same card over the past two years - which means they're at least somewhat active at our org, and apparently keep coming back, because our phone teams deal with people calling in about this issue still to this day. 20k cards is too many to delete one by one, and not small enough for us to ignore.
If there's a better Tessitura-sanctioned way to deal with the fallout of the bug, I'd love to hear your thoughts! It seems like any org that was on 14.0 would have data like this in their system that needs cleaning.