Hello,
I'm looking to receive guidance from consortia regarding the merge process. We are migrating data for two new consortium members into our existing DB. If anyone has any tips for how best to handle the merge process for a consortium, I'd greatly appreciate it.
Thank you,
Shereen
Hi Shereen
Our consortium is quite large, with a large number of dupes created all the time, so it's an ongoing issue.
Anyone with the merge permission can try to schedule a merge of any two constituent records.
We have a heavily modified lp_validate_cust_merge which tries to make a call, based on business rules that the consortium established early on, about which merges are allowed to happen.
Given that the major loss in a merge is General Tab address data, the business rules try to preference records which are likely to have better-kept data (eg subscribers or donors, or records with recent purchase activity).That is achieved by calculating for each proposed mergee a "Merge Priority" score.
Constituents whose Merge Priority score is above a certain high threshold can't be the delete in any merge. If both proposed constituenti are above the threshold, the merge cannot be scheduled by ordinary users, and has to be referred back to the consortium support team, who initiate a discussion with whichever consortium partners have touch on the two records, and reach a resolution. They are then able to bypass the validation rules, (using a special User Group), and schedule the merge.
We have a customised Identify Dupes script which runs several times a week - there's always lots of candidate merges listed in the merge screen.
We run the actual merge proc every night as a SQL job. There are also substantial customisations in our LP_CUST_MERGE script which can block certain merges, as well as changing the way some merge actions are handled.
...And we have been Very Brave, and developed an auto-merge script which will merge records without any human intervention at all, but I guess that's another story - We have more than 500,000 merged records in our database so far, so manual scheduling wouldn't really cut it, in the end.
All of that was a substantial chunk of work, but the key part of the process, which took a long time to work through, was coming to an agreement about the business rules - that requires the partners to first develop an understanding of how the merge works, so as to make sensible decisions. But that process was well worth doing, I think, because it gave all of us a much better handle on constituent data generally. And those of us who went through it would probably say that, in the end, it doesn't actually matter very much which way the merge goes, so long as you're confident that the two records really are the same person, because the merge process retains almost everything from both anyway. So that was an interesting result.
Ken
Hello Ken,
Thank you for sharing about your process. I'm impressed with your group's courage and determination to take on a project like this. I'd love to hear about it in more detail. In the long run, I would also like to automate the process as much as possible. Would you be open to sharing the customizations and script your group has developed?
The scripts are posted on my profile page here.
But those are still the v10 versions, I think.
We're currently doing some more work on them post-v11 changes, so I'll post the new versions when they're ready, if you like.