• Replies 13 replies
  • Subscribers 338 subscribers
  • Views 111 views
  • Users 0 members are here
Related

Consortium Merge

Shereen Marino
$organization

Hello,

I'm looking to receive guidance from consortia regarding the merge process.  We are migrating data for two new consortium members into our existing DB.  If anyone has any tips for how best to handle the merge process for a consortium, I'd greatly appreciate it.

Thank you,

Shereen

  • $organization
    Hi Shereen,

    We have a Data Integrity person that is responsible for our merges. Users from our organizations submit merge request to her and she reviews and schedules them. We do not let any orgs have access to merge constituents due to the lack of information that can be seen from their control groups. Merges are scheduled from noon on Monday to noon on Thursday. The job to actually merge constituents doesn't run until the next Monday at noon. This window gives users the opportunity to view the scheduled merges (via a custom report)     In the event that a merge should not have been scheduled, users may contact our Data Integrity person before the job runs. 

    I hope this helps!

    Cheers,
    KMK

    Sent from my iPhone

    On May 20, 2013, at 6:51 PM, "Shereen Marino" <bounce-shereenmarino5792@tessituranetwork.com> wrote:

    Hello,

    I'm looking to receive guidance from consortia regarding the merge process.  We are migrating data for two new consortium members into our existing DB.  If anyone has any tips for how best to handle the merge process for a consortium, I'd greatly appreciate it.

    Thank you,

    Shereen




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!
  • $organization

    Hi Shereen,

    We are a consortium of 4 organizations. We allow each organization to schedule merges. The merges take place in a scheduled job weekly.  The person scheduling the merge is required to enter a CSI if it is a shared record (i.e. in a nutshell, a record with activity in the last 3 years or so for two or more organizations). No merges are allowed to be scheduled after 1pm on Fridays to allow time for a review process.

    Before the weekly merge process, our ticketing functional support staff reviews the list of scheduled merges. They first validate to make sure the pair really are duplicates based on name/address type of data. In the past, they would manually look at records for potential issues that would need to be resolved before the merge could take place, like changes to memberships. So it was fairly time-consuming.

    We are in the process of finishing up customizations to the lp_validate_cust_merge procedure to trap issues upfront rather than relying on someone to manually review. It specifically looks for situations where we do not want a merge to take place, and reports what it found as specifically as possible to the user trying to schedule the merge. The weekly review of scheduled merges by our ticketing functional support team will be much easier and quicker now. We expect to go live with this in June.

    I hope this is helpful!

    Thanks,
    David

  • Former Member
    Former Member $organization

    Hi Shereen

    Our consortium is quite large, with a large number of dupes created all the time, so it's an ongoing issue.

    Anyone with the merge permission can try to schedule a merge of any two constituent records.

    We have a heavily modified lp_validate_cust_merge which tries to make a call, based on business rules that the consortium established early on, about which merges are allowed to happen. 

    Given that the major loss in a merge is General Tab address data, the business rules try to preference records which are likely to have better-kept data (eg subscribers or donors, or records with recent purchase activity).That is achieved by calculating for each proposed mergee a "Merge Priority" score.

     Constituents whose Merge Priority score is above a certain high threshold can't be the delete in any merge. If both proposed constituenti are above the threshold, the merge cannot be scheduled by ordinary users, and has to be referred back to the consortium support team, who initiate a discussion with whichever consortium partners have touch on the two records, and reach a resolution. They are then able to bypass the validation rules, (using a special User Group),  and schedule the merge.

    We have a customised Identify Dupes script which runs several times a week - there's always lots of candidate merges listed in the merge screen.

    We run the actual merge proc every night as a SQL job. There are also substantial customisations in our LP_CUST_MERGE script which can block certain merges, as well as changing the way some merge actions are handled.

    ...And we have been Very Brave, and developed an auto-merge script which will merge records without any human intervention at all, but I guess that's another story - We have more than 500,000 merged records in our database so far, so manual scheduling wouldn't really cut it, in the end.

    All of that was a substantial chunk of work, but the key part of the process, which took a long time to work through, was coming to an agreement about the business rules - that requires the partners to first develop an understanding of how the merge works, so as to make sensible decisions. But that process was well worth doing, I think, because it gave all of us a much better handle on constituent data generally. And those of us who went through it would probably say that, in the end, it doesn't actually matter very much which way the merge goes, so long as you're confident that the two records really are the same person, because the merge process retains almost everything from both anyway. So that was an interesting result.

    Ken


     


     

  • $organization in reply to Kayleb McKelvain

    Hello Kayleb,

     

    Thank you for your response.  Do you think I might be able to contact your Data Integrity person to discuss the details of your process?  It sounds like your organization is very similar to ours and I’d appreciate suggestions while we implement our consortium.

     

    Shereen

  • $organization

    Hello Kayleb,

     

    Thank you for your response.  Do you think I might be able to contact your Data Integrity person to discuss the details of your process?  It sounds like your organization is very similar to ours and I’d appreciate suggestions while we implement our consortium.

     

    Shereen

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Kayleb McKelvain
    Sent: Monday, May 20, 2013 5:14 PM
    To: Shereen Marino
    Subject: Re: [Tessitura Technical Forum] Consortium Merge

     

    Hi Shereen,

     

    We have a Data Integrity person that is responsible for our merges. Users from our organizations submit merge request to her and she reviews and schedules them. We do not let any orgs have access to merge constituents due to the lack of information that can be seen from their control groups. Merges are scheduled from noon on Monday to noon on Thursday. The job to actually merge constituents doesn't run until the next Monday at noon. This window gives users the opportunity to view the scheduled merges (via a custom report)     In the event that a merge should not have been scheduled, users may contact our Data Integrity person before the job runs. 

     

    I hope this helps!

     

    Cheers,

    KMK

    Sent from my iPhone


    On May 20, 2013, at 6:51 PM, "Shereen Marino" <bounce-shereenmarino5792@tessituranetwork.com> wrote:

    Hello,

    I'm looking to receive guidance from consortia regarding the merge process.  We are migrating data for two new consortium members into our existing DB.  If anyone has any tips for how best to handle the merge process for a consortium, I'd greatly appreciate it.

    Thank you,

    Shereen




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • $organization in reply to David Frederick

    Hello David,

    Thank you for your reply.  Yes, as I'm testing the process, I see that it can be time consuming.  I'm very interested in the customized process you're rolling out.  Could you let me know how it goes?  I'd like to discuss the process in more detail with you if possible.

    Thanks again,

    Shereen

  • $organization in reply to Former Member

    Hello Ken,

    Thank you for sharing about your process.  I'm impressed with your group's courage and determination to take on a project like this.  I'd love to hear about it in more detail.  In the long run, I would also like to automate the process as much as possible.  Would you be open to sharing the customizations and script your group has developed?  

    Shereen

  • $organization

    Hi Shereen,

     

    We would be happy to discuss this further with you. Please feel free to contact me directly at kayleb.mckelvain@attpac.org.

     

    Cheers,

    KMK

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Shereen Marino
    Sent: Wednesday, May 22, 2013 11:54 AM
    To: Kayleb McKelvain
    Subject: RE: [Tessitura Technical Forum] Consortium Merge

     

    Hello Kayleb,

     

    Thank you for your response.  Do you think I might be able to contact your Data Integrity person to discuss the details of your process?  It sounds like your organization is very similar to ours and I’d appreciate suggestions while we implement our consortium.

     

    Shereen

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Kayleb McKelvain
    Sent: Monday, May 20, 2013 5:14 PM
    To: Shereen Marino
    Subject: Re: [Tessitura Technical Forum] Consortium Merge

     

    Hi Shereen,

     

    We have a Data Integrity person that is responsible for our merges. Users from our organizations submit merge request to her and she reviews and schedules them. We do not let any orgs have access to merge constituents due to the lack of information that can be seen from their control groups. Merges are scheduled from noon on Monday to noon on Thursday. The job to actually merge constituents doesn't run until the next Monday at noon. This window gives users the opportunity to view the scheduled merges (via a custom report)     In the event that a merge should not have been scheduled, users may contact our Data Integrity person before the job runs. 

     

    I hope this helps!

     

    Cheers,

    KMK

    Sent from my iPhone


    On May 20, 2013, at 6:51 PM, "Shereen Marino" <bounce-shereenmarino5792@tessituranetwork.com> wrote:

    Hello,

    I'm looking to receive guidance from consortia regarding the merge process.  We are migrating data for two new consortium members into our existing DB.  If anyone has any tips for how best to handle the merge process for a consortium, I'd greatly appreciate it.

    Thank you,

    Shereen




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • $organization in reply to Kayleb McKelvain

    Great!  Thank you, Kayleb.

    Shereen

  • $organization

    Hi Shereen,

     

    I’d be happy to set up a time to discuss this further. Please feel free to email me directly at dfrederick@scfta.org to coordinate a time to discuss.

     

    Thanks,

    David

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Shereen Marino
    Sent: Wednesday, May 22, 2013 10:05 AM
    To: David Frederick
    Subject: Re: [Tessitura Technical Forum] Consortium Merge

     

    Hello David,

    Thank you for your reply.  Yes, as I'm testing the process, I see that it can be time consuming.  I'm very interested in the customized process you're rolling out.  Could you let me know how it goes?  I'd like to discuss the process in more detail with you if possible.

    Thanks again,

    Shereen

    From: David Frederick <bounce-davidfrederick9045@tessituranetwork.com>
    Sent: 5/21/2013 3:22:25 PM

    Hi Shereen,

    We are a consortium of 4 organizations. We allow each organization to schedule merges. The merges take place in a scheduled job weekly.  The person scheduling the merge is required to enter a CSI if it is a shared record (i.e. in a nutshell, a record with activity in the last 3 years or so for two or more organizations). No merges are allowed to be scheduled after 1pm on Fridays to allow time for a review process.

    Before the weekly merge process, our ticketing functional support staff reviews the list of scheduled merges. They first validate to make sure the pair really are duplicates based on name/address type of data. In the past, they would manually look at records for potential issues that would need to be resolved before the merge could take place, like changes to memberships. So it was fairly time-consuming.

    We are in the process of finishing up customizations to the lp_validate_cust_merge procedure to trap issues upfront rather than relying on someone to manually review. It specifically looks for situations where we do not want a merge to take place, and reports what it found as specifically as possible to the user trying to schedule the merge. The weekly review of scheduled merges by our ticketing functional support team will be much easier and quicker now. We expect to go live with this in June.

    I hope this is helpful!

    Thanks,
    David




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • Former Member
    Former Member $organization in reply to Shereen Marino

    Hi Shereen

    The scripts are posted on my profile page here.

    But those are still the v10 versions, I think.

    We're currently doing some more work on them post-v11 changes, so I'll post the new versions when they're ready, if you like.

    Ken

  • $organization

    That’s fantastic!  Thank you, Ken. 

     

    Shereen

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Ken McSwain
    Sent: Wednesday, May 22, 2013 8:06 PM
    To: Shereen Marino
    Subject: Re: [Tessitura Technical Forum] Consortium Merge

     

    Hi Shereen

    The scripts are posted on my profile page here.

    But those are still the v10 versions, I think.

    We're currently doing some more work on them post-v11 changes, so I'll post the new versions when they're ready, if you like.

    Ken

    From: Shereen Marino <bounce-shereenmarino5792@tessituranetwork.com>
    Sent: 5/22/2013 11:57:30 AM

    Hello Ken,

    Thank you for sharing about your process.  I'm impressed with your group's courage and determination to take on a project like this.  I'd love to hear about it in more detail.  In the long run, I would also like to automate the process as much as possible.  Would you be open to sharing the customizations and script your group has developed?  

    Shereen




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • $organization

    Thank you, David.  I’ll be in touch as I the more I dig into this, the more questions I have.

     

    Shereen

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of David Frederick
    Sent: Wednesday, May 22, 2013 1:56 PM
    To: Shereen Marino
    Subject: RE: [Tessitura Technical Forum] Consortium Merge

     

    Hi Shereen,

     

    I’d be happy to set up a time to discuss this further. Please feel free to email me directly at dfrederick@scfta.org to coordinate a time to discuss.

     

    Thanks,

    David

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Shereen Marino
    Sent: Wednesday, May 22, 2013 10:05 AM
    To: David Frederick
    Subject: Re: [Tessitura Technical Forum] Consortium Merge

     

    Hello David,

    Thank you for your reply.  Yes, as I'm testing the process, I see that it can be time consuming.  I'm very interested in the customized process you're rolling out.  Could you let me know how it goes?  I'd like to discuss the process in more detail with you if possible.

    Thanks again,

    Shereen

    From: David Frederick <bounce-davidfrederick9045@tessituranetwork.com>
    Sent: 5/21/2013 3:22:25 PM

    Hi Shereen,

    We are a consortium of 4 organizations. We allow each organization to schedule merges. The merges take place in a scheduled job weekly.  The person scheduling the merge is required to enter a CSI if it is a shared record (i.e. in a nutshell, a record with activity in the last 3 years or so for two or more organizations). No merges are allowed to be scheduled after 1pm on Fridays to allow time for a review process.

    Before the weekly merge process, our ticketing functional support staff reviews the list of scheduled merges. They first validate to make sure the pair really are duplicates based on name/address type of data. In the past, they would manually look at records for potential issues that would need to be resolved before the merge could take place, like changes to memberships. So it was fairly time-consuming.

    We are in the process of finishing up customizations to the lp_validate_cust_merge procedure to trap issues upfront rather than relying on someone to manually review. It specifically looks for situations where we do not want a merge to take place, and reports what it found as specifically as possible to the user trying to schedule the merge. The weekly review of scheduled merges by our ticketing functional support team will be much easier and quicker now. We expect to go live with this in June.

    I hope this is helpful!

    Thanks,
    David




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!