• Replies 10 replies
  • Subscribers 338 subscribers
  • Views 104 views
  • Users 0 members are here
Related

How Often to Merge Duplicate Records

Christina Bledsoe
$organization

So we are finally ready to start tackling merging duplicate records!  Sadly we have quite a large amount to tackle first.  About 12,000. Gulp.  What I'm thinking of doing is running the identify dupes procedure once and then scheduling the merge procedure to run nightly so that any dupes that were scheduled that day get merged and off the list.

After we tackle those then we had decided to schedule the procedures to run weekly, but I'm interested to hear how often other organizations schedule these procedures to run.  Do you find it's better to do it daily or weekly?

Any advice on how to get our massive amount of duplicates down quickly?  I think we're going to divide the alphabet up so we can have several people working on scheduling them at once.

Thanks!

  • $organization

    Christina,

    It sounds like you have some work ahead of you but a good plan.  We run the identify weekly (early Monday morning) then merge scheduled dupes nightly.  This works well for us since it gives me a week to schedule all of the identified dupes.  Having the merges done nightly is great for gift entry.  For example someone makes a gift online and a dupe account is created.  If there is a membership on the existing account and the gift is a membership gift, I like to merge the accounts before I do the gift entry - its cleaner for the memberships.  Having accounts merged nightly lets me do gift entry of online gifts in a timely manner.  Having merges happen only once a week would be inconvenient.

    Dale

  • $organization

    I identify duplicates at least once daily and merge at least once daily. Many duplicate accounts are built on the web; I want tickets to print with the correct patron number and contributions to go to the correct membership. During a period of heavy web activity, I can spend an hour or two on merging accounts; if I did it once a week, I think I’d lose a full day. However, I’m pretty compulsive about post-merge cleanup—deleting duplicate addresses, e-mail addresses, and phone numbers; making sure that web orders keep the correct address/eaddress references; checking all salutations; and other stuff like that.

     

    I don’t worry (too much) about duplicates that aren’t in actively used accounts. As people find them, we clean them up. We also ask our mail house to check for and suppress duplicate addresses when we are doing bulk mailings where the list pull criteria are very broad.

     

    Perhaps a way to handle the backlog is to set the Identify Duplicates procedure to look for accounts touched in the last day or week (since your last run), and clean up those—and not worry about the many accounts that are sitting there untouched. You could then attack the backlog slowly, at your leisure, as you have time.

     

    Lucie

     

    ___________________________________

    Lucie Spieler

    IT Development and Training Manager

    Editor, Season Program

     

    Florida Grand Opera

    8390 NW 25th Street

    Miami, FL 33122

    305.854.1643 x 1521

    Box Office: 800.741.1010

    To Give a Gift: 305.403.3315

    www.FGO.org

     

    2010-2011 | 70TH SEASON | THE MYSTERIES OF LOVE

    Turandot | The Tales of Hoffmann | Don Giovanni | Cyrano

     

  • $organization in reply to Lucie Spieler

    I can see the benefits in scheduling both procedures to run daily.  And some weeks we have very heavy web traffic, which is where the majority of our duplicates come from so it would be easier to keep on top of.  I like the idea of just focusing on merging the active accounts.  That makes the task seem a little less daunting.

    Thanks!

  • $organization

    I agree with Dale and Lucie: it's best to look for dupes more than once a week, every day if you can.They multiply out of control very quickly if left unattended.

    We run the merge process daily, overnight, and run various different id scripts manually during the day. Don't limit yourself to the standard dupe id program alone: it will miss some dupes, and a combination of scripts has better odds of finding the highest percentage of them.

    Also, if you are very confident of the quality of your dupes (i.e. the matching rules are very strict) it is possible to automate the scheduling of them via SQL. Early in our Tessitura life we did that with a large number of strictly identified dupes that were left over from our previous system, and were happy with the results.

  • $organization

    We run the merge process and the potential duplicate procedure nightly. 

    Dee Dee

    Dee Dee Fite |dfite@IndianapolisSymphony.org
    Director of Patron Systems

    Indianapolis Symphony Orchestra
    *: 32 E. Washington St. Suite 600 | Indianapolis, IN 46204-2919
    (: 317.229.7080 | (f) 317.262.1159

    From Classic Rock to Classical, the ISO celebrates your favorite music idols this January! Rock to The Music of Paul McCartney featuring Tony Kishman, Jan. 14-16 and delight in the "Emperor" Concerto when piano powerhouse Andre Watts Plays Beethoven, Jan. 21 & 22. Tickets at IndianapolisSymphony.org.
     
    Music of Paul McCartney link -
    http://www.indianapolissymphony.org/tickets/learnmore.aspx?id=3520
     
    Watts Plays Beethoven link -
    http://www.indianapolissymphony.org/tickets/learnmore.aspx?id=3567

     

    P Please don't print this e-mail unless you really need to.

     

     

     

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Christina Bledsoe
    Sent: Wednesday, December 29, 2010 1:42 PM
    To: Dee Dee Fite
    Subject: [Tessitura Technical Forum] How Often to Merge Duplicate Records

     

    So we are finally ready to start tackling merging duplicate records!  Sadly we have quite a large amount to tackle first.  About 12,000. Gulp.  What I'm thinking of doing is running the identify dupes procedure once and then scheduling the merge procedure to run nightly so that any dupes that were scheduled that day get merged and off the list.

    After we tackle those then we had decided to schedule the procedures to run weekly, but I'm interested to hear how often other organizations schedule these procedures to run.  Do you find it's better to do it daily or weekly?

    Any advice on how to get our massive amount of duplicates down quickly?  I think we're going to divide the alphabet up so we can have several people working on scheduling them at once.

    Thanks!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • $organization in reply to Dee Dee Fite (Past Member)

    Thanks for the advice!  On a related topic we had decided we would keep the account that has a membership or subscription because members get an id card with their account number on it and subscribers generally know their account number.  We have come across some duplicate records where one account has an active membership and the other has a subscription package.   I'm very hesitant to merge these accounts and we're trying to figure out a way to handle them.  I know that when you search for an account that was merged the kept account will get returned in the search.  So should I not worry about it?  Any advice?

  • $organization

    I wouldn't worry about it, the package will merge just fine into the other. The only thing I start getting concern about and look for is if both accounts have an active membership. It is rare but could happen.

     

    Marty

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Christina Bledsoe
    Sent: Tuesday, January 25, 2011 3:54 PM
    To: Martin A. Jones
    Subject: RE: [Tessitura Technical Forum] How Often to Merge Duplicate Records

     

    Thanks for the advice!  On a related topic we had decided we would keep the account that has a membership or subscription because members get an id card with their account number on it and subscribers generally know their account number.  We have come across some duplicate records where one account has an active membership and the other has a subscription package.   I'm very hesitant to merge these accounts and we're trying to figure out a way to handle them.  I know that when you search for an account that was merged the kept account will get returned in the search.  So should I not worry about it?  Any advice?

    From: Dee Dee Fite <bounce-deedeefite2929@tessituranetwork.com>
    Sent: 1/4/2011 1:47:54 PM

    We run the merge process and the potential duplicate procedure nightly. 

    Dee Dee

    Dee Dee Fite |dfite@IndianapolisSymphony.org
    Director of Patron Systems

    Indianapolis Symphony Orchestra
    *: 32 E. Washington St. Suite 600 | Indianapolis, IN 46204-2919
    (: 317.229.7080 | (f) 317.262.1159

    From Classic Rock to Classical, the ISO celebrates your favorite music idols this January! Rock to The Music of Paul McCartney featuring Tony Kishman, Jan. 14-16 and delight in the "Emperor" Concerto when piano powerhouse Andre Watts Plays Beethoven, Jan. 21 & 22. Tickets at IndianapolisSymphony.org.
     
    Music of Paul McCartney link -
    http://www.indianapolissymphony.org/tickets/learnmore.aspx?id=3520
     
    Watts Plays Beethoven link -
    http://www.indianapolissymphony.org/tickets/learnmore.aspx?id=3567

     

    P Please don't print this e-mail unless you really need to.

     

     

     

     

    From: Tessitura Technical Forum [mailto:forums-technical@tessituranetwork.com] On Behalf Of Christina Bledsoe
    Sent: Wednesday, December 29, 2010 1:42 PM
    To: Dee Dee Fite
    Subject: [Tessitura Technical Forum] How Often to Merge Duplicate Records

     

    So we are finally ready to start tackling merging duplicate records!  Sadly we have quite a large amount to tackle first.  About 12,000. Gulp.  What I'm thinking of doing is running the identify dupes procedure once and then scheduling the merge procedure to run nightly so that any dupes that were scheduled that day get merged and off the list.

    After we tackle those then we had decided to schedule the procedures to run weekly, but I'm interested to hear how often other organizations schedule these procedures to run.  Do you find it's better to do it daily or weekly?

    Any advice on how to get our massive amount of duplicates down quickly?  I think we're going to divide the alphabet up so we can have several people working on scheduling them at once.

    Thanks!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!




    This message was sent automatically to you by www.tessituranetwork.com because you subscribed to the Tessitura Technical Forum. You may reply to this message to post to the Technical forum or visit the site to search, read and post to the forums. In the interest of keeping the forum posts from becoming cluttered, we encourage you to delete previous message text from your reply before sending. Thank you!

  • $organization in reply to Christina Bledsoe

    Unknown said:

    Thanks for the advice!  On a related topic we had decided we would keep the account that has a membership or subscription because members get an id card with their account number on it and subscribers generally know their account number.  We have come across some duplicate records where one account has an active membership and the other has a subscription package.   I'm very hesitant to merge these accounts and we're trying to figure out a way to handle them.  I know that when you search for an account that was merged the kept account will get returned in the search.  So should I not worry about it?  Any advice?

    With a few exceptions (such as >1 sub pkg) we merge and keep the oldest record regardless of subscriptions and/or memberships. Memberships in particular need to be checked the next day, but I'd recommend checking merges the next day in general to make sure all is as expected.

  • $organization in reply to Chris Jensen

    Hi Chris, 

     

    You mentioned it is possible to automate the scheduling of  merging duplicates via SQL. How and where is this possible ?  

     

    Thanks,

    SS

  • $organization in reply to snehal sheth

    Unknown said:

    You mentioned it is possible to automate the scheduling of  merging duplicates via SQL. How and where is this possible ?  

    Add or modify a nightly SQL Server Agent Job to include a step that runs AP_MERGE_CUSTOMER2. Drop me a note offline and I can elaborate further if you like.