Extractions and Primary Addresses

We mail to only primary mailing address but many files have other mailing addresses that also have some permissions. We are finding it difficult in extractions to use the Communication Management piece as it pulls from all addresses and not just primary.

Is there an easy way to use the Communication Management in Extractions and only have this look at primary address? Our first thought was to ignore Comm Mgmt and just work this into the extraction inclusion but would like to make use of this functionality rather than doing yet another work-around.