The Duplication tag covers the topic of duplicated data and the strategies involved in removing the duplicates, often referred to as deduplication or simply dedupe.
Duplication refers to the inadvertent creation of duplicate entries in tables and the process for resolving the repeated entries.
In the simple case, this involves identifying a survivor row and eliminating the non-survivors.
A slightly more complex situation, survivorship logic must be employed. In the following example, the rule was to simply collapse non-empty values.
Before
CustomerName | Address | Phone
Bob | 100 Elm |
Bob | | 555.1234
After
CustomerName | Address | Phone
Bob | 100 Elm | 555.1234
The truly complex scenarios stem from conflicting values. Consider the following example, there is a conflicting value for the phone value in our deduplication candidate. Which value is the "right" is likely to be dependent on the supplied business rules and/or data collection logic.
Before
CustomerName | Address | Phone
Bob | 100 Elm | 555.4234
Bob | | 555.1234
After 1
CustomerName | Address | Phone
Bob | 100 Elm | 555.1234
After 2
CustomerName | Address | Phone
Bob | 100 Elm | 555.4234
After 3
CustomerName | Address | Phone
Bob | | 555.1234
Which After is correct is an "it depends" answer.
The first After merged the changes together. Perhaps the data collection system only records changed data (and we further assume said system required address for an entry to be valid)
The second After declared the first row completely correct. Perhaps there was an additional field showing it was the most recently updated. Or because it was most completely filled out, the assumption is that it was the survivor.
The third After declared the second row completely correct. It could be that there was another field showing it to be the most recently updated (Bob is now homeless, pity him).
