Remove duplicate sites that have different 'operators'

louiswatson
louiswatson Member
in Magic ETL

Hello, I have a large list of sites where there are multiple duplicates of the same 'Site Address'. I can't just use the 'Remove Duplicates' tile in ETL as there can be multiple panels on one site.

I'm looking to sort the sites using the 'Latest update' date field. Then I'd like to remove sites that have the same address but with a different 'Operator'. This would leave me with a list of sites that are In Charge.

Tagged:

Best Answers

  • trafalger
    trafalger Coach
    Answer ✓

    If I'm understanding correctly, I'd do a group by tile by Site, find the MAX(Latest Updated) and then do an inner join back on Site and MAX(Latest Updated) = LatestUpdated to only get the most recent rows per site.

  • ColemenWilson
    ColemenWilson Coach
    edited August 2023 Answer ✓

    Can you use 2 criteria for your remove duplicates tile: 'Site Address' and 'Panel'?

    If not, you can use the Rank & Window tile and choose 'Latest Update' as the order and set 'Site Address' as the partition - here again you can use multiple fields if needed.

    Next use a filter tile to only keep rows with the rank value you want to keep, presumably those with a rank of 1.

    If I solved your problem, please select "yes" above

Answers

  • trafalger
    trafalger Coach
    Answer ✓

    If I'm understanding correctly, I'd do a group by tile by Site, find the MAX(Latest Updated) and then do an inner join back on Site and MAX(Latest Updated) = LatestUpdated to only get the most recent rows per site.

  • ColemenWilson
    ColemenWilson Coach
    edited August 2023 Answer ✓

    Can you use 2 criteria for your remove duplicates tile: 'Site Address' and 'Panel'?

    If not, you can use the Rank & Window tile and choose 'Latest Update' as the order and set 'Site Address' as the partition - here again you can use multiple fields if needed.

    Next use a filter tile to only keep rows with the rank value you want to keep, presumably those with a rank of 1.

    If I solved your problem, please select "yes" above

Categories