Remove duplicate sites that have different 'operators'

louiswatson

Hello, I have a large list of sites where there are multiple duplicates of the same 'Site Address'. I can't just use the 'Remove Duplicates' tile in ETL as there can be multiple panels on one site.

I'm looking to sort the sites using the 'Latest update' date field. Then I'd like to remove sites that have the same address but with a different 'Operator'. This would leave me with a list of sites that are In Charge.

trafalger

If I'm understanding correctly, I'd do a group by tile by Site, find the MAX(Latest Updated) and then do an inner join back on Site and MAX(Latest Updated) = LatestUpdated to only get the most recent rows per site.

ColemenWilson

Can you use 2 criteria for your remove duplicates tile: 'Site Address' and 'Panel'?

If not, you can use the Rank & Window tile and choose 'Latest Update' as the order and set 'Site Address' as the partition - here again you can use multiple fields if needed.

Next use a filter tile to only keep rows with the rank value you want to keep, presumably those with a rank of 1.

trafalger

If I'm understanding correctly, I'd do a group by tile by Site, find the MAX(Latest Updated) and then do an inner join back on Site and MAX(Latest Updated) = LatestUpdated to only get the most recent rows per site.

ColemenWilson

Can you use 2 criteria for your remove duplicates tile: 'Site Address' and 'Panel'?

If not, you can use the Rank & Window tile and choose 'Latest Update' as the order and set 'Site Address' as the partition - here again you can use multiple fields if needed.

Next use a filter tile to only keep rows with the rank value you want to keep, presumably those with a rank of 1.