How to Remove Duplicate rows in analyzer?

SaloniShah

I have used the LinkedIn connector to export the stats. I see duplicated rows. How do I avoid them?

RobSomers

@SaloniShah The less than ideal solution is to create an ETL that and use the Remove Duplicates tile, but that doesn't solve the underlying issue. One thing you might try is delete and recreate the connector. If that doesn't solve the issue, it's most likely a bug with the connector and you should put in a support ticket.

jaeW_at_Onyx

@SaloniShah is the data actually duplicated in your dataset? If so that's probably something to be concerned about if it's not intentional.

Take a look at the connector that's bringing in the data, is it set to APPEND or REPLACE?

Is the data actually duplicated or are just a few rows duplicates? If you run the connector multiple times do you accumulate a copy of the dataset each time? (if so that implies that you're running an APPEND instead of REPLACE in your connector settings.

Sometimes Domo's datasets are 'broken', to deduplicate a row, you can use "remove duplicates" tile in Magic ETL or use a rank_window tile and apply a row_number to each row partitioned on date then apply a filter. (remove duplicates is probably conceptually easier to understand, rank_window is more precise).

SaloniShah

Hello @jaeW_at_Onyx , thanks for responding.

My data is set to Replace. Duplicates is definitely not the issue.

I'm attaching screenshots where in the Data tab I see single rows but when we go to visualize tab we see it duplicated. Example- 305 likes in May 2022.

MichelleH

@SaloniShah Based on your screenshots, those duplicates don't appear to actually exist in your dataset. Try removing the sorting in your card. Sometimes unnecessary sort fields can cause cards to not display correctly.

SaloniShah

Hey @MichelleH , thanks for your response. I did remove the sorting. Still, I see these duplicates. What else can the issue be?

RobSomers

@SaloniShah Looking back at what you posted about not seeing duplicates in your dataset in the data tab, I actually do see duplicate rows, they just aren't right next to each other:

SaloniShah

@RobSomers I just noticed. But I did check my dataset its getting replaced & not Appended. Why would it create duplicates ?

RobSomers

@SaloniShah The less than ideal solution is to create an ETL that and use the Remove Duplicates tile, but that doesn't solve the underlying issue. One thing you might try is delete and recreate the connector. If that doesn't solve the issue, it's most likely a bug with the connector and you should put in a support ticket.

SaloniShah

I recreated the connector and it still shows duplicates. I would try creating a ticket for support.

Thank you for your help.