Duplicate value handling in ETL

jakebutterfield

I have a column where there are around 34 values that are duplicated for one line that is supposed to be for two people. See picture for example. Ideally it'd be great to have one formula that can distinguish where the break would be so I don't have to manually create 34 lines of code, and have to continually add lines when a new duplicate value shows up. Right now my formula etl tile like this, "...WHEN 'Parent Lead Source' = 'OtherMagazine' then 'Other' WHEN..." and so on. I only need the first part of each string like in my example I only need 'Other'. I'm sure there has to be a better way...

GrantSmith

You could use magic ETL and a regular expression to format your data and then split it.

SPLIT_PART(REGEXP_REPLACE(`Parent Lead Source`, '([a-z])([A-Z])', '$1::$2'), '::', 1)

This will find the occurrence where a lowercase letter precedes an uppercase character and then injects '::' between them to be able to split it apart with the split_part function.

Alternatively, you could do a pure regex solution like:

REGEXP_REPLACE(`Parent Lead Source`, '^(.[a-z])[A-Z].*$', '$1')

MarkSnodgrass

It doesn't appear that you have a lot of consistency to go off of since some of your first words have spaces and sometimes your first word is also the second word. Are the words at the end consistently the same? If so, you could look at the end of the string using the RIGHT() function and if it equals Referral or Internet, then you could use the REPLACE function to replace that word with nothing.

GrantSmith

You could use magic ETL and a regular expression to format your data and then split it.

SPLIT_PART(REGEXP_REPLACE(`Parent Lead Source`, '([a-z])([A-Z])', '$1::$2'), '::', 1)

This will find the occurrence where a lowercase letter precedes an uppercase character and then injects '::' between them to be able to split it apart with the split_part function.

Alternatively, you could do a pure regex solution like:

REGEXP_REPLACE(`Parent Lead Source`, '^(.[a-z])[A-Z].*$', '$1')

MarkSnodgrass

It doesn't appear that you have a lot of consistency to go off of since some of your first words have spaces and sometimes your first word is also the second word. Are the words at the end consistently the same? If so, you could look at the end of the string using the RIGHT() function and if it equals Referral or Internet, then you could use the REPLACE function to replace that word with nothing.

jakebutterfield

That's a good point, I could do that for the instances where that occurs. Thanks Mark