Duplicate value handling in ETL

jakebutterfield
jakebutterfield Member
in Magic ETL
domoquestion.png

I have a column where there are around 34 values that are duplicated for one line that is supposed to be for two people. See picture for example. Ideally it'd be great to have one formula that can distinguish where the break would be so I don't have to manually create 34 lines of code, and have to continually add lines when a new duplicate value shows up. Right now my formula etl tile like this, "...WHEN 'Parent Lead Source' = 'OtherMagazine' then 'Other' WHEN..." and so on. I only need the first part of each string like in my example I only need 'Other'. I'm sure there has to be a better way...

Tagged:

Best Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You could use magic ETL and a regular expression to format your data and then split it.

    SPLIT_PART(REGEXP_REPLACE(`Parent Lead Source`, '([a-z])([A-Z])', '$1::$2'), '::', 1)

    This will find the occurrence where a lowercase letter precedes an uppercase character and then injects '::' between them to be able to split it apart with the split_part function.

    Alternatively, you could do a pure regex solution like:

    REGEXP_REPLACE(`Parent Lead Source`, '^(.[a-z])[A-Z].*$', '$1')

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • MarkSnodgrass
    MarkSnodgrass Coach
    Answer ✓

    It doesn't appear that you have a lot of consistency to go off of since some of your first words have spaces and sometimes your first word is also the second word. Are the words at the end consistently the same? If so, you could look at the end of the string using the RIGHT() function and if it equals Referral or Internet, then you could use the REPLACE function to replace that word with nothing.

    **Check out my Domo Tips & Tricks Videos

    **Make sure to <3 any users posts that helped you.
    **Please mark as accepted the ones who solved your issue.

Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You could use magic ETL and a regular expression to format your data and then split it.

    SPLIT_PART(REGEXP_REPLACE(`Parent Lead Source`, '([a-z])([A-Z])', '$1::$2'), '::', 1)

    This will find the occurrence where a lowercase letter precedes an uppercase character and then injects '::' between them to be able to split it apart with the split_part function.

    Alternatively, you could do a pure regex solution like:

    REGEXP_REPLACE(`Parent Lead Source`, '^(.[a-z])[A-Z].*$', '$1')

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • MarkSnodgrass
    MarkSnodgrass Coach
    Answer ✓

    It doesn't appear that you have a lot of consistency to go off of since some of your first words have spaces and sometimes your first word is also the second word. Are the words at the end consistently the same? If so, you could look at the end of the string using the RIGHT() function and if it equals Referral or Internet, then you could use the REPLACE function to replace that word with nothing.

    **Check out my Domo Tips & Tricks Videos

    **Make sure to <3 any users posts that helped you.
    **Please mark as accepted the ones who solved your issue.
  • jakebutterfield
    jakebutterfield Member

    That's a good point, I could do that for the instances where that occurs. Thanks Mark

Categories