Join Function in ETL

sky00221155
sky00221155
in Magic ETL

Hi,

I am trying to join a table on my DataSet, using the left join function. My original DataSet contains only 118k rows, and the other table contains 1405 rows. I thought that the output DataSet will stay 118k rows, and just have addition columns on it. However, the output DataSet is 4.6M which is way more then I expected. I am not sure if I misunderstand the join function. I want to use this function to add another column on my original DataSet by the matching primary key.

dojo.png


Tagged:

Answers

  • GrantSmith
    GrantSmith Coach

    Hi @sky00221155

    this is likely because you have the same value multiple times in the joining column. You need to either add more columns to tour join to make sure each record is joining to a single value or you can use a remove duplicates tile in your ETL. Refining your join columns is the best way to go if it’s possible as it’ll be more performant

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • sky00221155
    sky00221155

    Hi,

    After the duplicates function some of the data was lost since for the reference table, there are some attributes that have same primary key value. I tried to add a second primary key column to make the match more specific, but it doesn't seems to find any result.

Categories