Duplicate Key Limit Error in Magic ETL

User_1826 · September 2015

I am using the Magic ETL tool. I am trying to left join a DataSet with 4.5m records to a smaller table with ~1100 rows. After awhile, it crashes and I get the following error message:

"OnboardFlowExecution(10847) data flow execution id (ca085a327ae6417685896fb90f62e971) failed: MergeJoin: Duplicate key limit in left table exceeded! Limit: 10000"

How could I remedy this in order to successfully execute my join in the Magic ETL? Please advise. Thank you.

-Mark

dark_knight · October 2015

Have you tried to swap the table positions in your join? Currently there is a limitation in Magic ETL Beta that the table on the left can not have more than 10000 duplicate values.

If the table on the right has less duplicate values try moving that table to the left side and changing the join direction as well.

Please let me know if this helps.

Jon220 · September 2015

I am having the same issue and same error message:

OnboardFlowExecution(74) data flow execution id (01323331e23d4970a0e69e69fbf861ee) failed: MergeJoin: Duplicate key limit in left table exceeded! Limit: 10000

Mark, if you found a solution could you please share?

Thanks

Jon

dark_knight · October 2015

Have you tried to swap the table positions in your join? Currently there is a limitation in Magic ETL Beta that the table on the left can not have more than 10000 duplicate values.

If the table on the right has less duplicate values try moving that table to the left side and changing the join direction as well.

Please let me know if this helps.

Jon220 · October 2015

That worked. Thanks a million! I watched the videos but couldn't find that limitiaion.

Thanks for replying with a solution.

Jon

kshah008 · October 2015

@markrenson, did Dark_Knight's also help solve your issue?

Ravendran · October 2015

Swapping the table from left to right fixed the issue.

Duplicate Key Limit Error in Magic ETL

Best Answer

Answers

Categories