Option to Remove the 10,000 Duplicate Error for ETL
Because of how fast/easy ETL flows are, I prefer to use them for most flows. My problem is that sometimes I hit the following error, "Error joining data. The left input cannot include over 10,000 duplicates. Please switch your inputs, group your data, or remove duplicates before joining". The Dojo suggests that this error is only shown to stop users from accidentely making cartesian joins, but I know that's not what I'm doing. For example, if I want to join an item table onto a transaction table, then I have to do it in SQL because I'll get the error. This leads me to having multiple flows for the same end result.
I suggest you give admins the option to disable the error.
The super secret workaround we all use...
Right outer join instead of left.
This keeps people who don't know what they are doing from breaking stuff, but enables the rest of us to go on blissfully joining.
It'll seem awkward at first, but in a bit, you won't even have to think about it.
You want to set it up to pull all records from the table on the right, and only those which match from the left. Simple reverse of what you're used to.DataMaven
Breaking Down Silos - Building Bridges
Check out my video!
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"3
@DataMaven have you noticed this not working lately? I have a flow disabling because of this error even though it is a right join.0
@Figauro the question isn't whether you're using a LEFT or a RIGHT join.
The question is which side of the JOIN has the duplicates? (the many side of the 1:M relationship). You want the M side to be on the right.
If you think you already have that, then apply a GROUP BY `SHIP FROM ID` on the LEFT side before the JOIN. This guarantees non-duplicate values on the LEFT.Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
- 7.7K All Categories
- 7 Connect
- 922 Connectors
- 245 Workbench
- 477 Transform
- 1.8K Magic ETL
- 60 SQL DataFlows
- 446 Datasets
- 57 Visualize
- 205 Beast Mode
- 2.1K Charting
- 8 Variables
- 23 Cards, Dashboards, Stories
- 4 Automate
- 349 APIs & Domo Developer
- 84 Apps
- 15 Predict
- 3 Jupyter Workspaces
- 12 R & Python Tiles
- 242 Distribute
- 60 Domo Everywhere
- 241 Scheduled Reports
- 15 Manage
- 36 Governance & Security
- 43 Product Ideas
- 1.1K Ideas Exchange
- 3 Community Forums
- 16 Getting Started
- 3 Community Member Introductions
- 50 Community News
- 18 Event Recordings
- 576 日本支部