Input DataSet - Load New data
I am working on a new dataflow using Magic ETL. I have 11 input datasets. All are Workbench jobs with the Update method set to Append and they each have an assigned Upsert Key. I have configured each of the inputs to load New data. When I save my dataflow 3 of the 11 throw an error.
"This data set does not support data version filtering."
I have confirm that each dataset is indeed configured as I stated above. Why am I getting this error on these 3 when the other 8 are configured exactly the same and throw no error?
Answers
-
I reviewed the knowledgebase article and it looks as if you've set up everything correctly, have these datasets ran successfully since the update method was set to append? That's the only thing I can think of that might cause this.
KB article I'm referring to:
https://domo-support.domo.com/s/article/360045402273?language=en_US#ConfiguringInputDataSetTiles
0 -
I've gone through the knowledge base article several times myself. Yes, all of these datasets have been running successfully for months.
0 -
I can't see why only some of the tiles are displaying the error from what you've said and shown, but I can say that upsert DataSets don't support data version filtering.
Most DataSets (Replace and Append DataSets, excluding upsert) are loaded into Magic ETL in the form of a list of the originally uploaded files (usually CSV though sometimes other formats). Version filtering works by selecting a subset of those files to load, rather than loading all of them.
However, upsert DataSets are exposed to Magic ETL in the same way as DataSet Views. That is, the DataSet is exported from a database in its entirety. This exported table does not contain information to determine which rows belonged to which uploads. In the case of Views it's not clear how it ever could (since every row could be the product of any number of upstream tables), while in the case of upserts all we can say is that it does not. Currently.
I hope to see this limitation relaxed in the future. If you want to dig into the reason for the missing error messages on the other 8 of your inputs, feel free to DM me.Randall Oveson <randall.oveson@domo.com>
1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 747 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 395 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 6 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive