Input DataSet - Load New data

ARosser · 2024-11-05T19:40:33+00:00

There was an error rendering this rich post.

I am working on a new dataflow using Magic ETL. I have 11 input datasets. All are Workbench jobs with the Update method set to Append and they each have an assigned Upsert Key. I have configured each of the inputs to load New data. When I save my dataflow 3 of the 11 throw an error.

"This data set does not support data version filtering."

I have confirm that each dataset is indeed configured as I stated above. Why am I getting this error on these 3 when the other 8 are configured exactly the same and throw no error?

Find more posts tagged with

Magic ETL

Accepted answers

All comments

ggenovese

I reviewed the knowledgebase article and it looks as if you've set up everything correctly, have these datasets ran successfully since the update method was set to append? That's the only thing I can think of that might cause this.

KB article I'm referring to:

https://domo-support.domo.com/s/article/360045402273?language=en_US#ConfiguringInputDataSetTiles

ARosser

I've gone through the knowledge base article several times myself. Yes, all of these datasets have been running successfully for months.

rco

I can't see why only some of the tiles are displaying the error from what you've said and shown, but I can say that upsert DataSets don't support data version filtering.

Most DataSets (Replace and Append DataSets, excluding upsert) are loaded into Magic ETL in the form of a list of the originally uploaded files (usually CSV though sometimes other formats). Version filtering works by selecting a subset of those files to load, rather than loading all of them.

However, upsert DataSets are exposed to Magic ETL in the same way as DataSet Views. That is, the DataSet is exported from a database in its entirety. This exported table does not contain information to determine which rows belonged to which uploads. In the case of Views it's not clear how it ever could (since every row could be the product of any number of upstream tables), while in the case of upserts all we can say is that it does not. Currently.

I hope to see this limitation relaxed in the future. If you want to dig into the reason for the missing error messages on the other 8 of your inputs, feel free to DM me.