Does Magic ETL v2 Have a Max Row Limit

nstephens88
nstephens88 Member
edited October 16 in Magic ETL

Hello,

I am in the process of combining two DataSets with Magic ETL where they have a same identifying Key. One of the DataSets has 3.9 million rows.

When I use 'Join Data' between the two datasets, I end up with exactly 1,000,000 rows (when it should be in the 3 million range).

I tried doing another Magic ETL where I only use the one 3.9 million row data set and did something simple as duplicating a column. I then ended up with 1,000,000 rows again.

Is this a configuration issue or am I at a limit with the Magic ETL.

Thanks.

Answers

  • rco
    rco Domo Employee

    Where are you seeing the row count of 1,000,000? Does the DataFlow execution say it had 1,000,000 rows of output? The "Data Output" column here:

    image.png

    Randall Oveson <randall.oveson@domo.com>

  • nstephens88
    nstephens88 Member
    edited October 16

    It says the Data input and Data output is 1 million rows, however the dataset itself has 3.9 million and I am not doing any operations that should reduce that 3.9 million number. Here is an example where I simply duplicate a column.

    image.png

  • rco
    rco Domo Employee

    I've discussed and diagnosed this issue in private messages with OP, but for any others who come across this issue in the future:

    Federated DataSets, that is, DataSets that reside on external databases and which are not part of the Cloud Amplifier feature, are not supported as inputs to Magic ETL. We prevent you from selecting them directly as inputs, but they can still be used indirectly if they are wrapped in a View. Although the DataFlow will run, it must export the data from the external database first, and this process is subject to certain limitations. The export cannot take longer than 30 minutes, it cannot produce more than half a gigabyte in total output file size, and it cannot produce more than 1 million rows. These limitations are the reason that we try to prevent this class of DataSet from being used as Magic ETL inputs.

    Randall Oveson <randall.oveson@domo.com>

Categories