Ability to remove unnecessary columns directly in the input DataSet before loading all its rows

Options
Fabio Ferragina
Fabio Ferragina Member
edited September 2023 in Magic ETL Ideas

The functionality I require is already available in the MySQL and Redshift DataFlow…

Imagine.jpg

…in this example deselecting "department" and "state" columns.
This option would make it possible not to load all columns for all rows in the DataSet.
I hope it will be available in the Magic ETL input DataSets as soon as possible.
Thank you for your attention.

Tagged:
14
14 votes

Recommended Workaround · Last Updated

See comments.

Comments

  • Gionniz
    Gionniz Contributor
    Options

    I totally agree,
    having the possibility to drop some columns before processing
    sometime could save a lot of execution time.

    that's a must-have feature!

  • MichelleH
    MichelleH Coach
    Options

    Great idea! It would make sense to include this in the Data Handling section of the Input Dataset tile.

  • BSovers
    BSovers Member
    Options

    Just replace your dataset with a view of the dataset. Then in the view, you get to select the columns, and possibly add one or more constants or calculated fields.

    The only down-side is that is you add additional fields/columns to your dataset, you will also have to modify the view to include them (of you want them in your data flow.)

  • AndreaLovesData
    AndreaLovesData Contributor
    Options

    This is a great idea! I am going to review this with the team for inclusion on the roadmap.

    Domo Product Manager for Data Transformation (MagicETL)

  • AndreaLovesData
    AndreaLovesData Contributor
    Options

    Hi! Just wanted to follow up here. I spoke with our engineering team about this idea, and I learned something new about Magic ETL. Apparently, the reason that the feature does not exist is because it's not needed in Magic ETL. Magic already optimizes which columns are loaded into the DataFlow based on whether or not the columns are used throughout the DataFlow. I'm keeping this idea on my list as something to consider making more apparent in the UI, but for now this is already being solved behind the scenes by our DataFlow optimizer for Magic ETL. Let me know if you have any follow-up questions.

    Domo Product Manager for Data Transformation (MagicETL)

Categories