Ability to remove unnecessary columns directly in the input DataSet before loading all its rows

Fabio Ferragina · 2023-09-14T13:21:02+00:00

There was an error rendering this rich post.

The functionality I require is already available in the MySQL and Redshift DataFlow…

…in this example deselecting "department" and "state" columns.
This option would make it possible not to load all columns for all rows in the DataSet.
I hope it will be available in the Magic ETL input DataSets as soon as possible.
Thank you for your attention.

Find more posts tagged with

Dataset

Magic ETL

Data Flows

Recommended Workaround

Status: Recommended Workaround

Comments

Gionniz

I totally agree,
having the possibility to drop some columns before processing
sometime could save a lot of execution time.

that's a must-have feature!

MichelleH

Great idea! It would make sense to include this in the Data Handling section of the Input Dataset tile.

BSovers

Just replace your dataset with a view of the dataset. Then in the view, you get to select the columns, and possibly add one or more constants or calculated fields.

The only down-side is that is you add additional fields/columns to your dataset, you will also have to modify the view to include them (of you want them in your data flow.)

AndreaHenderson

This is a great idea! I am going to review this with the team for inclusion on the roadmap.

AndreaHenderson

Hi! Just wanted to follow up here. I spoke with our engineering team about this idea, and I learned something new about Magic ETL. Apparently, the reason that the feature does not exist is because it's not needed in Magic ETL. Magic already optimizes which columns are loaded into the DataFlow based on whether or not the columns are used throughout the DataFlow. I'm keeping this idea on my list as something to consider making more apparent in the UI, but for now this is already being solved behind the scenes by our DataFlow optimizer for Magic ETL. Let me know if you have any follow-up questions.