Block tiles in Magic ETL to simplify complex dataflows

KevinNorman

"Block Tiles" are larger tiles that contain a string of tiles inside. Block tiles could make large Magic ETLs (100+ tiles) easier to document, and much easier to read and understand.

These tiles would contain a group of tiles that have a single output tile (not an output dataset) to make for an easy way to connect block tiles to regular ones.

Benefits:

• Much easier time for new employees who need to understand what these ETLs are doing.
• Documentation of these ETLs could be mostly self-contained instead of needing to document elsewhere.
• Help existing engineers compartmentalize the origin of logic within an ETL, making for an easier and faster method of troubleshooting.

Here's a 4-minute video to help understand the impact this feature would have.

AndreaHenderson

This is such a good idea, and something I've definitely had on my personal Magic wishlist for a long time. I will be reviewing this soon with the team.