Pass Variables into a MagicETL from Workflows.

DomoDork

With the new workflow features on the horizon, I think it would be immeasurably valuable to pass output variables from workflows into Magic ETLs since workflows can execute ETLs already. For example:

An 'audit' workflow is triggered that evaluates a dataset for various criteria. As part of the evaluation it sets output variables. At the end of the workflow, it triggers a MagicETL via API but can pass in those variables to said MagicETL as part of it's execution.

Inside the MagicETL, assuming you pre-defined some optional "input parameters", those then become the values from your workflow execution. This would allow MagicETLs to be much more dynamic as those variables can be used in various MagicETL tiles to control things like filtering, formulas, or change the way my output dataset outputs a Replace/Append/Partition dynamically.

The scenerio I think best fits this idea is if I had some workflow that tries to identify missing data in a dataset by looking for missing dates in their entirety along with statistical analysis on historical record counts on that same day of the week for those missing dates. Once those dates are identified, I can pass those dates from my workflow into the MagicETL. Those dates can then be used in my ETL to only reprocess data for those missing dates as part of using them in my Filter tile and possibly change my output dataset configuration away from Parition replace to Append so those missing records are just appened to the dataset.

Another scenerio would be if I triggered a workflow the moment a new user logged in. Maybe I want to trigger a MagicETL related to a business process to get them inserted into various system level datasets we designed for auditing/monitoring and apps we've built. The workflow could trigger various ETL's, passing in that user as a variable to act as filter so the ETLs are only doing work on that one person. This would be much more efficient and ensure faster processsing speeds.