-
Comment out tiles in MagicETL
When building complex ETLs it would be handy to be able to "comment out" tiles so they are ignored for execution. For bonus points, I'd like to be able to have an unterminated chain of commented-out tiles not block the ETL's execution.
-
Ability to remove unnecessary columns directly in the input DataSet before loading all its rows
The functionality I require is already available in the MySQL and Redshift DataFlow… …in this example deselecting "department" and "state" columns. This option would make it possible not to load all columns for all rows in the DataSet. I hope it will be available in the Magic ETL input DataSets as soon as possible. Thank…
-
Support for having reusable ETL Logic
We are trying to redesign our existing ETLs (Magic ETL) whereby we want to have two identical sets of ETLs one for Full Load and another for incremental Load. Since the ETL is complex and there are number of ETLS that will adapt same pattern, we would like a functionality in Magic ETL where we can store entire Business…
-
Long Running Dataflow Alerting/Auto-Restart
We need to have the ability to understand when dataflows are running longer than the average duration. We would like to set user defined alert criteria based upon the dataflow's duration. For example: alert when dataflow has been running 15min, 1hr, etc. longer than the average for the dataflow. Having the functionality of…
-
Join Condition in ETL
1.For Some scenarios we will join tables based on the between dates condition with out columns mapping Eg select a.*,b.* from Table1 a , Table2 b where Date between date1 and date2 2.For Some scenarios we will join tables without any conditions Eg select a.*,b.* from Table1 a , Table2 b
-
Select All Columns in Remove Duplicates ETL Tile
Instead of having to add each column individually, it would be great if there was a select all columns feature for the remove duplicates step in the ETL.
-
Dataflow or View Query to run AFTER user filter pages
Sometimes it can be really difficult to build an end dataset that will be able to calculate the metrics that my users care about on the fly in a beastmode. For example: This contains 3 different fixed functions so that I can show a certain metric dynamically to the user. Because of the high computation (mixed with a big…
-
Colorize Column Headers in MagicETL Previews based on Connecting Line Color
With complex flows, it can sometimes be difficult to know what column came from what connecting source. It would be nice if MagicETL colorized column headers based on the connected line colors when in preview mode. As you can see in my beautiful artwork below (🤣), note the task I highlighted in yellow. That is the task I…
-
UUID / GUID Functions in Rank and Window
There are times where I may need to generate a unique ID so I can tie related batches of records together under said ID without worrying much about collisions with another group of related records in the same dataset. This will also allow me to treat them as a partition key if I want to target a handful of records for…
-
Magic ETL function for URL decode
I have logs that contain urls where I would like to parse out parameters and their values. In order to decode them today I need to use python scripting which runs into memory limitations. Please add a function for url decode so I can avoid the overhead of scripting.
-
"Documentation Mode" in Magic ETL/SQL
A toggle next to "Try the new Magic ETL" would read "Documentation Mode" and would: Put an opaque film over the whole Magic ETL map Disallow changes to the tiles Enable the ability to draw and color-code sections of the ETL Be able to notate what each section is doing An additional item in the Save menu would be "Save and…
-
Ability to save and easily reuse commonly used strings of logic within MagicETL/SQL
It would be really nice, when creating or editing a Magic ETL, to be able to easily reuse sequences of logic that could be easily adapted to any environment. Recursive append loops Rank and filter Split column and dynamic pivot The option to select which saved item you want to use would appear in the Magic ETL edit menu on…
-
Block tiles in Magic ETL to simplify complex dataflows
"Block Tiles" are larger tiles that contain a string of tiles inside. Block tiles could make large Magic ETLs (100+ tiles) easier to document, and much easier to read and understand. These tiles would contain a group of tiles that have a single output tile (not an output dataset) to make for an easy way to connect block…
-
Temp card in MagicETL on Preview data
I would love an interstitial card builder that I can build on preview data at different steps in an ETL. Often to ensure I'm filtering, or joining as expected I need to run whole dataflows or put aggregates to get counts (and list of unique categories) to check changes or where an issue is occuring in longer, more complex…
-
ETL Field Search Capability
It would be helpful in an ETL to be able to search for a field name and have the tiles that directly reference that field name be highlighted. Many times when supporting dataflow development we are looking for a field and we have to open every tile to see where that field is referenced - in join, in formulas, in filters,…
-
ETL | Alter Columns Tile | 'add select all columns'
For magic ETL, it would be lovely if we could have Alter Columns behave similarly as the Select columns having the ability to Add All columns. Thank you! Kristine Plunkett
-
More info in logs
It would be very useful if, when executing a Domo ETL, a log file was generated to see the details of potential failures in the process. Currently, if there is a failure in any of the Domo's own boxes, you see that the process has been interrupted in a part of the process but we don't see the details. Also, if the process…
-
Split Column by Number of Characters
Can we add an option to split a column by the number of characters in the field? I.e. split column by number of characters as far left or as far right. This is a very useful feature in PBI Power Query Editor especially when you have strange date formats like 1231201 or concatenated fields with an order number of a specific…
-
Allow for Min & Max in the Rank & Window Tile in Magic ETL
The minimum and maximum aggregation are standard calculations that should be included in the rank and window tile but currently are not. They should be added to avoid having to use a group by and then joining back to the dataset for simplicity and efficiency.
-
Add jump points to ETLs to clean up cluttered ETLs
When I create an ETL I do my best to make sure that all logical paths are kept together and that paths from one tile to another don't cross other paths if possible. Sometimes, however, I can't help but end up with a rat's nest of paths that clutter up the ETL and make it hard to follow. I would like to request jump points…
-
Magic ETL History Error Log Visual Representation
It would be great if in the history of the dataflow it gave some sort of visual representation on which FAILED had an actual error message I normally have to look at each one to find the one with the actual error
-
Magic ETL Organization Tiles
Would be nice to have a feature to add a box around several tiles in order to organise better the dataflow. Sometimes we have a complex dataflow with several business logic and a lot of tiles, and it would be easier to identify the several steps with that boxes. An example of that is like the one in Unreal Engine:
-
Pass Variables into a MagicETL from Workflows.
With the new workflow features on the horizon, I think it would be immeasurably valuable to pass output variables from workflows into Magic ETLs since workflows can execute ETLs already. For example: An 'audit' workflow is triggered that evaluates a dataset for various criteria. As part of the evaluation it sets output…
-
Add Hints to Deduplicate Tile
When I ingest the customer table from our POS, for example, and compare individual customer records to the last ingestion, I want to keep the older of the two records if selected fields show the records to be duplicates otherwise. The Deduplicate tile is the obvious choice to use, but appears to choose at random which…
-
Option to add prefix in Magic ETL Joins
The Pandas library and Power Query have this option, it's the option to add the table name as a prefix when joining tables. Huge benefits when working with system-generated datasets for example that can have 40+ columns with similar names in each table. The workarounds mentioned here don't quite cut it - if you're used to…
-
Sequence containers for magic etl or data flows
anyway of creating sequence containers for magic etl or dataflows we have many many dataflows and you can easily get lost. Since you cannot group them in a project folder. A sequence container like in ssis were we can group all the etls would be ideal.
-
Actual Run time listed in Lineage Tab
The last run time shows as “hours ago”. This needs to be the actual time the process ran so tracking down issues is much easier.
-
Allow Window Functions in MagicETL Formula Tile
It would be helpful to be able to write MagicETL formulas using window functions, similar to in beast mode. While the Rank & Window tile is often sufficient, there are some use cases where having this feature in the Formula tile would simplify the dataflow steps: Using a sum window function with unbounded rows This does…
-
Allowing comments on ETL schedule changes
We frequently have to disable automation on our Magic ETLs when troubleshooting issues. Currently there is no commenting option as exists when you save edits to the ETL logic. We need to track for auditing purposes why we disabled and then re-enabled automation and need to tie it to an incident number. If we had the…