-
Long Running Dataflow Alerting/Auto-Restart
We need to have the ability to understand when dataflows are running longer than the average duration. We would like to set user defined alert criteria based upon the dataflow's duration. For example: alert when dataflow has been running 15min, 1hr, etc. longer than the average for the dataflow. Having the functionality of…
-
Join Condition in ETL
1.For Some scenarios we will join tables based on the between dates condition with out columns mapping Eg select a.*,b.* from Table1 a , Table2 b where Date between date1 and date2 2.For Some scenarios we will join tables without any conditions Eg select a.*,b.* from Table1 a , Table2 b
-
Dataflow or View Query to run AFTER user filter pages
Sometimes it can be really difficult to build an end dataset that will be able to calculate the metrics that my users care about on the fly in a beastmode. For example: This contains 3 different fixed functions so that I can show a certain metric dynamically to the user. Because of the high computation (mixed with a big…
-
Make variable/column names visible in FULL from drop-down choices in ETL
It is pretty bad UI not being able to see the variable names in full. We often can only see the first few words in the non-expandable windows and hover doesnt surface names either. It becomes total guess work selecting variables when you have lots if variables and/or similarly named variables. Dear devs, it's the simle…
-
Colorize Column Headers in MagicETL Previews based on Connecting Line Color
With complex flows, it can sometimes be difficult to know what column came from what connecting source. It would be nice if MagicETL colorized column headers based on the connected line colors when in preview mode. As you can see in my beautiful artwork below (🤣), note the task I highlighted in yellow. That is the task I…
-
UUID / GUID Functions in Rank and Window
There are times where I may need to generate a unique ID so I can tie related batches of records together under said ID without worrying much about collisions with another group of related records in the same dataset. This will also allow me to treat them as a partition key if I want to target a handful of records for…
-
Magic ETL function for URL decode
I have logs that contain urls where I would like to parse out parameters and their values. In order to decode them today I need to use python scripting which runs into memory limitations. Please add a function for url decode so I can avoid the overhead of scripting.
-
Multiple outputs for Python modules
Python boxes in the Domo ETLs allowing multiple outputs.
-
Ability to save and easily reuse commonly used strings of logic within MagicETL/SQL
It would be really nice, when creating or editing a Magic ETL, to be able to easily reuse sequences of logic that could be easily adapted to any environment. Recursive append loops Rank and filter Split column and dynamic pivot The option to select which saved item you want to use would appear in the Magic ETL edit menu on…
-
Temp card in MagicETL on Preview data
I would love an interstitial card builder that I can build on preview data at different steps in an ETL. Often to ensure I'm filtering, or joining as expected I need to run whole dataflows or put aggregates to get counts (and list of unique categories) to check changes or where an issue is occuring in longer, more complex…
-
ETL | Alter Columns Tile | 'add select all columns'
For magic ETL, it would be lovely if we could have Alter Columns behave similarly as the Select columns having the ability to Add All columns. Thank you! Kristine Plunkett
-
More info in logs
It would be very useful if, when executing a Domo ETL, a log file was generated to see the details of potential failures in the process. Currently, if there is a failure in any of the Domo's own boxes, you see that the process has been interrupted in a part of the process but we don't see the details. Also, if the process…
-
Split Column by Number of Characters
Can we add an option to split a column by the number of characters in the field? I.e. split column by number of characters as far left or as far right. This is a very useful feature in PBI Power Query Editor especially when you have strange date formats like 1231201 or concatenated fields with an order number of a specific…
-
Back Button Functionality
The back button from a dataset sends the user to the 'Data' tab. workaround: The back button from the browser (tested in Chrome) should behave as expected and send the user to the previous page they were on. Scenario: Selecting an ETL, clicking one of it's datasets to view the data. Selecting the Domo back button, only to…
-
Add Notes and Titled Containers to ETLs
I find that my ETLs are often big, with multiple endpoints; so when I come back it takes me some time to catch up with what I did. I think that adding some text notes to the ETL view would make it easier to evaluate what I, or one of my coworkers, have done. As for titled containers, they would allow a user to draw a…
-
Allow for Min & Max in the Rank & Window Tile in Magic ETL
The minimum and maximum aggregation are standard calculations that should be included in the rank and window tile but currently are not. They should be added to avoid having to use a group by and then joining back to the dataset for simplicity and efficiency.
-
Add jump points to ETLs to clean up cluttered ETLs
When I create an ETL I do my best to make sure that all logical paths are kept together and that paths from one tile to another don't cross other paths if possible. Sometimes, however, I can't help but end up with a rat's nest of paths that clutter up the ETL and make it hard to follow. I would like to request jump points…
-
Magic ETL History Error Log Visual Representation
It would be great if in the history of the dataflow it gave some sort of visual representation on which FAILED had an actual error message I normally have to look at each one to find the one with the actual error
-
Pass Variables into a MagicETL from Workflows.
With the new workflow features on the horizon, I think it would be immeasurably valuable to pass output variables from workflows into Magic ETLs since workflows can execute ETLs already. For example: An 'audit' workflow is triggered that evaluates a dataset for various criteria. As part of the evaluation it sets output…
-
Add Hints to Deduplicate Tile
When I ingest the customer table from our POS, for example, and compare individual customer records to the last ingestion, I want to keep the older of the two records if selected fields show the records to be duplicates otherwise. The Deduplicate tile is the obvious choice to use, but appears to choose at random which…
-
Can we get more packages added to Python Scripting tiles?
I could really use pandas-datareader in an ETL. I know it's available in Jupyter Workspace, but its availability in a Magic ETL would be convenient for me.
-
Option to add prefix in Magic ETL Joins
The Pandas library and Power Query have this option, it's the option to add the table name as a prefix when joining tables. Huge benefits when working with system-generated datasets for example that can have 40+ columns with similar names in each table. The workarounds mentioned here don't quite cut it - if you're used to…
-
Sequence containers for magic etl or data flows
anyway of creating sequence containers for magic etl or dataflows we have many many dataflows and you can easily get lost. Since you cannot group them in a project folder. A sequence container like in ssis were we can group all the etls would be ideal.
-
Actual Run time listed in Lineage Tab
The last run time shows as “hours ago”. This needs to be the actual time the process ran so tracking down issues is much easier.
-
Allowing comments on ETL schedule changes
We frequently have to disable automation on our Magic ETLs when troubleshooting issues. Currently there is no commenting option as exists when you save edits to the ETL logic. We need to track for auditing purposes why we disabled and then re-enabled automation and need to tie it to an incident number. If we had the…
-
Magic ETL can use Federated Dataset
Hi, We have number of cases where we want to use Magic ETL using federated Dataset. Can discuss them? Thank You. Best Regards, Tejal
-
Magic ETL - Undo Button
Can we add an Undo Button to Magic ETL. This would be especially helpful when we accidentally delete a tile when clicking to drag it.
-
Add Left- and Right-Anti-Joins to JOIN Tile in Magic ETL 2.0.
Ever need to isolate rows in Table A that are not in Table B or vice versa? Anti-Joins to the rescue! Except, there is not this option in the Join tile in Magic ETL 2.0. Instead, we have to use a formula tile with a Best Mode to set a comparison and tag rows and a filter tile to isolate the rows needed. Would be great if…
-
Tag dataflows automatically based on tags of output dataset?
Is it possibly to tag the dataflows automatically with the tags added to the output dataset? So basically whenever adding tags to a dataset, the dataflow that spits out that output you are tagging should be default also get the tags. Currently when creating a new dataflow, I navigate to the output dataset to tag it with a…
-
Handle NULL in Magic
It would be nice to have a tile, or an option in Alter or Join tiles to mass handle NULL values. I understand that I can use a formula for every column as follows but after about 3 columns it gets tedious. case when `col1` is NULL then '' else `col1` end