Magic ETL

Magic ETL Canvas Settings in bulk
The ability to turn on Larger Grid and Grid Lines to all of a user's ETLs (my preference) or user selected ETLs in Magic. Currently users have to do this ETL by ETL. My belief is that if a user wants the grid enabled for one of their ETLs, they would probably want it enabled for all of them.
Conditional Gateway Tile
Greetings, I would like to propose the addition of a tile that enables early termination of a dataflow without writing to the output dataset when specific conditions are met. A key use case for this feature would be terminating the dataflow if the input dataset contains zero rows. In such cases, the process should end, and…
Group By tile not working with large dataset
Hi, I have a dataset of about 4.7M rows. When I try to use the Group By tile in Magic ETL to find a sum of values across IDs, and then join back on the ID, the new sum column is blank. However, in testing, when I filtered the dataset down to about 50K rows to see what was wrong, the example ID I was checking now had the…
Integer-Based Column Reordering in Select Columns Tile
Please add an option to manually reorder columns in the Select Columns tile by entering an integer position value next to each column—similar to reordering mechanisms seen in interfaces like Steam’s Wishlist or Spotify's playlist editor. This would allow users to explicitly control column output order by position, not just…
Comment out tiles in MagicETL
When building complex ETLs it would be handy to be able to "comment out" tiles so they are ignored for execution. For bonus points, I'd like to be able to have an unterminated chain of commented-out tiles not block the ETL's execution.
Calculating YTD for each week in table
Hi, I want to replicate a table that looks like this: My dataset currently has date, impressions, and goal, and in ETL I've mapped the dates to a week by joining with another table. How would I use beast mode to find the impressions to date for each week, as well as the delivery percentage of goal?
Fix ROUND() in ETL
It was noticed that ROUND() function rounds down in ETL when the last decimal is 5. It acts correctly in the card's beast mode.
Magic ETL Dataflow Schedule: add trigger for Dataset has not updated
The Dataflow schedule has a setting to trigger when input datasets have updated. It would be helpful if there was a way to schedule it run at a certain time if an input has not updated. There are occasionally issues with Workbench that cause input datasets to not run, but in some cases we have recursive dataflows…
Joins: Automatic Identification of Relationship Direction
Please add: Automatic identification of relationship direction within a Join tile. An option to amend this relationship direction (where appropriate) within the Join tile. E.G A drop-down with 1-2-1,1-2-Many, Many-2-1, Many-2-Many (inc warning) e.t.c A visible representation of this relationship direction with ERD arrows…
Options for Preview Run Limits in Magic ETL
The Magic ETL currently only max out at 400k row for the preview run from your input datasets. This is limiting the test preview in your flow as it only picks (from my understanding) the first 400k row of your dataset. So if you're trying to preview your actions (such as filtering) you may not get anything showing in the…
Prevent 2 People form Editing Same ETL at Same Time
When someone is editing an ETL, so no let another person enter edit mode on that same ETL.
Magic ETL Filter Question
Hi, I have two columns: Academic Plan (ACAD_PLAN) and Effective Date (EFFDT). Currently, I am using the Rank & Window and Filter tiles to retrieve the most recent EFFDT and, consequently, the latest ACAD_PLAN. However, when I apply the Rank & Window tile, sort EFFDT in descending order, and filter for Row Number = 1, I…
pivot tile?
hi I am working in Magic ETL and am having a tough time knowing how to do what I am trying to do. Below I have a dataset with 3 columns shown that I need. What I need to be able to do is keep the msp last run date on the left hand side but then pivot the `delinquent payment 2` into column names then show the loan count So…
Can I use Python libraries in Domo?
Hey everyone, I’d like to use scipy.optimize in Domo to perform optimization calculations (similar to Excel Solver). Since MagicETL has a Python Tile, is this where I should implement it? Thank you!
Equivalent to Excel Solver in Domo MagicETL?
Hi everyone, I'm working on a hotel lodging analysis and need to find something in Domo MagicETL with comparable capabilities to Excel’s Solver function. In Excel, we’ve used Solver to optimize the number of daily hotel room blocks (reservations made in advance) to minimize last-minute (adhoc) room bookings, reducing extra…
How to Filter for "Today AND Last 7 Days" in Domo Visualizer?
Hi everyone, I’m trying to implement a date range in Domo (using `Reservation Start Date`) that includes both Today AND the Last 7 Days at the same time. This will be for a weekly scheduled send. I know Domo has built-in filters for "Last 7 Days" and "Today," but I can't find a way to combine them in a single filter within…
New ETL Tile
It would be great to see an etl tile which can be placed before outputs of ETL which give the option to set different update frequencies for different outputs of the same ETLs.
Extract multiple sets of numbers from a single string
I am trying to extract two different sets of numbers from a string. I have tried substring and Regexp, but to no avail. Examples of the stringss- Possible string format #1 Company Name - 2024-07-01-2024-07-31 - GPPC - YTUBE - VYT - 63784 - 2782837 - 458573682 - [745-000121-003] Possible string format #2 Company Name -…
Help figuring out how to get a data set to work with a funnel
For context the data set image linked is a representation of the data I am working with to protect our real customer data. I do not have the ability to reformat the data coming in, only use magic ETL and SQL transformations to convert this data into the format it needs to be in. I can not find a way to make the funnels…
ETL fails due to "This data set does not support data version filtering."
I have my SQL raw dataset set to merge new rows which is working correctly: This SQL dataset is the input for my ETL attempting to process New rows then Upsert in the output dataset: However my ETL fails to run due to the stated error: Am I setting something up wrong? My raw SQL data is not partitioned but it is using the…
Magic ETL Column Search Improvements
The new Column Search feature is a godsend. But … it could be better with two modifications: Add a Match String checkbox to limit the search to match the text string exactly. I was searching for the source of a field in a very large ETL. It turned out that the name of the one I wanted (countAsCar, for example) is included…
Data Join Confusion - Magic ETL
Hi, I'm trying to join two sets of data together (assignments submission data & assignment summary data) For the output, I am trying to get a list of all assignments with a due date in a date range that have not been submitted by the students. The assignment submission data has a row for every student that lists every…
Failed to run dataset due to error from the bery beginning node
I have a dataset that failed to run due to an error from the very beginning node. It said "Failed to convert value '5785586.012810127' from type 'String' to type 'Date'." However, that input itself runs normally and the data looks legit. I checked, no abnormal and no value of 5785586 anywhere. I tried to change all…
How to calculate Months between two dates with DATEDIFF?
I was trying to use the formula DATEDIFF(month,date1,date2) to calculate the months between two dates based on the function details that shows up in Magic ETL (which I have pasted below). In the Unit form, the difference is returned in the specified units from the first argument until the second. For example, DATEDIFF(day,…
How should I change my calculation in the Filter Rows Tile in ETL?
Hi, I am using the Filter Row Tile in the Magic ETL. I want to filter the Overall Credits Column and only get the maximum for each student. However, the MAX and FIXED BY don't work. How should I fix this? I also tried the Add Formula Tile and it did not work either. MAX(Overall Credits)FIXED(BY Student ID)
Python Tile fillna on integer column keep return 0 unless I change the column to text
I have a column Bill-To (interger) I use Python Tile to run this script to fill the nulls cell with the previous cell value #Import the domomagic package into the script from domomagic import * #read data from inputs into a data frame input1 = read_dataframe('Select Columns') #write your script here input1['Bill-To'] =…
Do I have the right code in my Python Script?
Hi, Select Columns 2 has four columns, they are EFFDT, ACAD_PLAN, DESCR, and ACAD_CAREER. I wrote a Python script to filter the EFFDT column so that it only includes rows where the value is 'A'. However, when I try to join this dataset with another, I receive an error stating that the referenced column, which is the…
Why is it creating duplicate rows?
Hi, my first dataset has 21,683 rows, and the second dataset has 323 rows. However, when I joined them together, the resulting dataset has 31,045 rows. I noticed that there are duplicate rows. How can I fix this? Thank you!
Display Record Count in Magic ETL on each step
Tl;dr - display the record count in MagicETL on each step of the last successful run of the ETL. Detail: I often spend quite some time diagnosing and fixing ETLs. In larger ETLs, it can be very onerous to determine whether a record with an expected behaviour makes it through the entire flow to the end to validate the ETL.…
Display Row count at every step in Magic ETL
Currently if you want to know your row count in Magic ETL at any stage other than the out put, you have to place a few temporary tiles to get there. I.E. add a rank and window tile to establish a Row Number column then aggregate with a group by to get the max row. Would really love if each tile could show the rows…

Magic ETL

Discussion List