Magic ETL

Comment out tiles in MagicETL
When building complex ETLs it would be handy to be able to "comment out" tiles so they are ignored for execution. For bonus points, I'd like to be able to have an unterminated chain of commented-out tiles not block the ETL's execution.
Fix ROUND() in ETL
It was noticed that ROUND() function rounds down in ETL when the last decimal is 5. It acts correctly in the card's beast mode.
Magic ETL Dataflow Schedule: add trigger for Dataset has not updated
The Dataflow schedule has a setting to trigger when input datasets have updated. It would be helpful if there was a way to schedule it run at a certain time if an input has not updated. There are occasionally issues with Workbench that cause input datasets to not run, but in some cases we have recursive dataflows…
Joins: Automatic Identification of Relationship Direction
Please add: Automatic identification of relationship direction within a Join tile. An option to amend this relationship direction (where appropriate) within the Join tile. E.G A drop-down with 1-2-1,1-2-Many, Many-2-1, Many-2-Many (inc warning) e.t.c A visible representation of this relationship direction with ERD arrows…
Options for Preview Run Limits in Magic ETL
The Magic ETL currently only max out at 400k row for the preview run from your input datasets. This is limiting the test preview in your flow as it only picks (from my understanding) the first 400k row of your dataset. So if you're trying to preview your actions (such as filtering) you may not get anything showing in the…
Prevent 2 People form Editing Same ETL at Same Time
When someone is editing an ETL, so no let another person enter edit mode on that same ETL.
Magic ETL Filter Question
Hi, I have two columns: Academic Plan (ACAD_PLAN) and Effective Date (EFFDT). Currently, I am using the Rank & Window and Filter tiles to retrieve the most recent EFFDT and, consequently, the latest ACAD_PLAN. However, when I apply the Rank & Window tile, sort EFFDT in descending order, and filter for Row Number = 1, I…
pivot tile?
hi I am working in Magic ETL and am having a tough time knowing how to do what I am trying to do. Below I have a dataset with 3 columns shown that I need. What I need to be able to do is keep the msp last run date on the left hand side but then pivot the `delinquent payment 2` into column names then show the loan count So…
Can I use Python libraries in Domo?
Hey everyone, I’d like to use scipy.optimize in Domo to perform optimization calculations (similar to Excel Solver). Since MagicETL has a Python Tile, is this where I should implement it? Thank you!
Equivalent to Excel Solver in Domo MagicETL?
Hi everyone, I'm working on a hotel lodging analysis and need to find something in Domo MagicETL with comparable capabilities to Excel’s Solver function. In Excel, we’ve used Solver to optimize the number of daily hotel room blocks (reservations made in advance) to minimize last-minute (adhoc) room bookings, reducing extra…
How to Filter for "Today AND Last 7 Days" in Domo Visualizer?
Hi everyone, I’m trying to implement a date range in Domo (using `Reservation Start Date`) that includes both Today AND the Last 7 Days at the same time. This will be for a weekly scheduled send. I know Domo has built-in filters for "Last 7 Days" and "Today," but I can't find a way to combine them in a single filter within…
New ETL Tile
It would be great to see an etl tile which can be placed before outputs of ETL which give the option to set different update frequencies for different outputs of the same ETLs.
Extract multiple sets of numbers from a single string
I am trying to extract two different sets of numbers from a string. I have tried substring and Regexp, but to no avail. Examples of the stringss- Possible string format #1 Company Name - 2024-07-01-2024-07-31 - GPPC - YTUBE - VYT - 63784 - 2782837 - 458573682 - [745-000121-003] Possible string format #2 Company Name -…
Help figuring out how to get a data set to work with a funnel
For context the data set image linked is a representation of the data I am working with to protect our real customer data. I do not have the ability to reformat the data coming in, only use magic ETL and SQL transformations to convert this data into the format it needs to be in. I can not find a way to make the funnels…
ETL fails due to "This data set does not support data version filtering."
I have my SQL raw dataset set to merge new rows which is working correctly: This SQL dataset is the input for my ETL attempting to process New rows then Upsert in the output dataset: However my ETL fails to run due to the stated error: Am I setting something up wrong? My raw SQL data is not partitioned but it is using the…
Magic ETL Column Search Improvements
The new Column Search feature is a godsend. But … it could be better with two modifications: Add a Match String checkbox to limit the search to match the text string exactly. I was searching for the source of a field in a very large ETL. It turned out that the name of the one I wanted (countAsCar, for example) is included…
Data Join Confusion - Magic ETL
Hi, I'm trying to join two sets of data together (assignments submission data & assignment summary data) For the output, I am trying to get a list of all assignments with a due date in a date range that have not been submitted by the students. The assignment submission data has a row for every student that lists every…
Failed to run dataset due to error from the bery beginning node
I have a dataset that failed to run due to an error from the very beginning node. It said "Failed to convert value '5785586.012810127' from type 'String' to type 'Date'." However, that input itself runs normally and the data looks legit. I checked, no abnormal and no value of 5785586 anywhere. I tried to change all…
How to calculate Months between two dates with DATEDIFF?
I was trying to use the formula DATEDIFF(month,date1,date2) to calculate the months between two dates based on the function details that shows up in Magic ETL (which I have pasted below). In the Unit form, the difference is returned in the specified units from the first argument until the second. For example, DATEDIFF(day,…
How should I change my calculation in the Filter Rows Tile in ETL?
Hi, I am using the Filter Row Tile in the Magic ETL. I want to filter the Overall Credits Column and only get the maximum for each student. However, the MAX and FIXED BY don't work. How should I fix this? I also tried the Add Formula Tile and it did not work either. MAX(Overall Credits)FIXED(BY Student ID)
Python Tile fillna on integer column keep return 0 unless I change the column to text
I have a column Bill-To (interger) I use Python Tile to run this script to fill the nulls cell with the previous cell value #Import the domomagic package into the script from domomagic import * #read data from inputs into a data frame input1 = read_dataframe('Select Columns') #write your script here input1['Bill-To'] =…
Do I have the right code in my Python Script?
Hi, Select Columns 2 has four columns, they are EFFDT, ACAD_PLAN, DESCR, and ACAD_CAREER. I wrote a Python script to filter the EFFDT column so that it only includes rows where the value is 'A'. However, when I try to join this dataset with another, I receive an error stating that the referenced column, which is the…
Why is it creating duplicate rows?
Hi, my first dataset has 21,683 rows, and the second dataset has 323 rows. However, when I joined them together, the resulting dataset has 31,045 rows. I noticed that there are duplicate rows. How can I fix this? Thank you!
Display Row count at every step in Magic ETL
Currently if you want to know your row count in Magic ETL at any stage other than the out put, you have to place a few temporary tiles to get there. I.E. add a rank and window tile to establish a Row Number column then aggregate with a group by to get the max row. Would really love if each tile could show the rows…
Display Record Count in Magic ETL on each step
Tl;dr - display the record count in MagicETL on each step of the last successful run of the ETL. Detail: I often spend quite some time diagnosing and fixing ETLs. In larger ETLs, it can be very onerous to determine whether a record with an expected behaviour makes it through the entire flow to the end to validate the ETL.…
Join in ETL
Question: I have two separate databases where there is only one common column, which is the "salesperson." The first database contains sales values for each salesperson. The second database contains the area of operation and products associated with each salesperson. I plan to create a new column to calculate the…
How do I add the current date to new rows that get appended by a recursive dataflow?
Hello, currently I have the standard recursive flow set up in the magic ETL, which tile and where in the recursive dataflow should I add to get the current date of when a new row gets added to the dataset?
Database Management View Instead of Datasets/ETLs view
Currently users view their assets in a list in the Data tab. They have to use naming conventions, or tags to group these assets and the same goes for ETLs. The list can be filtered and includes shared assets by default but regardless it quickly becomes a mess that only encourages redundant asset creation due to low…
Notes Tile in ETL
I know that we can make notes on every tile, but what I would love to see is a note tile where all it is, is a note. Sometimes I want to make generic notes in the ETL that relate to a few steps, or an overall process, or something relating to the business question not the tile specifically. So a notes tile would be great.…
Need help with specific code that is not working in either beast mode or magic etl formula tile
Hi the following code I am using as a filter on my card and have it set to "on list" its called skus to be purged "CASE WHEN On Order = 0 AND (Future Demand = '' OR Future Demand IS NULL) AND (Future Books = '' OR Future Books IS NULL) AND (Reserved + WMAVAIL <= 10 AND WMAVAIL > 0) AND WAREHOUSE_CODE = 11 THEN 'On List'…

Magic ETL

Discussion List