MAGIC ETL/DATAFLOWS Q&A from Domopalooza

AnnaYardley

The product team enjoyed answering questions from the audience during the final session at Domopalooza. Below you will find the answers to all questions related to Magic ETL and Dataflows (including those that weren't mentioned on stage).

Q: When are you going to upgrade the MySQL version in MySQL DataFlows?
A: We do not have plans to upgrade the MySQL version in MySQL DataFlows, but rather leverage the new SQL Tile in Magic ETL to fulfill SQL needs (including windowing and analytical functions). The SQL Tile is currently in a limited beta. Watch for a broader beta this summer.

Q: Partitioning is great in some cases, but it's not always granular enough to meet my data pipeline needs. When will Upsert be available?
A: Upsert is currently available in a limited beta, with a broader beta in the coming months. Reach out to your account team for steps to get access to the Upsert beta.

Q: My DataFlow seems slow. What are some simple best practices I can use to improve pipeline performance?
A: First, make sure you are using Magic ETL - it is our premier transform product and will perform better than MySQL or Redshift DataFlow options. Second, think through your data update strategy - are you processing more data with each DataFlow run than you need to? Look into whether Partitioning (available in GA now) or Upsert (in beta) would decrease the need to process so much data each time.

Q: What about using AI to arrange and de spaghetti a Magic ETL dataflow?
A: We're excited about all the potential for interplay between AI and Magic ETL. This is something we've been considering as part of that.

Q: Is there plan to increase the number of rows in the preview window in ETL?
A: We are considering several authoring enhancements for Magic ETL over the coming year, including an ability to have an authoring query for input datasets to help you load the rows you want to make authoring easier.

Q: Are there any plans to allow sections in ETLs? Either boxing tiles together or color coding an area where you can attach a note to the entire section would really help with organizing our dataflows.
A: This is a very common request that we've been thinking and talking about. Look for general upgrades to the Magic ETL authoring experience in the coming year.

Q: Why can't you "select all" in the remove duplicates tile in Magic ETL, especially when you can already in a group by?
A: This is a feature request we've been hearing, and we'll be reviewing it. Stay tuned for updates in the coming year.

Q: When can we all get a dark theme DOMO!? Especially inside the ETL/SQL environment/pages?
A: SQL Tile in Magic ETL (beta soon) has dark mode for the editor. Dark mode for the rest of the ETL editor is something we're considering.

Q: I have a security concern. The version of MySQL Domo currently uses is no longer supported by Oracle. When can we expect to see a supported version of MySQL in Domo?
A: Please be assured that we use sandboxing technology for safeguarding your MySQL DataFlows. You don't share a MySQL instance with any other DataFlow executions. We're excited to have you use SQL tile in MagicETL and have migration assistance to help you migrate there.

Q: Could we get dataflow preview data sorted to most recent data so we can see some matching rows for our hundred million row datasets?
A; More control over the set of input rows loaded when authoring in Magic ETL is on our radar as another quality-of-life improvement.

Q: I use Magic ETL a lot and have a situation that calls for a larger complex layouts in the workflow, but the size limits of the workflow area lead to messy complexity. Can we remove those size limits?
A: Great feedback. Grouping of tiles is on our radar as an enhacement that can help here. Increasing or removing the size restrictions on the ETL layout is also on our radar.

Q: Can we change the append tile to default to "use all rows"
A: Good feedback. We've added it to our list for quality-of-life enhancements for Magic ETL.

Q: Can we PLEASE add grid lines to Magic ETL? Having squares that are the same size as the tiles to make them easier to line up in neat rows both up and down and left to right with other tiles would make life so much easier. Thank you. :)
A: This is a common request and one that we've been evaluating. Stay tuned for quality of life upgrades like this in the coming year.

Have any follow-up questions? Post them in the comments below. A product team owner is closely moderating this post and will respond promptly to your questions.