Best Of
MAGIC ETL/DATAFLOWS Q&A from Domopalooza
The product team enjoyed answering questions from the audience during the final session at Domopalooza. Below you will find the answers to all questions related to Magic ETL and Dataflows (including those that weren't mentioned on stage).
Q: When are you going to upgrade the MySQL version in MySQL DataFlows?
A: We do not have plans to upgrade the MySQL version in MySQL DataFlows, but rather leverage the new SQL Tile in Magic ETL to fulfill SQL needs (including windowing and analytical functions). The SQL Tile is currently in a limited beta. Watch for a broader beta this summer.
Q: Partitioning is great in some cases, but it's not always granular enough to meet my data pipeline needs. When will Upsert be available?
A: Upsert is currently available in a limited beta, with a broader beta in the coming months. Reach out to your account team for steps to get access to the Upsert beta.
Q: My DataFlow seems slow. What are some simple best practices I can use to improve pipeline performance?
A: First, make sure you are using Magic ETL - it is our premier transform product and will perform better than MySQL or Redshift DataFlow options. Second, think through your data update strategy - are you processing more data with each DataFlow run than you need to? Look into whether Partitioning (available in GA now) or Upsert (in beta) would decrease the need to process so much data each time.
Q: What about using AI to arrange and de spaghetti a Magic ETL dataflow?
A: We're excited about all the potential for interplay between AI and Magic ETL. This is something we've been considering as part of that.
Q: Is there plan to increase the number of rows in the preview window in ETL?
A: We are considering several authoring enhancements for Magic ETL over the coming year, including an ability to have an authoring query for input datasets to help you load the rows you want to make authoring easier.
Q: Are there any plans to allow sections in ETLs? Either boxing tiles together or color coding an area where you can attach a note to the entire section would really help with organizing our dataflows.
A: This is a very common request that we've been thinking and talking about. Look for general upgrades to the Magic ETL authoring experience in the coming year.
Q: Why can't you "select all" in the remove duplicates tile in Magic ETL, especially when you can already in a group by?
A: This is a feature request we've been hearing, and we'll be reviewing it. Stay tuned for updates in the coming year.
Q: When can we all get a dark theme DOMO!? Especially inside the ETL/SQL environment/pages?
A: SQL Tile in Magic ETL (beta soon) has dark mode for the editor. Dark mode for the rest of the ETL editor is something we're considering.
Q: I have a security concern. The version of MySQL Domo currently uses is no longer supported by Oracle. When can we expect to see a supported version of MySQL in Domo?
A: Please be assured that we use sandboxing technology for safeguarding your MySQL DataFlows. You don't share a MySQL instance with any other DataFlow executions. We're excited to have you use SQL tile in MagicETL and have migration assistance to help you migrate there.
Q: Could we get dataflow preview data sorted to most recent data so we can see some matching rows for our hundred million row datasets?
A; More control over the set of input rows loaded when authoring in Magic ETL is on our radar as another quality-of-life improvement.
Q: I use Magic ETL a lot and have a situation that calls for a larger complex layouts in the workflow, but the size limits of the workflow area lead to messy complexity. Can we remove those size limits?
A: Great feedback. Grouping of tiles is on our radar as an enhacement that can help here. Increasing or removing the size restrictions on the ETL layout is also on our radar.
Q: Can we change the append tile to default to "use all rows"
A: Good feedback. We've added it to our list for quality-of-life enhancements for Magic ETL.
Q: Can we PLEASE add grid lines to Magic ETL? Having squares that are the same size as the tiles to make them easier to line up in neat rows both up and down and left to right with other tiles would make life so much easier. Thank you. :)
A: This is a common request and one that we've been evaluating. Stay tuned for quality of life upgrades like this in the coming year.
Have any follow-up questions? Post them in the comments below. A product team owner is closely moderating this post and will respond promptly to your questions.
Re: How do you create multi-select controls?
@sfinnegan You wouldn't be able to do it because if multiple inputs were selected, how would your beast mode/equation know what the result should be? Ex: Using numbers as an example, if you had to evaluate if x=1, and you set x equal to 1 and 2 at the same time, your evaluation wouldn't know if the result is true or false. What's your end goal, and maybe there's a workaround?
Re: Confluence Card
I attended a couple of talks that dealt with documentation at Domopalooza 24. The "Building a Platform as a Service Framework" talk mentioned going from Google Docs → GitHub → Confluence to publish documentation in Confluence, but I don't think they mentioned pushing the documentation into Domo. "Cultivating a Community of Empowered Users" also talked a lot about documentation, but I think the example there was to use a doc card within Domo itself for the documentation and not another system.
Sorry if this wasn't helpful, just figured I'd share my notes. Maybe there was another talk I missed that mentioned a process for pushing Confluence → Domo.
Forcasting Data by Zip for a 24 month period
I am looking to forecast mover data out 12 months looking at a 24 month period of historical data for each zip code in the US. Currently we run this with a spreadsheet and a forcasting formula. Looking for anyone who might have done this before or has any ideas how this can be done in Domo. Open to other tools other than Magic ETL
SSO between two apps
We have an app that uses the same SSO as our DOMO app. Is there a way to programmatically log into Domo at the same time as we log into our app without the user having to interact with any DOMO UI?
We want to add a card to our app that uses PDP.
rptgeek
Re: Beast Mode Summary Number HTML for filled Gauge
Thanks David. That was exactly what was happening. I was not seeing it reflected in the analyzer so I never saved it to view the displayed card.
Travis_A
Re: "The Pivot Table is Only Showing Part of the Data" Error
I set the number of rows to 25K from "Filter and Sort" in Analyzer and I was able to solve our problem!
Re: "The Pivot Table is Only Showing Part of the Data" Error
For future reference this is mentioned just above the "Customizing Pivot Tables" section in the knowledge base: https://domohelp.domo.com/hc/en-us/articles/360043429473-Pivot-Table#3. By default there's a limit of 25,000 rows to display. See the section " What are the DataSet row limits for each chart type?" on this article: https://domohelp.domo.com/hc/en-us/articles/360042924214-Card-Building-FAQs
Re: AirTable connector import thumbnails
I'd agree with @brycec, the Domo connector isn't flattening your data so it's returning List as the value instead of breaking out each of the values within the list in the JSON. Use the JSON No Code connector to format the data as you need.
Re: Previous Month Last Year
Add the beast mode to the filter section of your card and then filter for 'Yes'.