Does domo allow a dataflow to run for more than 24 hours
I have dataflow that gets failed when it tries to completely run a dataflow that runs more than a day.
Is there any solution. can the dataflow can run more than a day.
Can anyone help me with this problem?
Best Answer
-
@Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?
0
Answers
-
@Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?
0 -
You only have 323k rows, so there are likely some chances for efficiency improvement. When I'm designing an ETL, I like to be a teapot—short and stout.
Short—Filter your data as soon as possible to reduce the number of rows. Use a select column or alter column to select or drop columns, respectively, so you're left with just the columns you need. This should also take into account your joins so that you don't accidentally do a giant cartesian join and exponentially increase the size of your datasets with your join.
Stout—Do many things at once. If possible, do several joins at the same time instead of chaining joins together. Don't just have a single flow of your data.
With that, you can also look at the execution history of the dataflow, click the success icon, and it'll show you the execution flow and path to give you a better understanding of which part of your pipeline is taking the longest to identify potential areas for efficiency gains.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1 -
I am using SQL Dataflow to do this Transformations and i am using procedure while loops in the SQL. So when the data increases. The while loop runs all the rows.
0 -
Hi, I am using SQL dataflow that uses procedures and functions to run through the while loop and where it runs through all the rows of the dataset.
0 -
@Nandha_Kumar_1 Can you tell us a little bit more about the purpose of the SQL dataflow and what the procedures and while loops? There may be alternative methods you can use to make the dataflow more efficient. @GrantSmith's guidelines are good best practices to apply regardless of which type of dataflow you are using.
0 -
I'd recommend switching your logic over to a Magic ETL. MySQL runs on 5.6 which is an older version and due to the nature of those dataflows they run sequentially and can run much slower. I've seen Magic ETL dataflows with the same logic take up to 90% less time to execute.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 296 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 99 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 727 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive