Does domo allow a dataflow to run for more than 24 hours

Nandha_Kumar_1

I have dataflow that gets failed when it tries to completely run a dataflow that runs more than a day.
Is there any solution. can the dataflow can run more than a day.

Can anyone help me with this problem?

MichelleH

@Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?

MichelleH

@Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?

GrantSmith

Hi @Nandha_Kumar_1

You only have 323k rows, so there are likely some chances for efficiency improvement. When I'm designing an ETL, I like to be a teapot—short and stout.

Short—Filter your data as soon as possible to reduce the number of rows. Use a select column or alter column to select or drop columns, respectively, so you're left with just the columns you need. This should also take into account your joins so that you don't accidentally do a giant cartesian join and exponentially increase the size of your datasets with your join.

Stout—Do many things at once. If possible, do several joins at the same time instead of chaining joins together. Don't just have a single flow of your data.

With that, you can also look at the execution history of the dataflow, click the success icon, and it'll show you the execution flow and path to give you a better understanding of which part of your pipeline is taking the longest to identify potential areas for efficiency gains.

Nandha_Kumar_1

https://community-forums.domo.com/main/discussion/comment/98984#Comment_98984

I am using SQL Dataflow to do this Transformations and i am using procedure while loops in the SQL. So when the data increases. The while loop runs all the rows.

Nandha_Kumar_1

https://community-forums.domo.com/main/discussion/comment/98986#Comment_98986

Hi, I am using SQL dataflow that uses procedures and functions to run through the while loop and where it runs through all the rows of the dataset.

MichelleH

@Nandha_Kumar_1 Can you tell us a little bit more about the purpose of the SQL dataflow and what the procedures and while loops? There may be alternative methods you can use to make the dataflow more efficient. @GrantSmith's guidelines are good best practices to apply regardless of which type of dataflow you are using.

GrantSmith

I'd recommend switching your logic over to a Magic ETL. MySQL runs on 5.6 which is an older version and due to the nature of those dataflows they run sequentially and can run much slower. I've seen Magic ETL dataflows with the same logic take up to 90% less time to execute.