Does domo allow a dataflow to run for more than 24 hours

Nandha_Kumar_1
Nandha_Kumar_1 Member
edited October 22 in SQL DataFlows

I have dataflow that gets failed when it tries to completely run a dataflow that runs more than a day.
Is there any solution. can the dataflow can run more than a day.

image.png

Can anyone help me with this problem?

Tagged:

Best Answer

  • MichelleH
    MichelleH Coach
    Answer ✓

    @Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?

Answers

  • MichelleH
    MichelleH Coach
    Answer ✓

    @Nandha_Kumar_1 Domo typically times out dataflows around the 24 hour mark. Though looking at the number of rows your dataflow is processing, there is likely opportunity to make your dataflow more efficient. What kind of transformations are you doing?

  • GrantSmith
    GrantSmith Coach

    Hi @Nandha_Kumar_1

    You only have 323k rows, so there are likely some chances for efficiency improvement. When I'm designing an ETL, I like to be a teapot—short and stout.

    Short—Filter your data as soon as possible to reduce the number of rows. Use a select column or alter column to select or drop columns, respectively, so you're left with just the columns you need. This should also take into account your joins so that you don't accidentally do a giant cartesian join and exponentially increase the size of your datasets with your join.

    Stout—Do many things at once. If possible, do several joins at the same time instead of chaining joins together. Don't just have a single flow of your data.

    With that, you can also look at the execution history of the dataflow, click the success icon, and it'll show you the execution flow and path to give you a better understanding of which part of your pipeline is taking the longest to identify potential areas for efficiency gains.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • Nandha_Kumar_1
    Nandha_Kumar_1 Member
    https://community-forums.domo.com/main/discussion/comment/98984#Comment_98984

    I am using SQL Dataflow to do this Transformations and i am using procedure while loops in the SQL. So when the data increases. The while loop runs all the rows.

  • Nandha_Kumar_1
    Nandha_Kumar_1 Member
    https://community-forums.domo.com/main/discussion/comment/98986#Comment_98986

    Hi, I am using SQL dataflow that uses procedures and functions to run through the while loop and where it runs through all the rows of the dataset.

  • MichelleH
    MichelleH Coach

    @Nandha_Kumar_1 Can you tell us a little bit more about the purpose of the SQL dataflow and what the procedures and while loops? There may be alternative methods you can use to make the dataflow more efficient. @GrantSmith's guidelines are good best practices to apply regardless of which type of dataflow you are using.

  • GrantSmith
    GrantSmith Coach

    I'd recommend switching your logic over to a Magic ETL. MySQL runs on 5.6 which is an older version and due to the nature of those dataflows they run sequentially and can run much slower. I've seen Magic ETL dataflows with the same logic take up to 90% less time to execute.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories