Data update schedules + Append method

user07549
user07549 Member
in Connectors

Can anyone help on the two point five issues I'm having:

  1. I have a connector that was set to run hourly.  It runs at 5 minutes past the hour which coincides with a lot of other internal processes and is causing errors in our data based systems....so I need to move the scheduled update to a quieter part of the hour (say 40 minutes past the hour)....however when set the schedule to 'Manual' to stop it, leave it for a couple of hours and then turn it back onto Hourly (at 40 minutes past the hour) it continues to update on its previous time of 5 minutes past the hour....I even tried Advanced scheduling/Update immediately but didnt see any different.....How do I move the minute of the hour that an hourly scheduled update runs on
  2. I have a dataset with half a million rows of which only perhaps a few hundred might change every hour....the large dataset seems to cause issues for Domo (IO speed Im guessing....it takes 5.2 seconds to run on the SSMS/SQL server console but it takes 10 minutes 32 seconds to run via Domo....this seems like a a good candidate for using the APPEND update method rather than the default replace method.....however when I set it over to APPEND the number of updated rows remains the same and the query time doesnt budge....Can anyone stop anything Im doing wrong with APPEND updates/point me in the direction of where to find this.

 

2.5 A subpoint on the above.....5 seconds on the server and over ten minutes on Domo...our CTO is a little less than happy with Domo's impact on his systems...Is there anywhere where I can diagnose the bottleneck/identify Domo connector query optimisations....I've got to agree with them, the run time difference is a little excessive....I notice Domo uses JDBC to connect to SQL Server which in my experience has been slower than other methods (ODBC/SQL Server native etc)...any ideas?

Best Answer

  • Unknown
    Unknown
    Answer ✓

    These are questions that are best suited with opening a ticket with Support. The Dojo is designed to be a community for the user base. By opening a ticket with Domo Support we can look specifically at your environment and get you a better response. 

    To speak to number 1 This is because the Dataset does not run every hour based on when you start it. It will run sometime during that hour. This is why you are seeing the behavior your seeing on the dataset. 

    As to the other two questions in general terms when you run the query on your client it is running natively in a local environment. When you run it through a connector in Domo you are going through a driver and through the internet and then it has to be parsed and indexing into our environment there are many more influencers and factors on speed and much more happening than running a query natively on the sql server. 

    With the append,  if you are not changing the query then I would not expect the data pulled to change. Switching it to append will just append the data that is brought in. Please reach out to Domo Support for more assistance on this. 

    Thanks 
    Justin 

Answers

  • Chronos
    Chronos Member

    Is anyone able to help out with this request?

  • Unknown
    Unknown
    Answer ✓

    These are questions that are best suited with opening a ticket with Support. The Dojo is designed to be a community for the user base. By opening a ticket with Domo Support we can look specifically at your environment and get you a better response. 

    To speak to number 1 This is because the Dataset does not run every hour based on when you start it. It will run sometime during that hour. This is why you are seeing the behavior your seeing on the dataset. 

    As to the other two questions in general terms when you run the query on your client it is running natively in a local environment. When you run it through a connector in Domo you are going through a driver and through the internet and then it has to be parsed and indexing into our environment there are many more influencers and factors on speed and much more happening than running a query natively on the sql server. 

    With the append,  if you are not changing the query then I would not expect the data pulled to change. Switching it to append will just append the data that is brought in. Please reach out to Domo Support for more assistance on this. 

    Thanks 
    Justin 

  • user07549
    user07549 Member

    Thanks Justin.

     

    I think that you're right re raising a ticket.  The general explanation makes sense but I'll raise a ticket to get some specifics on how to fix/optimise for my circumstance.

     

    Appreciate the detailed response.

     

    Thanks!

Categories