SQL DataFlow vs API access

Shane_Lester99
Shane_Lester99 Member
edited March 2023 in SQL DataFlows

Hi DOJO Community,

Good morning. Do any of you know why when I am querying the API the runtime between two dates is about 2 minutes (querying 130k rows from a 16MM row dataset). But when I query it from SQL given a query using the SQL ETL DataFlow, it takes a very long time to run ( I terminated it each time)

Best Answer

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • DashboardDude
    DashboardDude Coach

    Hi @Shane_Lester99 ,

    Can you send a screenshot of the ETL? I bet you have one step that is the culprit and taking forever. May be a join or something.

    Circle.png

    John Le

    Are you on my newsletter? If not, signup here so you don't miss out on my Domo tricks, alerts about my webinars, cooking tips and more

    Signup here:

    https://www.dashboarddudes.com/newsletter

  • Shane_Lester99
    Shane_Lester99 Member
    edited November 2022
    ETL.PNG

    SELECT * FROM `plg_small_parcel_recreated_invoice_prod` WHERE `Invoice Date` BETWEEN '2022-10-30' AND '2022-11-6'


    Above is the query.

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories