SQL DataFlow vs API access
Hi DOJO Community,
Good morning. Do any of you know why when I am querying the API the runtime between two dates is about 2 minutes (querying 130k rows from a 16MM row dataset). But when I query it from SQL given a query using the SQL ETL DataFlow, it takes a very long time to run ( I terminated it each time)
Best Answer
-
The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Answers
-
Hi @Shane_Lester99 ,
Can you send a screenshot of the ETL? I bet you have one step that is the culprit and taking forever. May be a join or something.
John Le
You're only one dashboard away.
Click here for more video solutions:
0 -
SELECT * FROM `plg_small_parcel_recreated_invoice_prod` WHERE `Invoice Date` BETWEEN '2022-10-30' AND '2022-11-6'
Above is the query.
0 -
The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 616 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 738 Beast Mode
- 57 App Studio
- 40 Variables
- 685 Automate
- 176 Apps
- 452 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 124 Manage
- 121 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive