SQL DataFlow vs API access
Hi DOJO Community,
Good morning. Do any of you know why when I am querying the API the runtime between two dates is about 2 minutes (querying 130k rows from a 16MM row dataset). But when I query it from SQL given a query using the SQL ETL DataFlow, it takes a very long time to run ( I terminated it each time)
Best Answer
-
The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Answers
-
Hi @Shane_Lester99 ,
Can you send a screenshot of the ETL? I bet you have one step that is the culprit and taking forever. May be a join or something.
John Le
You're only one dashboard away.
Click here for more video solutions:
0 -
SELECT * FROM `plg_small_parcel_recreated_invoice_prod` WHERE `Invoice Date` BETWEEN '2022-10-30' AND '2022-11-6'
Above is the query.
0 -
The MySQL dataflow runs slower because it must transfer 16MM rows to the MySQL server process it and then transfer the data back. Queuing via the api doesn’t need to transfer the data across systems so it’ll be much faster
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**1
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 297 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive