for loop in Large amount of Data in Workbench.
Hello all,
I'm tiring to pull large amount of data via Workbench, previously this was done with SSIS for loop container pulling only fraction of the time frame data at a time, and appending the results.
is there way to achieve this in workbench as I'm not able to find the proper documentation.
Answers
-
The partition feature in Workbench should help you accomplish it. Here is a link to the KB article.
https://domo-support.domo.com/s/article/360062446514?language=en_US
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
Thank you Mark, i have tried it, but its not very detailed and i'm a little bit confused.
i'm tried to pull data for the last 14 days, via partition but it fails, if I pull for a single day its works ok.
looking into the logs, it seems to me like its try to pull the whole thing.my date column shows results as 10/3/2023 12:00:00 AM
Data Type: DATETIMEpart of my query that define the time frame.
WHERE a.SK_DATE_ID BETWEEN TO_NUMBER(TO_CHAR(TRUNC(SYSDATE-14), 'YYYYMMDD')) AND TO_NUMBER(TO_CHAR(TRUNC(SYSDATE), 'YYYYMMDD'))0 -
Committing upload '2' for execution '2' and stream '5095' with update method 'APPEND' and data tag '2023-10-01'
[10.04.23 5:42:25 AM] Beginning upload of part 1 for execution 2 and upload 3.
[10.04.23 5:42:25 AM] Creating upload for execution '2' and stream '5095'
[10.04.23 5:42:25 AM] Upload 3 upload progress: 100%
[10.04.23 5:42:26 AM] Committing upload '3' for execution '2' and stream '5095' with update method 'APPEND' and data tag '2023-10-02'
[10.04.23 5:42:26 AM] Beginning upload of part 1 for execution 2 and upload 4.
[10.04.23 5:42:26 AM] Creating upload for execution '2' and stream '5095'
[10.04.23 5:42:26 AM] Beginning upload of part 1 for execution 2 and upload 5.
[10.04.23 5:42:26 AM] Upload 4 upload progress: 100%
[10.04.23 5:42:26 AM] Committing upload '4' for execution '2' and stream '5095' with update method 'APPEND' and data tag '2023-10-03'
[10.04.23 5:42:26 AM] Upload 5 upload progress: 100%
[10.04.23 5:42:26 AM] Committing upload '5' for execution '2' and stream '5095' with update method 'APPEND' and data tag '2023-09-30'
its odd that i can pull data for 4 days, but when I extend to the full range of 14 days it times out, like i don't try to pull the data 1 by 1 but actually try to preview it at once and it times out.0 -
if anyone come across this post, just FYI: WB tries to pull 100k records each time, and have some defaults time out. If the query aggregate to much data, it might stop the request due to timeout , we workaround it via pulling the row data, and do the aggregation in Domo
0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 295 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 608 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 709 Beast Mode
- 49 App Studio
- 39 Variables
- 667 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 44 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 115 Manage
- 112 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive