Custom connector - append all rows with an ID greater than last appended
I am making a custom connector. The data vendor’s api needs me to send the IDs of the rows I want to ingest, and they can then send up to 10-20k rows without a time-out. So, I need to break up my requests in blocks of 10k rows, and give the ID range each block. I can run this every hour, as each hour will be less than 10k rows, and append.
Question is, what’s the best way to query the existing table, grab the highest ID present, add one, then use this as a starting ID for the next pull?
Notes:
- the vendor has a version of the Api that works with dates, but even a single day can cause time outs.
- the feed is json
- I am using append, rather than replace
Comments
-
Can anyone help with this request?
Thanks,
0 -
I've just noticed in the Redshift Advanced connector, there's the ability to pass parameters into your SQL, including:
Enter the query parameter value, it is the initial value for query parameter. The last run date is optional by default it is '02/01/1700' if is not provided. For example: !{lastvalue:_id}!=1,!{lastrundate:start_date}!=02/01/1944
0 -
To be clear, this doesn't solve my issue, as my connector is JSON, not Redshift. Is there a way of achieving something similar with JSON?
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 299 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 616 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 730 Beast Mode
- 55 App Studio
- 40 Variables
- 682 Automate
- 175 Apps
- 451 APIs & Domo Developer
- 46 Workflows
- 10 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive