Custom connector - append all rows with an ID greater than last appended

I am making a custom connector. The data vendor’s api needs me to send the IDs of the rows I want to ingest, and they can then send up to 10-20k rows without a time-out. So, I need to break up my requests in blocks of 10k rows, and give the ID range each block. I can run this every hour, as each hour will be less than 10k rows, and append.
Question is, what’s the best way to query the existing table, grab the highest ID present, add one, then use this as a starting ID for the next pull?
Notes:
- the vendor has a version of the Api that works with dates, but even a single day can cause time outs.
- the feed is json
- I am using append, rather than replace
Welcome!
Comments
-
Can anyone help with this request?
Thanks,
0 -
I've just noticed in the Redshift Advanced connector, there's the ability to pass parameters into your SQL, including:
Enter the query parameter value, it is the initial value for query parameter. The last run date is optional by default it is '02/01/1700' if is not provided. For example: !{lastvalue:_id}!=1,!{lastrundate:start_date}!=02/01/1944
0 -
To be clear, this doesn't solve my issue, as my connector is JSON, not Redshift. Is there a way of achieving something similar with JSON?
0
Welcome!
Welcome!
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 308 Workbench
- 6 Cloud Amplifier
- 10 Federated
- 3.8K Transform
- 660 Datasets
- 117 SQL DataFlows
- 2.2K Magic ETL
- 815 Beast Mode
- 3.3K Visualize
- 2.5K Charting
- 84 App Studio
- 46 Variables
- 780 Automate
- 191 Apps
- 482 APIs & Domo Developer
- 84 Workflows
- 23 Code Engine
- 41 AI and Machine Learning
- 21 AI Chat
- 2 AI Projects and Models
- 18 Jupyter Workspaces
- 414 Distribute
- 122 Domo Everywhere
- 281 Scheduled Reports
- 11 Software Integrations
- 146 Manage
- 142 Governance & Security
- 8 Domo Community Gallery
- 49 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 115 Community Announcements
- 4.8K Archive