Pagination issue with JSON No Code Connector (API PennyLane)

Hello,
I'm trying to import a PennyLane table containing 58,000 rows into Domo via the JSON connector. The PennyLane API uses a pagination system where each query retrieves 100 rows at a time.
My connector works correctly without pagination (I retrieve the first 100 rows). However, as soon as I enable pagination and run the full import, it runs indefinitely (I waited 25 minutes before stopping, when it shouldn't take more than 5 minutes).
The API uses a pagination system based on two elements returned in each response:
- has_more (boolean): Indicates whether there are more results to retrieve (True means another query is required).
- next_cursor (string | null): Cursor to include in the cursor parameter of the next query. If next_cursor is null, it means there is no more data to retrieve.
The problem is that the connector only allows queries to be based on next_cursor, or it returns None at the end, which could prevent the process from stopping.
🔹 How do I configure Domo to stop queries when has_more = False?
🔹 Is there a better approach to correctly handle this pagination?
Thanks for your help!
Here is my setup:
Answers
-
Hi @Makam - I reviewed the documentation for that API and I see that the "List Ledger Entries" and "List Ledger Entry Lines of a Ledger Entry" endpoints use a page based pagination method rather than next token. As a workaround could you try first getting all Ledger Entries, and then with a second JSON connector loop through those entries and get the lines for each (using the dynamic URL option)?
1 -
Hello ggenovese,
This method worked very well, thank you for your help!
1
Categories
- All Categories
- 2K Product Ideas
- 2K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 310 Workbench
- 7 Cloud Amplifier
- 9 Federated
- 3K Transform
- 114 SQL DataFlows
- 654 Datasets
- 2.2K Magic ETL
- 4K Visualize
- 2.5K Charting
- 797 Beast Mode
- 78 App Studio
- 44 Variables
- 757 Automate
- 188 Apps
- 480 APIs & Domo Developer
- 72 Workflows
- 17 DomoAI
- 40 Predict
- 17 Jupyter Workspaces
- 23 R & Python Tiles
- 408 Distribute
- 119 Domo Everywhere
- 279 Scheduled Reports
- 10 Software Integrations
- 141 Manage
- 137 Governance & Security
- 8 Domo Community Gallery
- 47 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 114 Community Announcements
- 4.8K Archive