Pagination issue with JSON No Code Connector (API PennyLane)

Makam
Makam Member
in Connectors

Hello,

I'm trying to import a PennyLane table containing 58,000 rows into Domo via the JSON connector. The PennyLane API uses a pagination system where each query retrieves 100 rows at a time.

My connector works correctly without pagination (I retrieve the first 100 rows). However, as soon as I enable pagination and run the full import, it runs indefinitely (I waited 25 minutes before stopping, when it shouldn't take more than 5 minutes).

The API uses a pagination system based on two elements returned in each response:

  • has_more (boolean): Indicates whether there are more results to retrieve (True means another query is required).
  • next_cursor (string | null): Cursor to include in the cursor parameter of the next query. If next_cursor is null, it means there is no more data to retrieve.


The problem is that the connector only allows queries to be based on next_cursor, or it returns None at the end, which could prevent the process from stopping.

🔹 How do I configure Domo to stop queries when has_more = False?
🔹 Is there a better approach to correctly handle this pagination?

Thanks for your help!

Here is my setup:

image.png

Answers

  • ggenovese
    ggenovese Coach

    Hi @Makam - I reviewed the documentation for that API and I see that the "List Ledger Entries" and "List Ledger Entry Lines of a Ledger Entry" endpoints use a page based pagination method rather than next token. As a workaround could you try first getting all Ledger Entries, and then with a second JSON connector loop through those entries and get the lines for each (using the dynamic URL option)?

  • Makam
    Makam Member

    Hello ggenovese,

    This method worked very well, thank you for your help!

Categories