Recommendation Needed - Import NetSuite Transaction Data By Last Import

brow318

I am looking for the most effecient approach to import NetSuite sales orders into a Dataset. I was able to get data imported with a SQL query with no issues. However, the dataset refresh is reloading all order data every time it runs.

Is there a method where I can filter on createddate on the NetSuite transaction to be > that the last time the Domo import process started? Only need to import hundreds or couple thousand records a day. Seems improper to load over 1M rows of data to just get most recent additions.

ArborRose

If you store the last successful created date in a Domo variable, dataset or other….you could use that as part of your SQL query

-- Get the last loaded date from the tracking dataset
SELECT MAX(loaded_date) as last_loaded_date
FROM last_import_tracking;


-- Use that date to filter new records from NetSuite
SELECT *
FROM sales_order
WHERE createddate > (SELECT MAX(loaded_date) FROM last_import_tracking);

ABauman

@ArborRose Could you expound on this more, I am looking to do the same thing.

ArborRose

In December, I made my New Year's Resolution to stay off the leader board in this forum {sad face}.

My response to the question was that you can identify the most recent load date with a query, and use that to filter your query so it's increment (only gets the newest information).

I haven't looked at any of the NetSuite connectors in a minute, but I believe you can use incremental loading in the Domo NetSuite connector. When you set the data pull, use the `createddate` field as a filter by setting it to be greater than a specific timestamp (last import start or end date).

I'd have to think through a process. Something to store the timestamp of the last run. So after every successful import, you track the date. Then when configuring, choose Append instead of replace.