Recommendation Needed - Import NetSuite Transaction Data By Last Import

brow318
brow318 Member
in Connectors

I am looking for the most effecient approach to import NetSuite sales orders into a Dataset. I was able to get data imported with a SQL query with no issues. However, the dataset refresh is reloading all order data every time it runs.

Is there a method where I can filter on createddate on the NetSuite transaction to be > that the last time the Domo import process started? Only need to import hundreds or couple thousand records a day. Seems improper to load over 1M rows of data to just get most recent additions.

Answers

  • ArborRose
    ArborRose Coach

    If you store the last successful created date in a Domo variable, dataset or other….you could use that as part of your SQL query 

    -- Get the last loaded date from the tracking dataset
    
SELECT MAX(loaded_date) as last_loaded_date
    
FROM last_import_tracking;


    
-- Use that date to filter new records from NetSuite
    
SELECT *
    
FROM sales_order
    
WHERE createddate > (SELECT MAX(loaded_date) FROM last_import_tracking);

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • ABauman
    ABauman Member

    @ArborRose Could you expound on this more, I am looking to do the same thing.

  • ArborRose
    ArborRose Coach

    In December, I made my New Year's Resolution to stay off the leader board in this forum {sad face}.

    My response to the question was that you can identify the most recent load date with a query, and use that to filter your query so it's increment (only gets the newest information).

    I haven't looked at any of the NetSuite connectors in a minute, but I believe you can use incremental loading in the Domo NetSuite connector. When you set the data pull, use the `createddate` field as a filter by setting it to be greater than a specific timestamp (last import start or end date).

    I'd have to think through a process. Something to store the timestamp of the last run. So after every successful import, you track the date. Then when configuring, choose Append instead of replace.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Categories