Workbench Query Parameters

SamC
SamC Member
in Workbench

Is it possible to use query parameters in workbench?

Can the parameter receive it's value from a query on a domo dataset?

 

Here is what we are trying to do:

We have some large tables that we would like to refresh often. Currently they are uploaded using the Replace method.

We would like instead to use the Append method and push only data that is not yet in Domo.

Unfortuanly we have no way of tracking changes to our data from our side, however if we could get the MAX(Id) of one of the fields then it would be possible to know what to push.

 

The question is then: Can I query the Dataset in Domo first to get the MAX(Id) and use that in a paremeter in workbench to select our data where the Id > @parameter?

 

Thanks for the help!

Comments

  • Jarvis
    Jarvis Domo Employee

    Hey SamC,

    It sounds like you would benefit from a recursive dataflow in this situation. It will update as an Append, only bringing in new values, and also update existing values that may have changed since the previous import.

     

    If that sounds like something that would work for you, please check out the article for it here:

     

    https://knowledge.domo.com/Prepare/DataFlow_Tips_and_Tricks/Creating_a_Recursive%2F%2FSnapshot_ETL_DataFlow

     

    Thanks

     

    Jarvis

  • SamC
    SamC Member

    Hi Jarvice,

     

    Unfortunantly this is not possible given that we have now way of pushing only new values because there is no timestamp of any kind.

    But due to the data size we would like to avoid uploading it everyday.

    What would potentialy work is if workbench was able to first query the dataset to get the largest ID. Is that possible?

     

    Thanks,

    Sam

This discussion has been closed.

Categories