Magic ETL

Magic ETL

Saving States

Is there a way to save data and possibly append to it? Let me give an example. When I pull data from a database, I would like to pull data and when I pull data tomorrow I want the original data to still be there. So I want to be able to see user counts by region today, and then in a month I would still like to see what it was a month ago. Is there a way to do this in Domo? Right now when I run ETL's everything just overwrites.

Tagged:

Best Answers

  • Coach
    Answer ✓

    @mroker It depends on how you are pulling the data, but most connectors should have an option to change the update method from "Replace" to "Append" so it simply adds new rows to the existing data.

  • Coach
    Answer ✓

    Appends in ETLs aren't publicly available currently. You can use a recursive dataflow to do an append. Here's some more information on recursive dataflows:

    Site faviconCustomer Support Community

    Alternatively if you don't care about de-duping data and only want to take snapshots at a specific time interval you can also utilize the DataSet Copy connector and set that update method to append to take snapshots of your dataset.

    Site faviconCustomer Support Community

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • Coach
    Answer ✓

    @mroker It depends on how you are pulling the data, but most connectors should have an option to change the update method from "Replace" to "Append" so it simply adds new rows to the existing data.

  • Coach
    Answer ✓

    Appends in ETLs aren't publicly available currently. You can use a recursive dataflow to do an append. Here's some more information on recursive dataflows:

    Site faviconCustomer Support Community

    Alternatively if you don't care about de-duping data and only want to take snapshots at a specific time interval you can also utilize the DataSet Copy connector and set that update method to append to take snapshots of your dataset.

    Site faviconCustomer Support Community

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In