Add only new records

Options
Shawn_S
Shawn_S Member
in Magic ETL

Hello there.

I have a Jira connector set up that pulls users (Name, Email, Jira ID). However, it only pulls currently active.

One of the dashboards I am creating needs this data, but we also need to keep users who leave.

My thought was to export the data today, then import as a data file. Then, starting tomorrow, append any new records from the original source. I just can't seem to get this to work.

Any suggestions?

Answers

  • GrantSmith
    GrantSmith Coach

    I'd recommend importing your historical file into Domo, then creating an ETL to simply output your historical dataset to a dataflow dataset. Then you can remove the historical input and convert your magic ETL to be a recursive dataflow to then update any new records or update any records in your existing dataflow dataset not found in your daily export to not they're no longer active. You can read up on recursive dataflows here: https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL

    One key note make sure the historical dataset has the same schema as the daily JIRA connector dataset you're pulling in and the columns are named the same.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories