How to update field if value is different than existing?

BMilner
BMilner Member
in Magic ETL

Hello,

I feel like this should be straightforward, but I can't figure it out. During our master ETL, I have a lookup sheet that adds fields to the dataset (e.g. Account Owner, Region, Account tier (A, B, or C). However, if an account owner changes and I make a change to the lookup sheet, it's only updating new records since we are updating the dataset by batch. All prior records retain the old value. Is there an easy way to update all records without rerunning the entire dataset?

I'd even be fine with filtering the output dataset and remapping, but it looks like that applies the change to all records and not just the filtered rows.

Sorry, very new to this.

Thanks, Brian

Best Answer

  • ST_Superman
    ST_Superman Domo Employee
    Answer ✓

    @BMilner

    I would recommend joining this data in using a dataset view. These joins happen much faster than ETLs (near instant). The downside here would be that you would need to change any cards or content you have built to a new dataset.

Answers

  • ColemenWilson
    ColemenWilson Coach

    Pretty sure you'll have to re-run the entire dataset. Is there a reason you are against doing that one time to cleanup the account owners?

    If I solved your problem, please select "yes" above

  • BMilner
    BMilner Member

    Thanks for the response. Last time I tried to do that, it timed-out after 24 hours.

  • ColemenWilson
    ColemenWilson Coach

    Gotcha. Have you looked at using Magic ETL and/or Datafusion to speed things up and prevent a timeout?

    If I solved your problem, please select "yes" above

  • ST_Superman
    ST_Superman Domo Employee
    Answer ✓

    @BMilner

    I would recommend joining this data in using a dataset view. These joins happen much faster than ETLs (near instant). The downside here would be that you would need to change any cards or content you have built to a new dataset.

Categories