How to create a recursive dataflow with multiple unique dimensions?

Options
Stephanie_Price
edited January 18 in Magic ETL

I am working with a dataset now that several unique components, including date. Within a given day, there are about 12-15 new records produced based on these dimensions.

I tried creating a unique identifier that concatenates these dimensions (including date) with the hopes of creating a unique ID that will distinguish old data vs new data; however, the field is too long and as a result is truncated.

Attached is an example of what the data looks like everyday.

Has anyone else run into this/have solutions to this?

Thank you!

Best Answer

  • GrantSmith
    GrantSmith Coach
    Answer ✓
    Options

    Are you attempting to utilize a single field in your recursive dataflow join for your key field? You should be able to select multiple key columns which uniquely identify each record. Just make sure you're joining on all of your key columns in the join

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓
    Options

    Are you attempting to utilize a single field in your recursive dataflow join for your key field? You should be able to select multiple key columns which uniquely identify each record. Just make sure you're joining on all of your key columns in the join

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • Stephanie_Price
    Options

    Thank you @GrantSmith