Data Processing using Partition

Options
scpradhan
scpradhan Member
in Magic ETL

Why I am having data loss issue when using data partition in Domo ETL?

Answers

  • MichelleH
    MichelleH Coach
    Options

    @scpradhan Could you please share some details of how your ETL is set up?

  • ArborRose
    ArborRose Coach
    Options

    You need to make sure the data partition is configured correctly. And make sure the partition key accurately segments your data without omitting relevant data.

    One of the things I watch closely in Domo ETL processing is duplicate records affecting joins. Number of rows can easily go whack. Normally expanding not loss. With less records I would recheck things like filters and formulas. And possibly calculations affected by NULLs. If values come out as null, the calculations could vacate. Review the use of COALESCE.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • scpradhan
    scpradhan Member
    Options

    @MichelleH basically, in the ETL, we are pulling only the new data being appended in the input dataset and the partition is done by the date.

Categories