Odd issue with ETL and missing data

Options
NathanDorsch
NathanDorsch Member
in Magic ETL

I have an ETL I'm working in which data from one of the input datasets is not being pulled through when I run the ETL. Oddly however, I can see the data in preview. In particular, the "Test 1" output dataset will show data when I view it in preview (see screenshot below), but the resulting dataset does not contain any data.

I am absolutely perplexed and growing increasingly frustrated with this. Has anyone come across a similar issue to this?

image.png

Best Answer

  • NathanDorsch
    NathanDorsch Member
    edited October 2023 Answer ✓
    Options

    @ColemenWilson I finally discovered the issue from Domo support. Within the ETL, I had set up a filter by batch coming in from the input dataset…. not sure why the preview would still show the data, so apparently this is only applied when actually running the ETL.

    Thanks again for jumping in a call with me the other day!

    image.png

Answers

  • ColemenWilson
    ColemenWilson Coach
    edited September 2023
    Options

    My first thought would be that the column is being removed in the Append Rows 1 step. You could switch it to "include all columns" if it isn't already and see if it comes through.

    EDIT: re-read your post and to clarify, you are saying that for the test 1 output `Month` is not in the output?

    If I solved your problem, please select "yes" above

  • NathanDorsch
    NathanDorsch Member
    Options

    thanks for the suggestion @ColemenWilson , but I do have (all) the append rows steps set to "include all columns".

  • ColemenWilson
    ColemenWilson Coach
    Options

    If it is in the output dataset preview it should be in the output. I'd be happy to jump on a call with you and try and troubleshoot: https://calendly.com/cwilsondomo/15min

    If I solved your problem, please select "yes" above

  • NathanDorsch
    NathanDorsch Member
    Options

    Thank you @ColemenWilson for willing to jump on a call with me! I scheduled something on Monday - appreciate you offering to help!

  • NathanDorsch
    NathanDorsch Member
    edited October 2023 Answer ✓
    Options

    @ColemenWilson I finally discovered the issue from Domo support. Within the ETL, I had set up a filter by batch coming in from the input dataset…. not sure why the preview would still show the data, so apparently this is only applied when actually running the ETL.

    Thanks again for jumping in a call with me the other day!

    image.png

  • ColemenWilson
    ColemenWilson Coach
    Options

    No problem! Glad support got it figured out and this is good to know. I assumed since the data was showing in the preview this wouldn't be applied. I will check this troubleshooting in the future!!

    If I solved your problem, please select "yes" above

Categories