ETL using Append object not showing all records

Options
Jessica
Jessica Contributor
in Magic ETL

Has anyone noticed that all the other object/functions in ETL in the preview tab show the full number of records except Append Rows.

 

In a current ETL I am building and troubleshooting, I was trying to find out where data is dropping off in error and I got to the Append Rows object I was using and it is by default showing only 100 records.  I tested to make sure this is not where my error is occurring by creating a Test Output object directly after the Append object and the test dataset accurately includes the 2400 records I expected.

 

It makes it difficult to test when the preview function is not working accurately.

Comments

  • MarkSnodgrass
    MarkSnodgrass Coach
    Options

    I have found that I can't trust the Preview window, regardless of how many rows I tell it to be in the preview. The row amount you choose (10k, 100k, etc.) only affects how many rows it will pull from your input datasets. It doesn't affect how many rows will show in the tile preview windows. Unfortunately, this means that you will need to run the actual ETL and look at your output dataset to know for sure if your tiles are configured correctly. 

    **Check out my Domo Tips & Tricks Videos

    **Make sure to <3 any users posts that helped you.
    **Please mark as accepted the ones who solved your issue.
  • jaeW_at_Onyx
    jaeW_at_Onyx Coach
    Options

    .. it's not that Preview isn't showing your preview accurately, it shows a max of 100 rows by design.  That's just ... how it works.

     

    What you can do is FILTER some of your data before you get to the APPEND tile.  If you know you're interested in a specific date, customer, or amount, apply the filter and then look.

     

    Also, an APPEND will pull all the data from Dataset A and then UNION Input B.  If you're worried about a specifici input .... why not preview the data further upstream?

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories