row count at each step

ydigilov
ydigilov Member
in Magic ETL Ideas

You should be able to see the # of rows output after each step of the etl flow.

13
Up
13 votes

In Review · Last Updated

I love this idea, and it's something I've been thinking about as well. I will review with the team.

Comments

  • AndreaHenderson
    AndreaHenderson Domo Product Manager

    I love this idea, and it's something I've been thinking about as well. I will review with the team. I had a follow-up question - would you want to see these row counts while you're authoring the DataFlow (as part of the Preview process), or after the DataFlow runs in the DataFlow Execution Details?

    Domo Product Manager for Data Transformation (MagicETL)

  • jessdoe
    jessdoe Contributor

    Hi @AndreaLovesData, I'm not sure which option @ydigilov would prefer but for me, having this ability while authoring the dataflow as part of the previous process would be awesome.

  • jessdoe
    jessdoe Contributor

    Being able to see row counts without having to run the ETL would allow you to more quickly ad easily see errors in your etl build-out.

  • ydigilov
    ydigilov Member

    @AndreaLovesData Just seeing now that you asked that question. I don't care if I have to hit run preview or if it automatically shows - if that's what you're asking. I just hate to have to output a new dataset at each step to see how many rows there are. I would hope that at least after hitting run preview, it would give us how many rows are in each step.

  • Recluz
    Recluz Member

    I just came on to add this. It would be so helpful to make sure filters, remove duplicates, and joins are behaving as expected. Especially in consumption models where you don't want to run it and then find out there is a problem.

  • WHM
    WHM Contributor

    why does the SQL tile with a select count(*) as cnt from table not work? it just causes "an unknown error occurred". How are you supposed to trouble shoot in preview without knowing the row count?

  • NateBI
    NateBI Contributor
    edited March 12

    See the below idea for a similar issue for joins.

    I agree the Magic ETL hides really valuable information that supports testing and without experience they're easy to forget - or manually include based on experience and create a huge spaghetti mess - as ETLs become more complex.

    https://community-forums.domo.com/main/discussion/69221/joins-automatic-identification-of-relationship-direction
  • AndreaHenderson
    AndreaHenderson Domo Product Manager

    @WHM It looks like that's a bug. An engineer on my team has coded a fix for it and it will be rolled out within the next week. Thanks for spotting that for us!

    Domo Product Manager for Data Transformation (MagicETL)

Categories