Snapshot Fields Accuracy

renee12345
renee12345 Member
in Magic ETL

I'd like to create a new dataflow, preferably not MySQL, to show that the fields I intend to snapshot daily are actually being snapshotted. For more context, I have a list of companies with subscription updates that are currently being snapshotted, but I want to ensure that everyday, every company is included. Anyone have advice on how to tackle this?

Best Answer

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You can use a recursive dataflow and use a formula tile to calculate a TIMESTAMP on when the ETL runs using CURDATE() as the formula

    Ideally if you can calculate a run TIMESTAMP within your input dataset it would be more accurate as to when the data was actually pulled

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • DashboardDude
    DashboardDude Contributor

    Hi @renee12345 ,

    What's the datasource you're pulling data from? I ask because if there's something like a Last Run Date or something like that, you can leverage that.

    John Le image

    John Le

    You're only one dashboard away.

    Click here for more video solutions: https://www.dashboarddudes.com/pantry

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You can use a recursive dataflow and use a formula tile to calculate a TIMESTAMP on when the ETL runs using CURDATE() as the formula

    Ideally if you can calculate a run TIMESTAMP within your input dataset it would be more accurate as to when the data was actually pulled

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • renee12345
    renee12345 Member

    Thank you.

    I have the timestamp value, but how would I use that value to see if the number of companies, for example are accurately being snapshotted daily?

Categories