DataSet updated, Even though all inputs did not update

Options
sgraham
sgraham Member
in Archive

I have a Card built to attempt to give me the status of when my Datasets have been updated.

 

I have a ETL called Dashboard status.

 

It has an input flow 'Headcount Status' and a add constant that adds a column to represent the current date and time.  Output is called 'Dashboard Status'

 

So in theory, if my input is updated, it will then update the ETL Dashboard status and result in a data set with the current date and time of last update.

 

So, now my issue.

 

My input flow 'Headcount Status'  is actually made up of 5 input datasets all set to Run transform when any of these DataSets are updated

 

3 of those 5 actually failed and did not run successfully since last week Friday May 5th.  However, the ETL 'Headcount Status'  still ran and subsequently ran my Dashboard status ETL.

 

So my card is now showing it updated today May 8th.

 

See what am getting at?

 

i thought unless all the input datasets run successfully, the ETL 'Headcount Status' would not run.  How do i ensure that?

Comments

  • PodiumMason
    PodiumMason Contributor

    Hi there SGraham,

     

    In ETL/Dataflow dependencies, the setting is such that if any of the dependent datasets update successfully, any ETL flow that is tied to those datasets will update. 

     

    In your case, because 2 of the 5 dependent datasets did successfully update, the ETL ran and updated. 

     

    Let me know if you have any additional questions!

     

     

    **Say 'Thanks' by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as 'Accepted Solution'
  • sgraham
    sgraham Member

    OK,  So any idea how I would be able to have someway of accomplishing what I'm trying?  Given that there are 5 datasets?

     

     

  • PodiumMason
    PodiumMason Contributor

    I don't currently know of a way to adjust dependencies so that an ETL/Dataflow only runs after all the input datasets are updated. 

     

    It would be useful to suggest this in the ideas exchange so that it can be added to the product. 

     

     

    **Say 'Thanks' by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as 'Accepted Solution'
This discussion has been closed.

Categories