DataSet updated, Even though all inputs did not update

sgraham

I have a Card built to attempt to give me the status of when my Datasets have been updated.

 

I have a ETL called Dashboard status.

 

It has an input flow 'Headcount Status' and a add constant that adds a column to represent the current date and time.  Output is called 'Dashboard Status'

 

So in theory, if my input is updated, it will then update the ETL Dashboard status and result in a data set with the current date and time of last update.

 

So, now my issue.

 

My input flow 'Headcount Status'  is actually made up of 5 input datasets all set to Run transform when any of these DataSets are updated

 

3 of those 5 actually failed and did not run successfully since last week Friday May 5th.  However, the ETL 'Headcount Status'  still ran and subsequently ran my Dashboard status ETL.

 

So my card is now showing it updated today May 8th.

 

See what am getting at?

 

i thought unless all the input datasets run successfully, the ETL 'Headcount Status' would not run.  How do i ensure that?

PodiumMason

Hi there SGraham,

 

In ETL/Dataflow dependencies, the setting is such that if any of the dependent datasets update successfully, any ETL flow that is tied to those datasets will update. 

 

In your case, because 2 of the 5 dependent datasets did successfully update, the ETL ran and updated. 

 

Let me know if you have any additional questions!

 

 

sgraham

OK,  So any idea how I would be able to have someway of accomplishing what I'm trying?  Given that there are 5 datasets?

 

 

PodiumMason

I don't currently know of a way to adjust dependencies so that an ETL/Dataflow only runs after all the input datasets are updated. 

 

It would be useful to suggest this in the ideas exchange so that it can be added to the product.