Has anyone tried to create a custom schedule for an ETL job?

CoreVest
CoreVest Member
in Connectors

I am trying to create a daily snapshot of our datasets regardless of whether the data change or not. The current ETL process I have created can do such, so long as the dataset in which I am pulling data is update. However if the data is never updated, the ETL is never kicked off, which can create a gap in our day to day snapshots. Anyone?

 

I have been told to create a separate dataset that can help trigger my jobs, but how would i schedule the new dataset to run on a specific time, let alone update the data it would be storing to kick off my ETL job? Anyone?

Best Answer

  • CoreVest
    CoreVest Member
    Answer ✓

    Thank you Godzilla but no. The solution i came up with is to create a datasource that consistently runs at a specific time, once created add that source into my MySQL ETL and select the box that runs the job when that data source is update. I hate using work arounds but seeing as to how we are not implementing workbence, it is my only option.

     

     

Answers

  • Godzilla
    Godzilla Contributor

    i'm assuming you're using the current version of workbench correct? if so, you may have to revert back to an older version of workbench to be able to schedule jobs at specific times. I'm not sure if version 3 allows to to control the times but i know version 2 does. I'm still using version 2 because of this. 

    Domo Arigato!

    **Say 'Thanks' by clicking the thumbs up in the post that helped you.
    **Please mark the post that solves your problem as 'Accepted Solution'
  • CoreVest
    CoreVest Member
    Answer ✓

    Thank you Godzilla but no. The solution i came up with is to create a datasource that consistently runs at a specific time, once created add that source into my MySQL ETL and select the box that runs the job when that data source is update. I hate using work arounds but seeing as to how we are not implementing workbence, it is my only option.

     

     

Categories