How do I add a timestamp to the new rows that are coming in through my recursive magic etl?

tsplucky
tsplucky Member
in Magic ETL

I currently have a recursive magic etl set up to 'screenshot' data that is different from the source data. This information has been helpful because we can see the historical data any time a row in a column is changed. Like keeping a data log of all changes. However, I need to add a timestamp each time the recursive magic etl runs so I know the order in time each row comes in as new. Could you help me by providing a screenshot of a recursive data flow that includes how to add the timestamp?

Tagged:

Best Answer

  • ggenovese
    ggenovese Coach
    edited January 31 Answer ✓

    It depends on your goal, if you want to preserve the timestamp that was applied in an earlier run then you would place it directly after your non-recursive input dataset. This would apply the timestamp on the new data and preserve the timestamp on the recursive data.

    But if your goal is to overwrite any prior timestamps then you would place it right before your output. I would recommend using the add formula tile in this instance because you can overwrite an existing column, which you cannot do with the add constants tile.

    Hope that makes sense!

Answers

  • ggenovese
    ggenovese Coach

    You can use an Add Constants tile or an Add Formula tile, if you go the Add Formula route the the function you would use is called CURRENT_TIMESTAMP()

  • tsplucky
    tsplucky Member

    Hey ggenovese, do you know where in the recursive flow I should insert this formula tile?

  • ggenovese
    ggenovese Coach
    edited January 31 Answer ✓

    It depends on your goal, if you want to preserve the timestamp that was applied in an earlier run then you would place it directly after your non-recursive input dataset. This would apply the timestamp on the new data and preserve the timestamp on the recursive data.

    But if your goal is to overwrite any prior timestamps then you would place it right before your output. I would recommend using the add formula tile in this instance because you can overwrite an existing column, which you cannot do with the add constants tile.

    Hope that makes sense!

Categories