How to Use Partitioning to Retain Data in a Monthly Rolling Window Dataset

DataLawton
DataLawton Member
in Datasets

Hi Domo Community,

I’m currently working with a dataset (via MongoDB Connector) that operates on a 12-month rolling window, where data older than 12 months is deleted on the 1st of each month. Long story short, I cannot change this configuration. But, I would like to retain some of this older data using partitioning before it gets deleted.

Would partitioning be the solution for this? Could anyone advise on the best approach to set up partitioning in Domo to achieve this? Specifically, I’m looking to keep data from certain months beyond the 12-month window while still allowing the rolling window to function as intended for the rest of the data.

Any guidance or step-by-step instructions would be greatly appreciated!

Thanks in advance for your help

Tagged:

Best Answer

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You can configure a MagicETL dataflow Output Dataset tile to use a Partition update method and specify the month field as your key. This will keep around any data that is no longer being pulled in from your source dataset.

    This is a beta feature so you may need to reach out to your account executive.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Answers

  • GrantSmith
    GrantSmith Coach
    Answer ✓

    You can configure a MagicETL dataflow Output Dataset tile to use a Partition update method and specify the month field as your key. This will keep around any data that is no longer being pulled in from your source dataset.

    This is a beta feature so you may need to reach out to your account executive.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories