How to Use Partitioning to Retain Data in a Monthly Rolling Window Dataset

DataLawton · 2024-08-15T15:41:36+00:00

There was an error rendering this rich post.

Hi Domo Community,

I’m currently working with a dataset (via MongoDB Connector) that operates on a 12-month rolling window, where data older than 12 months is deleted on the 1st of each month. Long story short, I cannot change this configuration. But, I would like to retain some of this older data using partitioning before it gets deleted.

Would partitioning be the solution for this? Could anyone advise on the best approach to set up partitioning in Domo to achieve this? Specifically, I’m looking to keep data from certain months beyond the 12-month window while still allowing the rolling window to function as intended for the rest of the data.

Any guidance or step-by-step instructions would be greatly appreciated!

Thanks in advance for your help

Find more posts tagged with

Dataset

Magic ETL

Partitioning

Accepted answers

GrantSmith

You can configure a MagicETL dataflow Output Dataset tile to use a Partition update method and specify the month field as your key. This will keep around any data that is no longer being pulled in from your source dataset.

This is a beta feature so you may need to reach out to your account executive.

All comments

GrantSmith

This is a beta feature so you may need to reach out to your account executive.