UUID / GUID Functions in Rank and Window

DomoDork
DomoDork Contributor
edited August 2023 in Magic ETL Ideas

There are times where I may need to generate a unique ID so I can tie related batches of records together under said ID without worrying much about collisions with another group of related records in the same dataset. This will also allow me to treat them as a partition key if I want to target a handful of records for reprocessing in situations where a normal key like dates would cause a ton more reprocessing of records and more overhead.

It would be nice to have this option in Rank and Window tiles so I could easily generate unique IDs when needed. Sure, you could do this with row numbers, or concatenating but that doesn't guarantee unique keys on historical data if row counts change or there arent enough fields in your dataset to concatenate to make a good unique key in the first place.

In my scenario, I have an ETL that generates/updates a dataset that one of our apps also reads from and updates. Having a UUID/GUID across batches of my records on the ETL side to act as a key of sorts would allow the App in AppDB to know which records belong together for processing and storing on its side and vice versa. It just allows for more clean conversation between two processes that work on the same records.

Tagged:
1
1 votes

Active · Last Updated