MD5 or SHA1 (or similar) encoding in Adrenaline Dataflows


related to

Sometimes users will use MD5 or SHA1 encoding to create a surrogate key for application in dimensional modelling-informed workflows.

in our use case we needed to create a dimensional table by perform a set of operations on a distinct set of values spread across several columns.

the dataset itself was 50M+ rows, the number of distinct values was around 20k. we need MD5 or SHA1 encoding to create a surrogate key between the transactions and the set of distinct values.

Currently this pipeline starts in Magic (generate the surrogate key) -> Adrenaline (perform the ETL -> Reprocess the data in Magic so we can Accumulate history using Partitions

Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨‍💻

**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"
2 votes

Active · Last Updated