is it possible to create generic transforms available to all datasets/dataflows?

I know this has been asked before, in different ways and different contexts, but I am hoping to create a general (ie object-oriented) framework that standardizes common ETL tasks across multiple datasets.

I know I can create a Magic ETL dataflow which applies a common transform across multiple inputs and gives multiple outputs, but that tends to look quite messy when things get large. The data lineage becomes very confusing, plus I still have to set up each individual data transform by hand. What I really want is the ability to call a general User Defined Function (UDF) in a transform.

Using redshift or mysql, I can almost do that now. I can create a UDF as one step of my tranform, and call it in later steps, which is cool. What I really need, though, is a UDF stored centrally and independently so that I can call it from any transform using any dataset. Any ideas on how to implement that?

Find more posts tagged with

Accepted answers

This is a great idea, and has been mentioned before (like in this thread)

I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data. You could do it programatically, just not self-contained in Domo yet, is my thinking.

All comments

This is a great idea, and has been mentioned before (like in this thread)

I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data. You could do it programatically, just not self-contained in Domo yet, is my thinking.

Laurette

Thanks. I haven't played in the API yet, but I was wondering if that was the way to go. Sounds like it's time to take things to the next level. More time involved, but fun.

I imagine that might look like an external application that utilizes the endpoints Domo provides. Then you can schedule that application with its procedures to run the transforms. The devil's in the details, for sure, but Domo does provide a pretty open pathway.