is it possible to create generic transforms available to all datasets/dataflows?

Laurette
Laurette Member
in Magic ETL

I know this has been asked before, in different ways and different contexts, but I am hoping to create a general (ie object-oriented) framework that standardizes common ETL tasks across multiple datasets. 

 

I know I can create a Magic ETL dataflow which applies a common transform across multiple inputs and gives multiple outputs, but that tends to look quite messy when things get large.  The data lineage becomes very confusing, plus I still have to set up each individual data transform by hand.  What I really want is the ability to call a general User Defined Function (UDF) in a transform.

 

Using redshift or mysql, I can almost do that now.  I can create a UDF as one step of my tranform, and call it in later steps, which is cool.  What I really need, though, is a UDF stored centrally and independently so that I can call it from any transform using any dataset.  Any ideas on how to implement that? 

 

Best Answer

  • AS
    AS Coach
    Answer ✓

    This is a great idea, and has been mentioned before (like in this thread)

    I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data.  You could do it programatically, just not self-contained in Domo yet, is my thinking.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Answers

  • AS
    AS Coach
    Answer ✓

    This is a great idea, and has been mentioned before (like in this thread)

    I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data.  You could do it programatically, just not self-contained in Domo yet, is my thinking.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • Laurette
    Laurette Member

    Thanks.  I haven't played in the API yet, but I was wondering if that was the way to go.  Sounds like it's time to take things to the next level.  More time involved, but fun. 

  • AS
    AS Coach

    I imagine that might look like an external application that utilizes the endpoints Domo provides.  Then you can schedule that application with its procedures to run the transforms.  The devil's in the details, for sure, but Domo does provide a pretty open pathway.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories