is it possible to create generic transforms available to all datasets/dataflows?
I know this has been asked before, in different ways and different contexts, but I am hoping to create a general (ie object-oriented) framework that standardizes common ETL tasks across multiple datasets.
I know I can create a Magic ETL dataflow which applies a common transform across multiple inputs and gives multiple outputs, but that tends to look quite messy when things get large. The data lineage becomes very confusing, plus I still have to set up each individual data transform by hand. What I really want is the ability to call a general User Defined Function (UDF) in a transform.
Using redshift or mysql, I can almost do that now. I can create a UDF as one step of my tranform, and call it in later steps, which is cool. What I really need, though, is a UDF stored centrally and independently so that I can call it from any transform using any dataset. Any ideas on how to implement that?
Best Answer
-
This is a great idea, and has been mentioned before (like in this thread)
I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data. You could do it programatically, just not self-contained in Domo yet, is my thinking.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1
Answers
-
This is a great idea, and has been mentioned before (like in this thread)
I'm not aware of a functionality that exists like this outside of using Domo's APIs to push and pull data. You could do it programatically, just not self-contained in Domo yet, is my thinking.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"1 -
Thanks. I haven't played in the API yet, but I was wondering if that was the way to go. Sounds like it's time to take things to the next level. More time involved, but fun.
0 -
I imagine that might look like an external application that utilizes the endpoints Domo provides. Then you can schedule that application with its procedures to run the transforms. The devil's in the details, for sure, but Domo does provide a pretty open pathway.
Aaron
MajorDomo @ Merit Medical
**Say "Thanks" by clicking the heart in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 297 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 616 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 54 App Studio
- 40 Variables
- 678 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 46 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive