Export MagicETL logic

GrantSmith

With MySQL Data Flows it's easy to share the underlying code with other Domo users (either internally or on the Dojo) since it's simply a copy and paste of SQL code however there are some users who aren't as proficient with MySQL and prefer MagicETL. It'd be great if there was a way to export a MagicETL data flow (or even better allow for portions of a Magic ETL data flow to be exported). This would help with showing others how certain logic works and they could then import it directly into their data flow instead of relying on screen shots to attempt to duplicate which can be error prone.

MichelleH

This would be especially helpful for cases where multiple MagicETLs follow a similar sequence of transforms for a portion of the dataflow. Currently we have to either create a copy of the entire MagicETL and then change the rest of the logic, or rebuild that section from scratch. It would be nice to be able to copy a section of tiles to a different canvas, similar to the duplicate option that already exists.

timehat

While this isn't the full functionality you've requested, as a stop-gap you can select multiple tiles in MagicETL and the left sidebar will update to include a "Copy to Clipboard" option:

 

 

You can paste that into a new MagicETL flow or paste it to a text editor/send to a coworker. But know we understand the larger use case for reusing actions. ?

MichelleH

@timehat This is great, thank you! 

swagner

@GrantSmith I submitted a similar idea here:

https://dojo.domo.com/t5/Ideas-Exchange/Magic-ETL-Advanced-Editor-see-the-code-you-are-building-with-the/idi-p/46671

 

@MichelleH @timehat 

 

[Deleted User]

@jaeW_at_Onyx can you chime in on this!

Thanks!

jaeW_at_Onyx

you CAN use the javaCLI to export a JSON file of an ETL (or card, or dataset schema).

list-dataset -i <etl_id> -f <exportfile>

 

I use this to back up my work in github so i can do change comparisons OR to copy a complex SQL dataflow into another instance.

 

before i can copy to another instance i need to alter the json file's dataset_ids to a valid dataset_id in the new instance.

 

then i can use the javacli comannd set-dataflow-properties -c -n <name> -d <json file> to create a new dataflow!!

... i should make a video.

swagner

Thanks @jaeW_at_Onyx that would be a great topic for a video!

jaeW_at_Onyx

@user018556, Domo has extended the Java CLI with support for syncing with GitHub repositories.  Just download the most recent build of the CLI and type HELP you'll see a pair of Github commands

 

You should be able to sync object definitions from there and then use text processing or the version DIFF features in GitHub to monitor changes in a dataflow.

 

... that said, given that you want to use GitHub for tracking changes, i'm assuming you're writing SQL intensive dataflows. .. i would STRONGLY recommend you refactor for Magic 2.0 b/c it is consistently outperforming most SQL based data pipelines I've worked with.  And while Magic 2.0 doesn't necessarily address your versioning question, understanding  what a Magic 2.0 dataflow does tends to be much easier than reading a wall of someone else's poorly formatted (usually ? SQL.

 

Also Magic makes refactoring easier b/c it forces you to think in Logic Chuncks wheras in SQL it's all too easy to introduce several pieces of logic into one transform that obfuscate where the root problem is during troubleshooting.  But that's another convo for another day.

 

GLHF