Best Of
Re: How to remove columns with Null values?
You could try using a Dynamic Unpivot Tile, then a filter tile to remove any rows with a null value and then pivot the remaining data again.
You could also look into creating a dataset view of the data. When you are looking at the "Overview" for a dataset, you will see a section called "Data Profiler". This section will provide you with some recommended actions:
If you click on "VIEW ISSUE", you can create a view of your dataset that will remove any columns that are null:
Then just use this view as the input for your ETL.
The advantage to using the unpivot/pivot approach would be that it would be dynamic, meaning that if data did come through in one of the fields, the field would no longer be deleted. However, once you remove the field using the dataset view, then that view will never contain the field.
Re: Where do Cloned Beast Modes Come From?
@zcameron this might be something for your team?
@James_Dougherty Domo has ... a handful of quiet releases and sometimes doesn't update the documentation to reflect updated language in the product.
I think it's important to consider how Beast Modes can be created / duplicated (and then test and you'll figure it out!)
1) a beast mode can be duplicated if it exists only in the card (i.e. not "shared to dataset") and then you duplicate the card. For sure that will create a new entry for the beast mode in beast mode manager because that beast mode is tied to that card. i'm unsure if it would have the language of "cloned" or not.
2) a beast mode can be duplicated if you duplicate the dashboard (and in the same process duplicate the card). it is unclear to me what impact it would have on language in beast mode manager.
3) if you go to the data center and move a card from one dataset to another (or do the same in analyzer), Analyzer will try to duplicate the beast modes from your card on that new dataset (it is unclear to me, test it and let us know! -- whether Analyzer will only duplicate the fields actually in use on the card or ALL the beast modes in the card that are NOT shared to the dataset, or ALL the beast modes including those shared to the dataset).
Hope that helps give a framework for your investigagtion..
Re: Reduce ETL Processing Time
@GrantSmith is spot on with his advice with one small addendum.
If you can perform your JOIN in a Dataset View (which is basically Fusions 2.0 so ... ignore fusions) then, assuming your VIEW can return a result within 1 minute (processing requirement imposed on VIEWS / Adrenaline) then you should see significant improvement to your Magic ETL dataflow performance.
In a nutshell. Views do operate on the database layer (Adrenaline) so therefore your JOINs should be more performant because the benefit from Indexing, columnar compression, and all the things we love about databases.
Magic ETL is not a database engine (just wrote a longish explanation here -- https://dojo.domo.com/main/discussion/55367/sql-view-of-datasets ) so try to just do the tasks in ETL that a streaming text pipeline should be good at.
If you're using many JOINs to convert a Snowflake schema into a more traditional Dimensional table (i.e. a ton of lookups on lookups) it may make sense -- as Grant described to create your dimensional table separately from the JOIN on to the transactional stuff (i.e. "chaining joins" as Grant describes).
Hope that helps.
Re: PYDOMO SCRIPTS DEPLOYMENT
In DP22 Domo released the ability to schedule Jupyter Notebooks just like any other dataflow. Definitely worth checking out!
https://domohelp.domo.com/hc/en-us/articles/360047400753-Jupyter-Workspaces#4.4.
Re: PYDOMO SCRIPTS DEPLOYMENT
Domo Magic ETL scripting tiles.
Re: Triggering/Scheduling DOMO Jupyter Workspace Files/Notebooks
UPDATE -- @user09644 this feature was released at DP22. It's now possible (and frankly IMHO there's no reason to use Python or R tiles in Magic Dataflows anymore!
https://domohelp.domo.com/hc/en-us/articles/360047400753-Jupyter-Workspaces#4.4.
Re: Triggering/Scheduling DOMO Jupyter Workspace Files/Notebooks
@GrantSmith is correct, Jupyter Notebooks cannot be integrated into an automated pipeline by design.
The tool for pipeline automation would be the MagicETL scripting tiles!
Re: Dataset/Card Certification History
@Jmoreno I'm going to create product feedback for this here: https://dojo.domo.com/main/discussion/55701/domo-certification-progress-and-community-engagement-datasets/p1?new=1 feel freet o upvote.
I know my org needs it.
Re: How to periodically REPLACE in addition to APPEND jobs?
@JunkDoom have a close look at Partitioning.
Many Domo connectors are starting to support partition schemes based on Date. If your data is not partitioned by Date you can use the CLI to create your own partition tags as you upload data.
the in-Domo ETL version of "Partitioning" can be recreated with Recursive Dataflows. https://domohelp.domo.com/hc/en-us/articles/360057087393-Creating-a-Recursive-Snapshot-DataFlow-in-Magic-ETL
There is a beta (or is it GA @GrantSmith ) for Magic ETL + Partitions.
You want to be careful mixing Partition with Replace. Replace will Replace your entire dataset WITHOUT partition_tags. meaning that if you tried to mix Replace with Append+Partition you would duplicate data.
Re: SQL View of DataSets
@RobSomers unfortunately, "The Magic ETL is just a visual representation of SQl" this is not an accurate statement and is potentially misleading.
Although Magic ETL transforms borrow heavily from transforms we are accustomed to performing in SQL and even the Formula Tile allows you to write functions borrowed from MySQL, it is not accurate to assume that the engine underneath Magic ETL is a SQL database.
I believe the functions under the covers are implemented in Java and they are performing operations in more of a streaming text pipeline.
This might be splitting hairs, but it's important to understand because if you think of Magic as a database layer with Tables where you're issuing UPDATE commands against a set of records, it may discourage you from thinking of ways you might try to support Magic as it subdivides your tasks across multiple nodes for parallel processing.
In fact the biggest difference and improvement in Magic over your SQL pipelines (Adrenaline, Redshift, and MySQL) include
1) Magic does not have to load all the data into a table before it can be processing the data. Because the data is moving in a text stream, as first as the first row of data arrives it can start flowing through your ETL tiles.
2) Magic does not have to Index. (this is a double edged sword). Because Magic is not a database engine there is no concept of indexing. therefore JOINs, RANK, and SORT operations would be slower than the equivalent operation performed in Redshift or Adrenaline (assuming all other factors were equal).