Data Flows

Hi y'all! Hope everyone's having a good Monday so far. I'm trying to create a table card with a row for each DataFlow that has a trigger to run on a schedule. I looked through the DomoStats DataFlow History Dataset and Domo's Dataflow Details, but I can't find it. Is it accessible anywhere? I also looked through the API…

Today, groups cannot be assigned as the owner of any dataflows. This means, that any output of a given dataflow is by default set as the same owner. It also means that by default, error alerts are only sent to one person instead of a team. We have a use case involving a development team who cannot be granted the manage…

We need to have the ability to understand when dataflows are running longer than the average duration. We would like to set user defined alert criteria based upon the dataflow's duration. For example: alert when dataflow has been running 15min, 1hr, etc. longer than the average for the dataflow. Having the functionality of…

Hi, I’m working with a dataset that is updated monthly through a Magic ETL, where new data gets appended to the historical dataset. I now need to make a one-time correction to the existing historical data in that output dataset. However, when I try to create a new Magic ETL for this, it generates a separate dataset instead…

I have a MySQL Dataflow that I'm using to create a temporal dataset. The query isn't overly complicated and I'm returning less than 1000 records currently. It takes approximately 21 minutes to run. I've added some indexing to the source tables, but that didn't change anything. I've tried to run "Explain SQL" to see the…

We have a difficult time getting people to document the changes they have made to a dataflow. I would like the ability to go back and document a certain version of a dataflow after it has been saved.

When a dataflow runs, whether it's a SQL or ETL dataflow, if there are multiple outputs, the dataflow will say it has completed its run when the first dataflow output completes. So even if another output is still processing, the dataflow looks like it's complete. It makes it difficult to know when a dataflow output has…

A recent question (Any way to view schedule for dataflows?) brought about a valuable solution that I want to share with everyone. A comment with useful information from @nathankilcrease lead me to create the following solution. In Python I created code to produce a dataset showing all dataflow information. Basically - when…

Hi Community, We have a few providers that are mapped to different galleys. This is usually easy and clean, but of course there's exceptions that confound the entire process. Here's current state: Appointment Provider Galley Evaluation Mitch John Evaluation Justin John Evaluation Kristin Tanner Evaluation Kristin Tanner…

The current method of creating tags within Domo is quite straightforward and simple, and probably for many smaller organisations the ability for anyone to create a new tag would be quite useful. The lack of control over the different variations of tags in larger organisations though can result in things becoming messy.…

We have an ETL that needs to run on the last day of each month. Our current solution is to run the ETL on the 28th, 29th, 30th, and 31st. Can Domo add a default Last Day of the Month option similar to the Last Day of the Month Option in the SFTP scheduling options?

Use Case: When building data flows with data sets in millions of rows, it's hard to test if the logic works across those rows unless I add filter tiles that filter out data at specific points during the Data Flow build process. This can add complexity when adding many test cases at specific points. The Idea: It would be…

It would be great if we could integrate SQL fluff into beast modes and the sql tile in Domo or have a format button with in those options as users can copy and past code it can cause readability issues. https://github.com/sqlfluff/sqlfluff

Hi yall! I'm watching the intro videos on the Domo Youtube channel, and the video on Inputs and Outputs introduced the Data Handling section under the Configuration tab for input datasets in a dataflow. I have a dataflow that changed a column's datatype via a 3-node process, so I thought I would quickly optimize this.…

We had an issue, and the error below is what shows on the dataflow: Adrenaline data export failed. DataFlow cannot start. Datasource: xxxxxxxxxxxxxxxxxx Once we received information from Engineering, they showed the error they were seeing was: Error from Dremio: Unable to process results: RESOURCE ERROR: Query cancelled by…

Team, I have built an extensive ETL and I want to "Save and Run" to explore and analyze the output. Unfortunately, the "Save" button isn't responsive. I will click on it and it won't actually save. I'll click "Save and Run" and it won't actually save… Weird issue that I've never ran into before so hoping there's a simple…

Certification is currently requested and managed via a dataset, however we've found that in some cases changes to the dataflow don't cause the certification to expire. This introduces potential risk in cases where changes could be made that affect the accuracy of the data, but the dataset could still be flagged as…

The Domo functions such as NOW(), CURRENT_TIME(), CURRENT_TIMESTAMP(), CURTIME() all return times in UTC. It would be great if these could all be modified to take additional optional parameter of time zone so that the return values are shifted to the local time appropriate to the given time zone (including daylight savings…

Just had a user accidentally do a join that we identified 7 hrs after the process started that was up to 72 billion rows. It was going to top out over 300 billion, but we hit cancel. However, that overflow credit usage topped out over 3,000 credits, which is not the kind of mistake you'd like your team to be making. So,…

Hi, I have a dataset of about 4.7M rows. When I try to use the Group By tile in Magic ETL to find a sum of values across IDs, and then join back on the ID, the new sum column is blank. However, in testing, when I filtered the dataset down to about 50K rows to see what was wrong, the example ID I was checking now had the…

Hi Domos, Does anyone have experience connecting PayCom (HR & Payroll software) to Domo? There is no canned Connector in the AppStore.

Hi, I'm trying to join two sets of data together (assignments submission data & assignment summary data) For the output, I am trying to get a list of all assignments with a due date in a date range that have not been submitted by the students. The assignment submission data has a row for every student that lists every…

Tl;dr - display the record count in MagicETL on each step of the last successful run of the ETL. Detail: I often spend quite some time diagnosing and fixing ETLs. In larger ETLs, it can be very onerous to determine whether a record with an expected behaviour makes it through the entire flow to the end to validate the ETL.…

I am trying to determine who provided the evaluation prior to an upgraded premium procedure. I have all the data I could ever need, but I'm not sure of the logic to make it happen. The task is made more difficult because there could be multiple evaluations prior to a procedure, and evaluations on different dates than the…

Hi Team, The formula below is validating but it is showing "NaN" for all CPM values. For context estimated_youtube_ad revenue and ad_impressions may include the following values: 0, Null values, integers greater than 0. Any potential way to fix this? Thank you in advance! Select* ,…

Has anyone successfully resolved this error when trying to schedule as a DataFlow in Jupyter workspace? I don't have any other DataFlows scheduled in this workspace.

You can only schedule one dataflow per workspace. Documentation is misleading saying you can schedule per notebook. Not true. https://domo-support.domo.com/minasan/s/article/36004740075?language=en_US#schedule_notebook The moment you open a second notebook and try to schedule it from within the same Jupyter workspace you…

In our current environment, we use an external repository (Bitbucket) to maintain version control and the ability to run detailed diffs before making changes to our SQL dataflows. It would be helpful to have a way to export either the code or some representation of the current state of a Magic ETL dataflow to be used in…

We have been cleaning up our Domo instance removing unnecessary outputs from Magic ETL flows. I am assuming these datasets aren't truly being deleted, just disconnected from their source to not get updated any more. Is there a way to find these "orphaned" outputs so that they can be properly deleted?