WB4 Error executing DataSet Job - Error sending data to Domo

Hello,

I have just recently implemented Domo using WB4 on a server. I created a dataset job to extract a table from SQL and upload into the Domo server. Only ETL going on in the query is changing the column headings and using transform to set the timezone. This job is set to load every 15 minutes and for the most part it does. However it will randomly fail with the message I added below. With no changes being made it will start to run succesfully again.

Has anyone run into this issue?

[ERROR] [23100:8] 2016-05-17 12:00:56,796 - Error executing DataSet Job
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 461
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__1c.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 141

Quick Links

Accepted answers

eizquierdo

Found the issue, thanks to Nathan WAshburn with Domo. There were a bunch of corrupted files in C:\Windows\TEMP\DomoCache\. Once deleted the problem went away.

All comments

cmarkum

Try expanding it to 20 minutes or resetting the time (e.g 8:05AM instead of 8:00AM).I have also seen this if the SQL Server is running an incremental backup at the same time (which is why i'm suggesting to change the time) Hope this helps.

eizquierdo

Thank you for your reply @cmarkum.

I attempted to set the reload times at 30min intervals and received the same results. After your feedback I attempted to start the reloads at multiple starting points within the hour block, again same results. Unfortunately, the error message is not very explicit in normal terms. I've opened a ticket with Domo support, waiting to her back.

-Emilio

kshah008

Hi @eizquierdo,

Please keep us updated!

eizquierdo

@kshah008 Will do, it was just recently escalated.

alexpeay

@paynejg & @jeremyhurrencan you take a look at the error message above and see if anything jumps out at you?

jeremyhurren

The error message doesn't have great info. Is there anything else in the log that might indicate what the failure is? There is usually a series of messages about a part failed for some reason, and then that it is retrying.

The random nature of the failures make me think that it's likely to be something like a network connection timeout (504) or something similar.

eizquierdo

@JeredU - Here is a full log of an error message. I'd have to agree that it seems to be a network issue, just can't ping point it.

[INFO] [15064:1] 2016-06-02 00:15:24,394 - Starting Workbench.DataCollector v4.0.5891.36387
[INFO] [15064:1] 2016-06-02 00:15:25,481 - [domain:frontpoint.domo.com][job:20] Saving the Workbench DataSet Job to Domo.
[INFO] [15064:1] 2016-06-02 00:15:25,496 - [domain:frontpoint.domo.com][job:20] Updating Agent Job '20' configuration on the Domo server.
[INFO] [15064:9] 2016-06-02 00:15:28,071 - [domain:frontpoint.domo.com][job:20] Opening a connection to the data source
[INFO] [15064:6] 2016-06-02 00:15:28,198 - [domain:frontpoint.domo.com][job:20] Query String: SELECT fsap.[Account_ID] [Account ID]
,[Sales_Employee_Key] [Sales Employee Key]
,fsap.[Sale_Date_Key] [Sale Date Key]
,[Return_Date_Key] [Return Date Key]
,cast(fsap.[Sale_Date] as date)[Sale Date]
,cast(fsap.[Return_Date] as date)[Return Date]
,case when datepart(hour, fsap.[Sale_Date]) between 0 and 7
then dateadd(hour,8,dateadd(day,DATEDIFF(day,0,fsap.[Sale_Date]),0))
else dateadd(minute, datediff(minute,0,fsap.[Sale_Date]) / 10 * 10, 0)
end [Sale Date Time]
,case when datepart(hour, fsap.[Return_Date]) between 0 and 7
then dateadd(hour,8,dateadd(day,DATEDIFF(day,0,fsap.[Return_Date]),0))
else dateadd(minute, datediff(minute,0,fsap.[Return_Date]) / 10 * 10, 0)
end [Return Date Time]
,fsap.[Customer_Monitoring_Plan_Code] [Customer Monitoring Plan Code]
,fsap.[Customer_Contract_Term] [Customer Contract Term]
,[Initial_Order_Sales_Discount_Amount] [Initial Order Sales Discount Amount]
,case when [Is_Qolsys_Sale] = 1 then 'True' else 'False' end [is Qolsys Sale]
,case when [is_ACH] = 1 then 'True' else 'False' end [is ACH]
,[Points] [Sale Points]
,[Return_Points] [Return Points]
,fsap.[AR_Number] [AR Number]
,dls.[Lead_Source_Name] [Lead Source Name]
,dls.[Lead_Source_Sales_Index_Group_Name] [Lead Source Sales Index Group Name]
,dls.[Lead_Source_Partner_Name] [Lead Source Partner Name]
,dls.[Lead_Source_Channel_Name] [Lead Source Channel Name]
,dls.[Lead_Source_Division_Name] [Lead Source Division Name]
,dls.[Lead_Source_Type_Name] [Lead Source Type Name]
,dls.[Lead_Source_Strategy_Name] [Lead Source Strategy Name]
,dls.[Lead_Source_Driver_Name] [Lead Source Driver Name]
,dls.[Lead_Source_Effective_From_Date] [Lead Source Effective From Date]
,dls.[Lead_Source_Effective_To_Date] [Lead Source Effective To Date]
,case when dd1.full_date = cast(getdate() as date) then 'True' else 'False' end [is Sale Today]
,case when dd2.full_date = cast(getdate() as date) then 'True' else 'False' end [is Return Today]
,case when dd1.Is_Week_Day = 1 then 'True' else 'False' end [is Weekday Sale]
,case when dd2.Is_Week_Day = 1 then 'True' else 'False' end [is Weekday Return]
--,case when dd1.full_date < cast(getdate() as date) then 'True' else 'False' end [is Sale Historical]
--,case when dd2.full_date < cast(getdate() as date) then 'True' else 'False' end [is Return Historical]
FROM [BIS].[dbo].[Fact_Sales_Agent_Points_NRT] fsap with (nolock)
Left Join [BIS].[dbo].[Vw_Dim_Date] dd1 on fsap.[Sale_Date_Key] = dd1.[Date_Key]
Left Join [BIS].[dbo].[Vw_Dim_Date] dd2 on fsap.[Return_Date_Key] = dd2.[Date_Key]
Left Join [BIS].[dbo].[Dim_Customer] dc on fsap.[Account_ID] = dc.[Account_ID]
Left Join [BIS].[dbo].[vw_Fact_Lead_Workflow] flw on dc.[Lead_ID] = flw.[Lead_ID]
Left Join [BIS].[dbo].[Dim_Lead_Source] dls on flw.[Lead_Source_Key] = dls.[Lead_Source_Key]
Where ((dd1.[is_Last_24_Months] = 1 or dd2.[is_Last_24_Months] = 1) or (dd1.[is_Current_Month] = 1 or dd2.[is_Current_Month] = 1) or (cast(getdate() as date) = dd1.full_date or cast(getdate() as date) = dd2.full_date))
[INFO] [15064:15] 2016-06-02 00:15:53,970 - [domain:frontpoint.domo.com][job:20] Finished reading 100,000 data rows.
[INFO] [15064:17] 2016-06-02 00:16:00,943 - [domain:frontpoint.domo.com][job:20] Requesting Domo DataSet Schema.
[INFO] [15064:10] 2016-06-02 00:16:01,050 - [domain:frontpoint.domo.com][job:20] Sending data to Domo.
[INFO] [15064:10] 2016-06-02 00:16:01,071 - [domain:frontpoint.domo.com][job:20] Beginning upload of 2 data parts.
[INFO] [15064:10] 2016-06-02 00:16:01,080 - [domain:frontpoint.domo.com][job:20] Initiating Domo data upload.
[INFO] [15064:13] 2016-06-02 00:16:04,260 - [domain:frontpoint.domo.com][job:20] Finalizing Domo data upload for upload ID '1036'.
[ERROR] [15064:10] 2016-06-02 00:16:04,439 - Error executing DataSet Job
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 403
[INFO] [15064:10] 2016-06-02 00:16:04,447 - [domain:frontpoint.domo.com][job:20] Saving Workbench DataSet Job changes to Domo.
[INFO] [15064:10] 2016-06-02 00:16:04,448 - [domain:frontpoint.domo.com][job:20] Updating Agent Job '20' configuration on the Domo server.
[INFO] [15064:10] 2016-06-02 00:16:04,544 - [domain:frontpoint.domo.com][job:20] Reporting DataSet Job execution failure to Domo.
[INFO] [15064:10] 2016-06-02 00:16:04,570 - [domain:frontpoint.domo.com][job:20] Reporting Agent '2' execution status.
[INFO] [15064:19] 2016-06-02 00:16:04,663 - [domain:frontpoint.domo.com][job:20] Sending DataSet Job failure notifications.
[INFO] [15064:19] 2016-06-02 00:16:04,677 - [domain:frontpoint.domo.com][job:20] Requesting Domo DataSet.
[INFO] [15064:18] 2016-06-02 00:16:04,724 - [domain:frontpoint.domo.com][job:20] Reporting Agent execution error for recipients 'domoadministrator@frontpointsecurity.com'.
[ERROR] [15064:10] 2016-06-02 00:16:04,840 - [job:20]
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 461
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__17.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 103
[ERROR] [15064:1] 2016-06-02 00:16:04,870 - The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Close()
at Workbench.DataCollector.Program.Main(String[] args) in c:\jenkins\workspace\Workbench\WorkbenchBuild\Workbench.DataCollector\Program.cs:line 148

kshah008

@jeremyhurren @alexpeay, any ideas?

eizquierdo

Found the issue, thanks to Nathan WAshburn with Domo. There were a bunch of corrupted files in C:\Windows\TEMP\DomoCache\. Once deleted the problem went away.

unknown

This issue also comes when the file has no data in it.

other categories

Product Ideas
Have a Domo product enhancement idea? Submit or upvote on ideas in the Ideas Exchange.
Ideas Exchange
Suggest & vote on new features you would like to see implemented in the Domo Product.
Data Connections
Ask questions about Connectors, Workbench, Cloud Amplifier and get best practices from Domo peers
Connectors
Connectors, Custom Connectors, Writeback
Workbench
Ask questions about Workbench, a secure, client-side solution for uploading your on-premise data to Domo.
Cloud Integrations
Ask questions about Cloud Integrations and Federated Data connection to your data warehouse or lake.
Data & ETL
Ask questions about Magic ETL, SQL DataFlows, DataFusion, Dataset Views and get best practices from Domo peers
Magic ETL
Ask Magic ETL questions and get answers from Domo peers
SQL DataFlows
Ask SQL DataFlow questions and get answers from Domo peers
Datasets
Ask DataFusion and Dataset Views questions and get answers from Domo peers
Visualize & Apps
Ask questions about Beast Mode, Cards, Charting, Dashboards, Stories, Variables and get best practices from Domo peers
Dashboards
Ask Cards, Dashboards, and Stories questions and get answers from Domo peers
App Studio
Ask questions about building apps in App Studio.
Pro-code Components
Ask questions about pro-code components and Domo Bricks and get answers from Domo peers.
Charting & Analyzer
Ask questions about charting and Analyzer and get answers from Domo peers.
Calculations & Variables (Beast Mode)
Ask questions about using calculated fields and Variables (Beast Modes) in Analyzer.
AI & Data science
Ask questions about DomoAI and get answers from Domo peers.
Domo AI & AI Chat
Ask questions about AI Chat and AI assistants.
Managing AI
Ask questions about managing AI with AI Playground, AI projects, AI models, and more.
Jupyter Workspaces
Ask questions about Jupyter Workspaces, Notebooks, and file share.
Automate
Ask questions about App Framework, Workflows, Domo Bricks, Domo Developer, API and get best practices from Domo peers
Workflows
Ask questions about Task Center, building automations with Domo Workflows, and executing JavaScript or Python code with Code Engine.
Alerts
Ask questions about managing alerts in Domo and get answers from Domo peers.
Distribute
Ask questions about Domo Everywhere, Scheduled Reports, Mobile and get best practices from Domo peers
Domo Everywhere
Ask questions about embedded analytics with Domo Everywhere.
Reporting
Ask questions about Scheduled Reports, Report Builder, and Slideshow Publications.
Manage
Ask questions about Governance Administration, Approvals, Teams, Alerts, and Buzz and get best practices from Domo peers
Governance & Security
Ask questions about People, Groups, Roles, Sandbox, Activity log, Buzz, Teams, Approvals and PDP and get best practices from Domo peers
Navigation & Productivity
Ask questions about navigation, Projects & Tasks, Goals, and Buzz chat.
APIs
Ask APIs and Developer.domo.com questions and get answers from Domo peers
Add-ins & Plugins
Ask questions about plugins, Microsoft add-ins, and other third-party software integrations.
Domo Community Gallery
Watch how our Customers are using Domo to solve their complex problems.
Product Releases
Domo support and product teams are here to live-answer questions about the most recent product releases. Please post questions in this Forum board for all users to benefit (rather than submitting a support ticket).
Domo University
Questions or discussions related to Domo University, trainings and certifications
Community Forums
Getting Started
Welcome to Domo's Community Forums! You'll find everything you need to get started in this category.
Community Announcements
Get the latest from Domo's Community Team.
Social Groups
Archive
Old or outdated content that could still be found helpful.