WB4 Error executing DataSet Job - Error sending data to Domo

eizquierdo

Hello,

 

I have just recently implemented Domo using WB4 on a server. I created a dataset job to extract a table from SQL and upload into the Domo server. Only ETL going on in the query is changing the column headings and using transform to set the timezone. This job is set to load every 15 minutes and for the most part it does. However it will randomly fail with the message I added below. With no changes being made it will start to run succesfully again.

 

Has anyone run into this issue?

 

[ERROR] [23100:8] 2016-05-17 12:00:56,796 - Error executing DataSet Job
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 461
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__1c.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 141

eizquierdo

Found the issue, thanks to Nathan WAshburn with Domo. There were a bunch of corrupted files in C:\Windows\TEMP\DomoCache\. Once deleted the problem went away.

cmarkum

Try expanding it to 20 minutes or resetting the time (e.g 8:05AM instead of 8:00AM).I have also seen this if the SQL Server is running an incremental backup at the same time (which is why i'm suggesting to change the time) Hope this helps.

eizquierdo

Thank you for your reply @cmarkum.

 

I attempted to set the reload times at 30min intervals and received the same results. After your feedback I attempted to start the reloads at multiple starting points within the hour block, again same results. Unfortunately, the error message is not very explicit in normal terms. I've opened a ticket with Domo support, waiting to her back.

 

-Emilio

kshah008

Hi @eizquierdo,

 

Please keep us updated!

eizquierdo

@kshah008 Will do, it was just recently escalated.

alexpeay

@paynejg & @jeremyhurrencan you take a look at the error message above and see if anything jumps out at you?

jeremyhurren

The error message doesn't have great info. Is there anything else in the log that might indicate what the failure is? There is usually a series of messages about a part failed for some reason, and then that it is retrying.

 

The random nature of the failures make me think that it's likely to be something like a network connection timeout (504) or something similar.

eizquierdo

@JeredU - Here is a full log of an error message. I'd have to agree that it seems to be a network issue, just can't ping point it.

 

[INFO] [15064:1] 2016-06-02 00:15:24,394 - Starting Workbench.DataCollector v4.0.5891.36387
[INFO] [15064:1] 2016-06-02 00:15:25,481 - [domain:frontpoint.domo.com][job:20] Saving the Workbench DataSet Job to Domo.
[INFO] [15064:1] 2016-06-02 00:15:25,496 - [domain:frontpoint.domo.com][job:20] Updating Agent Job '20' configuration on the Domo server.
[INFO] [15064:9] 2016-06-02 00:15:28,071 - [domain:frontpoint.domo.com][job:20] Opening a connection to the data source
[INFO] [15064:6] 2016-06-02 00:15:28,198 - [domain:frontpoint.domo.com][job:20] Query String: SELECT fsap.[Account_ID] [Account ID]
,[Sales_Employee_Key] [Sales Employee Key]
,fsap.[Sale_Date_Key] [Sale Date Key]
,[Return_Date_Key] [Return Date Key]
,cast(fsap.[Sale_Date] as date)[Sale Date]
,cast(fsap.[Return_Date] as date)[Return Date]
,case when datepart(hour, fsap.[Sale_Date]) between 0 and 7
then dateadd(hour,8,dateadd(day,DATEDIFF(day,0,fsap.[Sale_Date]),0))
else dateadd(minute, datediff(minute,0,fsap.[Sale_Date]) / 10 * 10, 0)
end [Sale Date Time]
,case when datepart(hour, fsap.[Return_Date]) between 0 and 7
then dateadd(hour,8,dateadd(day,DATEDIFF(day,0,fsap.[Return_Date]),0))
else dateadd(minute, datediff(minute,0,fsap.[Return_Date]) / 10 * 10, 0)
end [Return Date Time]
,fsap.[Customer_Monitoring_Plan_Code] [Customer Monitoring Plan Code]
,fsap.[Customer_Contract_Term] [Customer Contract Term]
,[Initial_Order_Sales_Discount_Amount] [Initial Order Sales Discount Amount]
,case when [Is_Qolsys_Sale] = 1 then 'True' else 'False' end [is Qolsys Sale]
,case when [is_ACH] = 1 then 'True' else 'False' end [is ACH]
,[Points] [Sale Points]
,[Return_Points] [Return Points]
,fsap.[AR_Number] [AR Number]
,dls.[Lead_Source_Name] [Lead Source Name]
,dls.[Lead_Source_Sales_Index_Group_Name] [Lead Source Sales Index Group Name]
,dls.[Lead_Source_Partner_Name] [Lead Source Partner Name]
,dls.[Lead_Source_Channel_Name] [Lead Source Channel Name]
,dls.[Lead_Source_Division_Name] [Lead Source Division Name]
,dls.[Lead_Source_Type_Name] [Lead Source Type Name]
,dls.[Lead_Source_Strategy_Name] [Lead Source Strategy Name]
,dls.[Lead_Source_Driver_Name] [Lead Source Driver Name]
,dls.[Lead_Source_Effective_From_Date] [Lead Source Effective From Date]
,dls.[Lead_Source_Effective_To_Date] [Lead Source Effective To Date]
,case when dd1.full_date = cast(getdate() as date) then 'True' else 'False' end [is Sale Today]
,case when dd2.full_date = cast(getdate() as date) then 'True' else 'False' end [is Return Today]
,case when dd1.Is_Week_Day = 1 then 'True' else 'False' end [is Weekday Sale]
,case when dd2.Is_Week_Day = 1 then 'True' else 'False' end [is Weekday Return]
--,case when dd1.full_date < cast(getdate() as date) then 'True' else 'False' end [is Sale Historical]
--,case when dd2.full_date < cast(getdate() as date) then 'True' else 'False' end [is Return Historical]
FROM [BIS].[dbo].[Fact_Sales_Agent_Points_NRT] fsap with (nolock)
Left Join [BIS].[dbo].[Vw_Dim_Date] dd1 on fsap.[Sale_Date_Key] = dd1.[Date_Key]
Left Join [BIS].[dbo].[Vw_Dim_Date] dd2 on fsap.[Return_Date_Key] = dd2.[Date_Key]
Left Join [BIS].[dbo].[Dim_Customer] dc on fsap.[Account_ID] = dc.[Account_ID]
Left Join [BIS].[dbo].[vw_Fact_Lead_Workflow] flw on dc.[Lead_ID] = flw.[Lead_ID]
Left Join [BIS].[dbo].[Dim_Lead_Source] dls on flw.[Lead_Source_Key] = dls.[Lead_Source_Key]
Where ((dd1.[is_Last_24_Months] = 1 or dd2.[is_Last_24_Months] = 1) or (dd1.[is_Current_Month] = 1 or dd2.[is_Current_Month] = 1) or (cast(getdate() as date) = dd1.full_date or cast(getdate() as date) = dd2.full_date))
[INFO] [15064:15] 2016-06-02 00:15:53,970 - [domain:frontpoint.domo.com][job:20] Finished reading 100,000 data rows.
[INFO] [15064:17] 2016-06-02 00:16:00,943 - [domain:frontpoint.domo.com][job:20] Requesting Domo DataSet Schema.
[INFO] [15064:10] 2016-06-02 00:16:01,050 - [domain:frontpoint.domo.com][job:20] Sending data to Domo.
[INFO] [15064:10] 2016-06-02 00:16:01,071 - [domain:frontpoint.domo.com][job:20] Beginning upload of 2 data parts.
[INFO] [15064:10] 2016-06-02 00:16:01,080 - [domain:frontpoint.domo.com][job:20] Initiating Domo data upload.
[INFO] [15064:13] 2016-06-02 00:16:04,260 - [domain:frontpoint.domo.com][job:20] Finalizing Domo data upload for upload ID '1036'.
[ERROR] [15064:10] 2016-06-02 00:16:04,439 - Error executing DataSet Job
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 403
[INFO] [15064:10] 2016-06-02 00:16:04,447 - [domain:frontpoint.domo.com][job:20] Saving Workbench DataSet Job changes to Domo.
[INFO] [15064:10] 2016-06-02 00:16:04,448 - [domain:frontpoint.domo.com][job:20] Updating Agent Job '20' configuration on the Domo server.
[INFO] [15064:10] 2016-06-02 00:16:04,544 - [domain:frontpoint.domo.com][job:20] Reporting DataSet Job execution failure to Domo.
[INFO] [15064:10] 2016-06-02 00:16:04,570 - [domain:frontpoint.domo.com][job:20] Reporting Agent '2' execution status.
[INFO] [15064:19] 2016-06-02 00:16:04,663 - [domain:frontpoint.domo.com][job:20] Sending DataSet Job failure notifications.
[INFO] [15064:19] 2016-06-02 00:16:04,677 - [domain:frontpoint.domo.com][job:20] Requesting Domo DataSet.
[INFO] [15064:18] 2016-06-02 00:16:04,724 - [domain:frontpoint.domo.com][job:20] Reporting Agent execution error for recipients 'domoadministrator@frontpointsecurity.com'.
[ERROR] [15064:10] 2016-06-02 00:16:04,840 - [job:20]
System.IO.InvalidDataException: Error sending data to Domo. 1 data file(s) failed during upload.
at DomoApi.Manager.DataSetsManager.<UploadFilesAsync>d__bd.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\DomoApi\DomoApi\Manager\DataSetsManager.cs:line 857
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Helpers.DataSource.DataSourceV2MultiPart.<UploadFilesAsync>d__3.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Helpers\DataSource\DataSourceV2MultiPart.cs:line 79
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__53.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 461
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at WorkbenchSDK.Execution.WorkbenchJobExecutor.<ExecuteJobAsync>d__17.MoveNext() in c:\jenkins\workspace\Workbench\WorkbenchBuild\WorkbenchSDK\Execution\WorkbenchJobExecutor.cs:line 103
[ERROR] [15064:1] 2016-06-02 00:16:04,870 - The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.
System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.ServiceModel.ICommunicationObject.Close()
at Workbench.DataCollector.Program.Main(String[] args) in c:\jenkins\workspace\Workbench\WorkbenchBuild\Workbench.DataCollector\Program.cs:line 148

kshah008

@jeremyhurren @alexpeay, any ideas?

eizquierdo

Found the issue, thanks to Nathan WAshburn with Domo. There were a bunch of corrupted files in C:\Windows\TEMP\DomoCache\. Once deleted the problem went away.

Unknown

This issue also comes when the file has no data in it.