Workbench doesn't process all rows in *.csv file

user08458

Hello,

 

I'm uploading a .csv file through Workbench. Some of the rows (around 3%) failed to upload without any warning. Oddly enough, all of these rows also happen to have a column A that starts with a hashtag. If I delete the hashtag, that row will be uploaded into Domo without any issues. What's the matter?

 

Please help troubleshoot.

 

Thanks,

n8isjack-ret

Thanks for offering to load a sample file, normally that would be useful. I was able to recreate what you are seeing though and you are correct, it is unrelated to the risks already outlined in this thread.

 

It appears that the # character, when it is the very first character on the row, is being treated as a "comment" character. This is common in many programming languages. I do not believe it is expected behavior for a csv file.

 

I will submit this to our programming group to be reviewed and, if they agree, to have a patch submitted for workbench to correct that behavior.

 

In the meantime is it possible for you rearrange the order of fields on your csv file? So that another field without the # character is the first field on the file.

KaLin

Can anyone help with troubleshooting Workbench?

n8isjack-ret

There could be a few things that will cause that to happen. If the field is a numeric or date field for example, the hash tag is not a valid character and it will cause an error.

 

These types of errors can be handled in a few ways using the Settings in your workbench dataset job:

 

 

From your description of what is happening it seems that your job is set to the second option: "Skip entire row if it contains invalid cells"

 

If you need to allow the hashtag you can go into the "Schema" section and manually set that field to be a text field if you want to allow hashtags. 

 

It also may be possible for you to use a transform in Workbench to remove the hashtag automatically.

user08458

It is definitetly not the first issue, since the column is of datatype 'STRING'. Keep in mind that other rows (and even some with a hashtag) upload without any problems. 

 

Also, I have the 'Replace Invalid cells with NULL values' checked, so that shouldn't be the issue.

 

Since the last time I tried converting the file to xls format and uploading Excel file through the connector. The rows that I had a problem with still don't register.

 

I can come up with a sample file for troubleshooting purposes, would that be useful?

 

 

 

 

 

n8isjack-ret

Thanks for offering to load a sample file, normally that would be useful. I was able to recreate what you are seeing though and you are correct, it is unrelated to the risks already outlined in this thread.

 

It appears that the # character, when it is the very first character on the row, is being treated as a "comment" character. This is common in many programming languages. I do not believe it is expected behavior for a csv file.

 

I will submit this to our programming group to be reviewed and, if they agree, to have a patch submitted for workbench to correct that behavior.

 

In the meantime is it possible for you rearrange the order of fields on your csv file? So that another field without the # character is the first field on the file.

Rich

Hi,

 

I used an example file containing the # sign as the first character in a row. I was able to get this load into Domo by using the Excel connector and the rows came in. I see you're using workbench so the experience may be different there. Perhaps you're using workbench for the scheduling or transforms features or due to file size constraints. If it is a one-time load I'd suggest using the Excel connector.  

jstan

Hi, I have had similar issues with certain characters in string fields not uploading through Workbench with the native local file approach.  This included a comma quote deliminted CSV files that had something "" in the field value and a \ at the end of a string.  We ended up subscribing to cdata.com ODBC drivers and use that driver for CSV or Excel files that might contain strange characters that we cannot control (at least without manual intervention).  This third party driver worked well with our extracts from Concur and SAP Employee Central Success Factors.

 

Thanks,

Josh