Workbench doesn't process all rows in *.csv file

user08458
user08458 Member
in Workbench

Hello,

 

I'm uploading a .csv file through Workbench. Some of the rows (around 3%) failed to upload without any warning. Oddly enough, all of these rows also happen to have a column A that starts with a hashtag. If I delete the hashtag, that row will be uploaded into Domo without any issues. What's the matter?

 

Please help troubleshoot.

 

Thanks,

Best Answer

  • n8isjack-ret
    n8isjack-ret Domo Employee
    Answer ✓

    Thanks for offering to load a sample file, normally that would be useful. I was able to recreate what you are seeing though and you are correct, it is unrelated to the risks already outlined in this thread.

     

    It appears that the # character, when it is the very first character on the row, is being treated as a "comment" character. This is common in many programming languages. I do not believe it is expected behavior for a csv file.

     

    I will submit this to our programming group to be reviewed and, if they agree, to have a patch submitted for workbench to correct that behavior.

     

    In the meantime is it possible for you rearrange the order of fields on your csv file? So that another field without the # character is the first field on the file.

    Former Domo employee you can find me in the Dojo Community here @n8isjack

Answers

  • KaLin
    KaLin Member

    Can anyone help with troubleshooting Workbench?

  • n8isjack-ret
    n8isjack-ret Domo Employee

    There could be a few things that will cause that to happen. If the field is a numeric or date field for example, the hash tag is not a valid character and it will cause an error.

     

    These types of errors can be handled in a few ways using the Settings in your workbench dataset job:

     

    Workbench Settings

     

    From your description of what is happening it seems that your job is set to the second option: "Skip entire row if it contains invalid cells"

     

    If you need to allow the hashtag you can go into the "Schema" section and manually set that field to be a text field if you want to allow hashtags. 

     

    It also may be possible for you to use a transform in Workbench to remove the hashtag automatically.

    Former Domo employee you can find me in the Dojo Community here @n8isjack
  • user08458
    user08458 Member

    It is definitetly not the first issue, since the column is of datatype 'STRING'. Keep in mind that other rows (and even some with a hashtag) upload without any problems. 

     

    Also, I have the 'Replace Invalid cells with NULL values' checked, so that shouldn't be the issue.

     

    Since the last time I tried converting the file to xls format and uploading Excel file through the connector. The rows that I had a problem with still don't register.

     

    I can come up with a sample file for troubleshooting purposes, would that be useful?

     

     

     

     

     

  • n8isjack-ret
    n8isjack-ret Domo Employee
    Answer ✓

    Thanks for offering to load a sample file, normally that would be useful. I was able to recreate what you are seeing though and you are correct, it is unrelated to the risks already outlined in this thread.

     

    It appears that the # character, when it is the very first character on the row, is being treated as a "comment" character. This is common in many programming languages. I do not believe it is expected behavior for a csv file.

     

    I will submit this to our programming group to be reviewed and, if they agree, to have a patch submitted for workbench to correct that behavior.

     

    In the meantime is it possible for you rearrange the order of fields on your csv file? So that another field without the # character is the first field on the file.

    Former Domo employee you can find me in the Dojo Community here @n8isjack
  • Rich
    Rich Domo Employee

    Hi,

     

    I used an example file containing the # sign as the first character in a row. I was able to get this load into Domo by using the Excel connector and the rows came in. I see you're using workbench so the experience may be different there. Perhaps you're using workbench for the scheduling or transforms features or due to file size constraints. If it is a one-time load I'd suggest using the Excel connector.  

  • jstan
    jstan Contributor

    Hi, I have had similar issues with certain characters in string fields not uploading through Workbench with the native local file approach.  This included a comma quote deliminted CSV files that had something "" in the field value and a \ at the end of a string.  We ended up subscribing to cdata.com ODBC drivers and use that driver for CSV or Excel files that might contain strange characters that we cannot control (at least without manual intervention).  This third party driver worked well with our extracts from Concur and SAP Employee Central Success Factors.

     

    Thanks,

    Josh

     

This discussion has been closed.

Categories