Append data issue using CSV upload

user08392
user08392 Member
in Archive

Hi All,

 

I'm uploading CSV file via workbench and I have scheduled every four hours. I'm getting all the records without any update. My requirement is to get all the records which are not present and which are present those records to be updated. But, what is happening is records which are present are not updating but adding an additional row. Please help with this. Thanks

 

Comments

  • user05685
    user05685 Member

    Arav,

     

    It sounds like what you need to do is run your file through a dataflow to remove duplicate rows every time it is uploaded and/or perhaps do a mysql transformation to tell it to keep only the most recent records in cases where there has been an update. 

     

    I hope that helps!

  • Unknown
    Unknown

    Hi User,

     

    Thanks for the reply. Is there a way to use merge while uploading CSV file in the workbench. So, that it can search for already existing records and replace with new records. Also, using your solution, may I know exactly what is the process to follow?? Thanks.

  • rado98
    rado98 Contributor

    I am not sure I understand what your are trying to do. Does the CSV file uploaded every 4 hours contain all records since the beggining of time or is there new data evety 4 hours? If you can elaborate I might be able to give you a simpler solution than below.

     

    Use the info in the thread below to generate an appeding CSV dataset that will delete all duplicated rows

     

    https://dojo.domo.com/t5/Beast-Mode-ETL-Dataflow/Struggling-with-appending-data-to-progressively-build-up-a/m-p/36515#M5871

     

     

This discussion has been closed.

Categories