How to append data from S3 Bucket into Domo

hamza_123
hamza_123 Contributor
in Connectors

I am going to recieve one file daily into my S3 bucket, which I need to bring into Domo through the S3 Advanced Bucket connector. I have to keep historical data in Domo, which means that I have to append mydataset in Domo everyday with a new file that is coming into the S3 bucket daily. 

 

I do not understand how to do this. I'm looking for an option that does not involve any scripting to append files. More specifically, I saw the 'Append' option in the Domo S3 Advanced connector. I'm curious about how Domo will know which file to append into the old data, if there are multiple files in the S3 bucket (old and new). 

 

e.g If there is one file in S3 bucket which came yesterday called "Finances_13_July", and it was imported into Domo through the connector. Next day, a file called "Finances_14_July" comes in the S3 bucket. Will the append option in the Domo S3 Advanced connector realize which file was the old one (already imported into Domo) and which is the new file to be appended (given that both todays and yesterday's file are in the S3 bucket)?

Comments

  • AnnK
    AnnK Member

    Hi there!

     

    I actually have multiple datasets that append from the advanced s3 connector.  One of my datasets brings in a new file with daily totals each day.  What I have found to work best is to have a consistent naming scheme for all files and add a timestamp to the name.  For example, today domo brought in the file named "2019-09-20 DailyTotals" tomorrow it will bring in the dataset "2019-09-21 DailyTotals".  I do this by selecting the "partial file name" option under the "How would you like to choose your file name" drop down.  For the next drop down I select file contains a specific string.  That way domo knows to bring in the most recent file with that name and add it to the existing dataset.

     

    To get it to actually append, you'll have to choose the "append" method of updating in the scheduling section.

    I hope that helps!

  • hamza_123
    hamza_123 Contributor

    Thank you for the reply. I will try this out and see if it works. 

  • mhouk
    mhouk Member

    Did this work out for you i am going to set up a S3 connection soon and was wondering if i could set up something similar…

Categories