How to get latest files from s3 using the Domo s3 connector

user07506
user07506 Member
edited March 2023 in Scheduled Reports

Hello,

 

I have setup an S3 Connector to one of my buckets in AWS. This bucket it being constantly updated with new files multiple times a day and I would like the data in these files to be appended to the S3 Connector dataset. What would be the best way to automate getting the latest file or files from this s3 bucket into domo and append it to an existing dataset?

 

John

Best Answers

  • user07506
    user07506 Member
    Answer ✓

    Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at. 

  • guitarhero23
    guitarhero23 Contributor
    Answer ✓

    This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.


    **Make sure to like any users posts that helped you and accept the ones who solved your issue.**

Answers

  • user07506
    user07506 Member
    Answer ✓

    Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at. 

  • hamza_123
    hamza_123 Contributor

    Hello. I am facing the same challenge as you were. Could you tell how you were able to solve the issue?

  • guitarhero23
    guitarhero23 Contributor
    Answer ✓

    This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.


    **Make sure to like any users posts that helped you and accept the ones who solved your issue.**
  • hamza_123
    hamza_123 Contributor

    That sounds reasonable. 

     

    Also if could ask, I cannot post questions in Dojo. There's no search bar which asys "ask a question" in Dojo for me. Even though I can reply to other threads. If you could please tell me what I can do to start asking questions in Dojo. 

  • guitarhero23
    guitarhero23 Contributor

    You should see it once you click on a section then lower you'll see "Start a Topic". Welcome to Dojo!

     

    Dojo1.JPG

     

    Dojo2.JPG


    **Make sure to like any users posts that helped you and accept the ones who solved your issue.**
  • hamza_123
    hamza_123 Contributor

    I get it. Thanks. 

     

    On my first question, I will be recieving one new file everyday in my S3 bucket. I have two options:

     

    .1. Append new data with old before it comes to S3, so then I can bring all of that to Domo

    .2. Bring all files (old and new) to Domo (with a similar naming convention so that Domo can pick up all of them) and them apply transformations to them in Domo to append them. 

     

    Option 1 seems hard for me. While I am not sure if Domo gives the option for Option 2. 

    I have seen "append" option in the Amazon S3 Advanced connector. However I am not sure what this means. How will Domo know which files (in the S3 bucket) are old ones and should be left untouched and which one is the new one so that it can append the data

This discussion has been closed.

Categories