How to get latest files from s3 using the Domo s3 connector
Hello,
I have setup an S3 Connector to one of my buckets in AWS. This bucket it being constantly updated with new files multiple times a day and I would like the data in these files to be appended to the S3 Connector dataset. What would be the best way to automate getting the latest file or files from this s3 bucket into domo and append it to an existing dataset?
John
Best Answers
-
Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at.
0 -
This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0
Answers
-
Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at.
0 -
Hello. I am facing the same challenge as you were. Could you tell how you were able to solve the issue?
0 -
This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0 -
That sounds reasonable.
Also if could ask, I cannot post questions in Dojo. There's no search bar which asys "ask a question" in Dojo for me. Even though I can reply to other threads. If you could please tell me what I can do to start asking questions in Dojo.
0 -
You should see it once you click on a section then lower you'll see "Start a Topic". Welcome to Dojo!
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0 -
I get it. Thanks.
On my first question, I will be recieving one new file everyday in my S3 bucket. I have two options:
.1. Append new data with old before it comes to S3, so then I can bring all of that to Domo
.2. Bring all files (old and new) to Domo (with a similar naming convention so that Domo can pick up all of them) and them apply transformations to them in Domo to append them.
Option 1 seems hard for me. While I am not sure if Domo gives the option for Option 2.
I have seen "append" option in the Amazon S3 Advanced connector. However I am not sure what this means. How will Domo know which files (in the S3 bucket) are old ones and should be left untouched and which one is the new one so that it can append the data
2
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 102 SQL DataFlows
- 626 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 755 Beast Mode
- 61 App Studio
- 41 Variables
- 693 Automate
- 178 Apps
- 456 APIs & Domo Developer
- 49 Workflows
- 10 DomoAI
- 38 Predict
- 16 Jupyter Workspaces
- 22 R & Python Tiles
- 398 Distribute
- 115 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 130 Manage
- 127 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 11 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 110 Community Announcements
- 4.8K Archive