How to get latest files from s3 using the Domo s3 connector
Hello,
I have setup an S3 Connector to one of my buckets in AWS. This bucket it being constantly updated with new files multiple times a day and I would like the data in these files to be appended to the S3 Connector dataset. What would be the best way to automate getting the latest file or files from this s3 bucket into domo and append it to an existing dataset?
John
Best Answers
-
Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at.
0 -
This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0
Answers
-
Looks like I was about to create a work around by scripting a program to refresh a bucket with the latest file that domo is looking at.
0 -
Hello. I am facing the same challenge as you were. Could you tell how you were able to solve the issue?
0 -
This thread is old so incase this person doesn't respond it appears they coded a script that instead of uploading new files to S3 all the time they just added new data to one file, then Domo could just be set on a schedule Ex. Hourly, to do a REPLACE and always look at that one file. It depends on the systems your data is coming from and programming language of choice how this could be achieved.
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0 -
That sounds reasonable.
Also if could ask, I cannot post questions in Dojo. There's no search bar which asys "ask a question" in Dojo for me. Even though I can reply to other threads. If you could please tell me what I can do to start asking questions in Dojo.
0 -
You should see it once you click on a section then lower you'll see "Start a Topic". Welcome to Dojo!
**Make sure to like any users posts that helped you and accept the ones who solved your issue.**0 -
I get it. Thanks.
On my first question, I will be recieving one new file everyday in my S3 bucket. I have two options:
.1. Append new data with old before it comes to S3, so then I can bring all of that to Domo
.2. Bring all files (old and new) to Domo (with a similar naming convention so that Domo can pick up all of them) and them apply transformations to them in Domo to append them.
Option 1 seems hard for me. While I am not sure if Domo gives the option for Option 2.
I have seen "append" option in the Amazon S3 Advanced connector. However I am not sure what this means. How will Domo know which files (in the S3 bucket) are old ones and should be left untouched and which one is the new one so that it can append the data
2
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 294 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 97 SQL DataFlows
- 607 Datasets
- 2.1K Magic ETL
- 3.8K Visualize
- 2.4K Charting
- 707 Beast Mode
- 49 App Studio
- 39 Variables
- 667 Automate
- 170 Apps
- 446 APIs & Domo Developer
- 44 Workflows
- 7 DomoAI
- 33 Predict
- 13 Jupyter Workspaces
- 20 R & Python Tiles
- 391 Distribute
- 111 Domo Everywhere
- 274 Scheduled Reports
- 6 Software Integrations
- 115 Manage
- 112 Governance & Security
- Domo Community Gallery
- 31 Product Releases
- 9 Domo University
- 5.3K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 103 Community Announcements
- 4.8K Archive