Mutiple Zip files files in S3 Bucket
I am having 2 problems with the S3 Connectors (Advanced and Standard). I am struggling to find which one is better suited to handle multiple files.
Here are the details:
Our vendor is posting a new Zip file each day to an S3 Bucket. The subfiles within the new daily Zip all have the prefix change based on a physical location identifier. For example,
1345_08_14_20.json
1298_08_14_20.json
where 1345 and 1298 are the location identifers
Using the advanced connector, I am only able to pull one json file at a time I include the prefix (i.e. 1345) in the SUBFILE field, however, it will only extract that first json file. I need to pull all the json files within that Zip. Would re-naming the files to something with a generic name file the issue (i.e. Daily File_1345_08_14_20.json) where i use the Daily_File in the subfile field?
The second question is how to append the new file(s) each day as we are not able to enter a variable. I see the regular S3 Connector will grab most recently modified, however, the Zip option requires that you discover the zipped files using a drop down.
Any hints or suggestions would be amazing!!! Thanks
Comments
-
1) it would probably be easiest to perform server-side scripting that combines the data into one file BEFORE sending it to Domo.
2) most connectors have an option to Replace or Append data during ingestion to Domo (check if there's an option under "Scheduling" during connector configuration).
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"0 -
@jaeW_at_Onyx We were able to combine the files into 1 json within the Amazon bucket, however, the JSON file name will change every day within the zip.
Today's file is 2020-09-21_nl_replica.json
Tomorrow's file will be 2020-09-22_nl_replica.json
Will the Discovery option automatically pull the 2020-09-21_nl_replica.json file tomorrow or will it go for a new file?
0 -
I just tried the Advanced Connector and it gives the ability to enter a partial name for the Subfile. That may work for me.
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 297 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 616 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 54 App Studio
- 40 Variables
- 678 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 46 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive