Mutiple Zip files files in S3 Bucket

Sam_Y
Sam_Y Member
in Connectors

I am having 2 problems with the S3 Connectors (Advanced and Standard).  I am struggling to find which one is better suited to handle multiple files.

 

Here are the details:

Our vendor is posting a new Zip file each day to an S3 Bucket.  The subfiles within the new daily Zip all have the prefix change based on a physical location identifier.  For example,

 

1345_08_14_20.json

1298_08_14_20.json

 

where 1345 and 1298 are the location identifers

 

Using the advanced connector, I am only able to pull one json file at a time I include the prefix (i.e. 1345) in the SUBFILE field, however, it will only extract that first json file.  I need to pull all the json files within that Zip.  Would re-naming the files to something with a generic name file the issue (i.e. Daily File_1345_08_14_20.json) where i use the Daily_File in the subfile field?

 

The second question is how to append the new file(s) each day as we are not able to enter a variable.  I see the regular S3 Connector will grab most recently modified, however, the Zip option requires that you discover the zipped files using a drop down.

 

Any hints or suggestions would be amazing!!! Thanks

 

 

Comments

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    1) it would probably be easiest to perform server-side scripting that combines the data into one file BEFORE sending it to Domo.

    2) most connectors have an option to Replace or Append data during ingestion to Domo (check if there's an option under "Scheduling" during connector configuration).

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • Sam_Y
    Sam_Y Member

    @jaeW_at_Onyx   We were able to combine the files into 1 json within the Amazon bucket, however, the JSON file name will change every day within the zip.

     

    Today's file is 2020-09-21_nl_replica.json

     

    Tomorrow's file  will be 2020-09-22_nl_replica.json

     

    Will the Discovery option automatically pull the 2020-09-21_nl_replica.json file tomorrow or will it go for a new file?

     

     

  • Sam_Y
    Sam_Y Member

    I just tried the Advanced Connector and it gives the ability to enter a partial name for the Subfile.  That may work for me.

Categories