How to include nested folders in S3 advanced connector

user095063 · 2021-09-13T16:36:21+00:00

There was an error rendering this rich post.

Hello. I am trying to set up the connector that would fetch the data daily from S3 bucket. I am working on an advanced s3 connector as it seems to have a more intuitive way to grab the data.

My S3 schema looks like this: `s3://adtech-third-party-data/AmazonA9/2021/09/`

In DOMO settings credentials, I put adtech-third-party-data/ as the name of the bucket.

In the Details section for * FILE NAME I have AmazonA9/2021/09/ It worked, however, it didn't fetch all the files.

Below is the s3 bucket for folder adtech-third-party-data/AmazonA9/2021/09/ :

Screen Shot 2021-09-13 at 12.32.51 PM.png

And this is what I have in DOMO:

Screen Shot 2021-09-13 at 12.34.07 PM.png

It skipped a couple of days, for some reason.

What is the right * FILE NAME path I should use in order to fetch all the data?

Find more posts tagged with

Accepted answers

All comments

GrantSmith

Have you confirmed that the file names are included in your dataset or not? What specifically does your date field represent?

Christianjmzc

I know this is old... However, if anyone is having this issue, you just need to select "Append" on the Scheduling section. That will pull all the files within the path/folder.

pstrauss

@GrantSmith I'm having a similar problem with the S3 AssumeRole Advanced connector. I'm trying to read all of the .json files within a subdirectory in an S3 bucket. I'm able to successfully connect, but I get this error when the job runs:

Domo is ready but did not find files that match '/logs/entity=BuffersAndBarriers/*'

That's the S3 path I'm trying to read inside of the bucket, and I put that value into the File Name field in the connector. Any ideas on how to get this working?

Here's a screen grab of the files in the subdirectory in question:

UPDATE: Removing the leading slash and wildcard on logs/entity=BuffersAndBarriers/ gets rid of the file not found error, but no data is loaded.