Connectors

Connectors

S3 Advanced Connector Not Loading JSON Files

Member
edited March 2024 in Connectors

I'm trying to automatically load all files ending in ".json" from an S3 bucket using the S3 Advanced Connector. However, no data is loading. I set the File Name field to ".json" and the File Name Match Type to "File name contains the specified string."

I have no problem reading individual files from this bucket without a problem using the standard S3 connector. Any ideas what I might be doing wrong?

On a related note, will the S3 Advanced Connector automatically load each file only one time? That's the behavior we would like - we're basically streaming JSON log files into a bucket and need each one added to Domo, and each log is in a separate file.

I've attached screenshots of my connector settings, as well as a screenshot of my test file bucket showing some example file names.

Screenshot 2024-02-29 at 10.29.03 PM.png Screenshot 2024-02-29 at 10.22.15 PM.png
Tagged:

Best Answer

  • Coach
    Answer ✓

    I'm about to be configuring the same thing. A few things you might try…

    Check your IAM role for permissions.
    Since you are trying to load files with ".json", set the file name field to "*.json".
    Make sure files are truly formatted JSON.
    Make sure your files are not being overwritten or deleted.


    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • Coach
    Answer ✓

    I'm about to be configuring the same thing. A few things you might try…

    Check your IAM role for permissions.
    Since you are trying to load files with ".json", set the file name field to "*.json".
    Make sure files are truly formatted JSON.
    Make sure your files are not being overwritten or deleted.


    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In