Moving files from SFTP site to S3
Hello everyone,
Has anyone used Domo to move files from SFTP to S3 while keeping the original file format (pipe delimited csv). We receive a daily file from our vendor that we want to:
- Archive the daily file
- Load from stage/archive location to our Snowflake warehouse (future state)
We currently use another tool to do this but that will not be a solution in the near future. Hoping to use Domo to replicate the pattern.
Best Answer
-
You could setup the SFTP connector or use Workbench to pull from the SFTP. Make sure the file is fetched in CSV. Then use Python or Domo's AWS S3 writeback connector to transfer to S3. Use a dataflow to do any processing or transformations. For the archive, write a step that trigger the S3 Writeback to upload to the desired archive location.
And for Snowflake, I believe there is a Domo Snowflake connector. You can integrate using a new dataflow or workflow that reads the archived file from S2 and pushes it to Snowflake.** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **0
Answers
-
You could setup the SFTP connector or use Workbench to pull from the SFTP. Make sure the file is fetched in CSV. Then use Python or Domo's AWS S3 writeback connector to transfer to S3. Use a dataflow to do any processing or transformations. For the archive, write a step that trigger the S3 Writeback to upload to the desired archive location.
And for Snowflake, I believe there is a Domo Snowflake connector. You can integrate using a new dataflow or workflow that reads the archived file from S2 and pushes it to Snowflake.** Was this post helpful? Click Agree or Like below. **
** Did this solve your problem? Accept it as a solution! **0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 101 SQL DataFlows
- 622 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 748 Beast Mode
- 59 App Studio
- 41 Variables
- 686 Automate
- 176 Apps
- 453 APIs & Domo Developer
- 47 Workflows
- 10 DomoAI
- 36 Predict
- 15 Jupyter Workspaces
- 21 R & Python Tiles
- 396 Distribute
- 113 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 125 Manage
- 122 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 109 Community Announcements
- 4.8K Archive