Connect to SFTP - Select ALL files in root directory

gllrusso · 2021-10-04T10:52:23+00:00

There was an error rendering this rich post.

Hello, I'm here to ask you if there is any possibility with the built-in connectors to connect to an SFTP server and get the data coming from ALL .csv files in the root directory (supposing all files have the same schema).

This is because I'm receiving new files on a daily/weekly basis and I would definitely want to automatize the process of importing them to Domo.

If (as I think) this is not doable, do you think it is possible to create a custom connector for doing this? If so, how can I connect through SFTP in javascript?

Find more posts tagged with

Comments

GrantSmith

Hi @usergru

It's not possible with the SFTP connector however have you thought about utilizing workbench, scheduling an SFTP sync to download files in the folder to a DataSync folder and let workbench handle creating any jobs that are needed? https://domohelp.domo.com/hc/en-us/articles/4407019838487-Configuring-a-DataSync-Folder-in-Workbench-5-1

gllrusso

We did not think about Workbench because we would like to avoid needing a local machine running this and if I'm not wrong this is what Workbench is based on, am I right?

May a custom connector be a possible solution to avoid local applications?

GrantSmith

A custom connector may prove to be difficult as it's must be written on ES5 which is an older version. You may be able to hack something together to interact with an SFTP folder, pull all the files and then loop through all of the files but you'll be restricted to what you can utilize. I'd be interested to hear if you're able to get one written and working.

Workbench does run on a machine (virtual or physical) within your own intranet.

gllrusso

Just FYI, we opted for a python script working with Stream API. Creating a connector was not feasible for us.