Connecting to a large dataset available for download

damen · February 6

Hi All.

I am working with try and connect to a few HMDA data sets that are available and am not sure how to do so.

I remember a consultant we had connecting to data from a federated data source but also, not sure if that is what we are needing to do.

More specifically, we are trying to connect the Loan/Application Records (LAR) CSV file from this site that is available below. It currently comes in a zipped file and when I unzip it, the file is almost a full 6MB - and too big to connect to directly.

https://ffiec.cfpb.gov/data-publication/snapshot-national-loan-level-dataset/2022

Can someone help with this?

ArborRose · February 7

According to Domo support documentation, Domo Workbench has no limit. I use Workbench to schedule daily uploads. If that doesn't work, you might be able to do it with Python. I extract using Python but I never tried uploading using Python.

https://domo-support.domo.com/s/article/360043437373?language=en_US#:~:text=What%20is%20the%20maximum%20size,the%20Workbench%205%20User%20Interface.

MichelleH · February 7

@damen What connector are you trying to use? The File Upload connector can handle files up to 250 MB, so a 6 MB file should not be a problem

damen · February 7

@MichelleH I am trying to use the file connector

Sorry I did horrible math. It is 5,916,478 KB so its almost 6,00MB

ArborRose · February 7

According to Domo support documentation, Domo Workbench has no limit. I use Workbench to schedule daily uploads. If that doesn't work, you might be able to do it with Python. I extract using Python but I never tried uploading using Python.

https://domo-support.domo.com/s/article/360043437373?language=en_US#:~:text=What%20is%20the%20maximum%20size,the%20Workbench%205%20User%20Interface.

Connecting to a large dataset available for download

Best Answer

Answers

Categories