Domo Query API - Row limit?

imelendez

Hello, Experts,

 

Is there a limit to the # of rows you can pull down from a Domo dataset using the Domo Query Dataset method?

 

https://developer.domo.com/docs/dataset-api-reference/dataset#Query%20a%20DataSet

 

I have built a https request using python and then leveraging writing that response to a python pandas data frame. However, (full disclosure: have not tested writing to db yet) I can only see 1M rows in the data frame. Is 1M rows the limit?

 

I am doing something like:

 

{"sql": "select * from table"}

 

I will eventually clean this up to pull a smaller subset of data based on a data field in the dataset but wanted to get all the data in one run to load my historical DB on my remote server.

 

Let me know if this makes sense/ 

jaeW_at_Onyx

i do not know 100% but would not be surprised / it would make sense.  APIs wouldn't want to stream billion row datasets to you.  instead it would be secure (in case of network interruption) to send data in n-sized row chunks.  

 

so ... yeah.. makes sense.  

 

in a similar way, i would not read the data directly into a pandas dataframe, i think i would download the data to a csv, and then read the data into python.  that way there's fewer chances of catastrophic error in case of network outage.  

 

also ... even though it's free ... don't set yourself up for frustration.  SELECT * ?

for visualization we are very wasteful with data storage, we produce very fat, very tall datasets and Adrenaline handles it like a champ.  If you're sending decently large datasets over API, you probably want to be a bit more parsimonious with column selection.