Comments
-
Yeah, you'll still have to split the and gzip the query results but once you have created the parts, the Streams API is very quick if you program it to push asynchronously. For example, we've written a little python cmd line application for pushing data up to domo using the Streams API. On one dataset, we sent data to domo…
-
You could avoid the Domo Workbench software altogether and use python and dataframes for manipulating and joining your data from the multiple database queries you mentioned. Dataframes are convientent because they allow SQL like joining of data from within your script. Once you've joined the data using a pandas dataframe,…
-
Ahh gotchya. Thanks for the info. Very helpful as we test and try to optimize our own script. Let me know how the 100 MB file sizes work out!
-
Glad that you find them at least somewhat helpful @robsmith! We actually havent experimented too much with part size. On our tests, our split files were 100K rows each. That turned out to be gzipped files that were about 10-12 mb. Maybe too small? However, They still uploaded very quickly as mentioned above. How long has…
-
That was actually uncompressed. If I had Gzipped it, im sure the speed would have increased. Probably to somewhere around 1 minute vs the 3.5 minutes i was seeing. We just tested a similar dataset and successfully uploaded a gzipped 6 Million row csv in 60-90 seconds. On the Domo workbench, the send portion of the upload…
-
After further research and testing i realize that the "slowness" to the streams api that I was seeing was really just an error in my script. Once i correctly executed the asynchronous upload using the streams api I was able to upload a csv that was 6.2 Million rows and around 130 columns wide in 3.5 minutes.
-
Do you guys have any timeline on this?
-
Appreciate the help. I'll reach out to our account exec as suggested. Thanks
