Can you stream Salesforce data in real-time?

travorhouse

Hi Domo Community,

Can you stream data from Salesforce into Domo in real-time?

We recently switched to Domo for dashboarding and reporting. Our call center relies on Domo dashboards to set in-person appointments. The issue is that by default data can only be sent to Domo every 15 minutes. (Note: Our previous tool allowed us to stream data in real-time.)

Because of this limitation, sometimes people get overbooked. To solve this, we are looking to stream data instantaneously. We've looked into using the Stream API, but noticed that it only allows commits every 15 minutes as well.

Our Domo rep said that they're able to get this time down to about 5 minutes for us, which will be super helpful. But, I'm still curious if there's a way to get that time down further than 5 minutes. I can't imagine that we are the only ones who have run into this issue.

Does anybody have a solution or recommendation for streaming data in real-time?

Thanks in advance,

Travor

jaeW_at_Onyx

@travorhouse is it possible for Domo to ingest data and store it in Domo in (almost) real time? ... theoretically yes.

Domo does have IOT use cases but they are still updating Domo in batches (increasingly smaller batches but still not true streaming). It just costs money. and Domo's pricing for low latency pipelines historically has not been ... something you can just look up :D

You could trigger a dataset to update as soon as it finished the previous update using automation something like what's presented in this video: https://www.youtube.com/watch?v=v-8HYp6b4KU&t=8s (but Domo wouldn't thank you for it) :D

for smaller batches, especially if you can do some sort of incremental load scheme or take advantage of a webhook connector + APPEND https://youtu.be/YOaOzpXGtbo, you could get pretty fast updates (probably lower than 5 minutes ON AVERAGE.

but i stress average, because sometimes the Domo platform has ... stuff going on where your batch updates might be delayed 5+ minutes while under the covers system maintenance stuff is going on.

if you needed TRUE streaming, your best bet might be to accumulate the data outside of Domo and then use a Federated Query to access the data, but then you can't transform it in Domo.

https://domohelp.domo.com/hc/en-us/articles/360042932974-Using-the-Federated-Data-Solution

MarkSnodgrass

Depending on what is available in your salesforce instance, you could leverage the JSON Webhook Connector to push data to Domo in real-time.

https://domohelp.domo.com/hc/en-us/articles/360042931494-JSON-Webhook-Connector

Using JSON Webhooks, you can get access to real-time data from weather devices, vending machines, wearables, and more. The Webhook link includes a long-living single use token so anyone with the link can publish data to your DataSet. 

jaeW_at_Onyx

+1 with @MarkSnodgrass , except it's important to keep in mind that even the webhook connector executes (and therefore ingests data) in batches. despite it collects data from a webhook pipeline, it must follow the same rules constraints of any other connector in Domo's arsenal (i.e. you do have to execute it in Domo and that data does have to get loaded into Amazon S3 / Vault and then it does have to get loaded into Adrenaline).

It is not true streaming it is still processed in batches and therefore falls prey to some of the stuff i described above.

jaeW_at_Onyx

@travorhouse is it possible for Domo to ingest data and store it in Domo in (almost) real time? ... theoretically yes.

Domo does have IOT use cases but they are still updating Domo in batches (increasingly smaller batches but still not true streaming). It just costs money. and Domo's pricing for low latency pipelines historically has not been ... something you can just look up :D

You could trigger a dataset to update as soon as it finished the previous update using automation something like what's presented in this video: https://www.youtube.com/watch?v=v-8HYp6b4KU&t=8s (but Domo wouldn't thank you for it) :D

for smaller batches, especially if you can do some sort of incremental load scheme or take advantage of a webhook connector + APPEND https://youtu.be/YOaOzpXGtbo, you could get pretty fast updates (probably lower than 5 minutes ON AVERAGE.

but i stress average, because sometimes the Domo platform has ... stuff going on where your batch updates might be delayed 5+ minutes while under the covers system maintenance stuff is going on.

if you needed TRUE streaming, your best bet might be to accumulate the data outside of Domo and then use a Federated Query to access the data, but then you can't transform it in Domo.

https://domohelp.domo.com/hc/en-us/articles/360042932974-Using-the-Federated-Data-Solution

MarkSnodgrass

Depending on what is available in your salesforce instance, you could leverage the JSON Webhook Connector to push data to Domo in real-time.

https://domohelp.domo.com/hc/en-us/articles/360042931494-JSON-Webhook-Connector

Using JSON Webhooks, you can get access to real-time data from weather devices, vending machines, wearables, and more. The Webhook link includes a long-living single use token so anyone with the link can publish data to your DataSet. 

jaeW_at_Onyx

+1 with @MarkSnodgrass , except it's important to keep in mind that even the webhook connector executes (and therefore ingests data) in batches. despite it collects data from a webhook pipeline, it must follow the same rules constraints of any other connector in Domo's arsenal (i.e. you do have to execute it in Domo and that data does have to get loaded into Amazon S3 / Vault and then it does have to get loaded into Adrenaline).

It is not true streaming it is still processed in batches and therefore falls prey to some of the stuff i described above.

travorhouse

Thanks for the response @jaeW_at_Onyx and @MarkSnodgrass ! I am going to get our engineering team to take a look at these suggestions.