Why are published datasets presenting to subscribers as federated?

lbg3222

I have an excel dataset in the parent instance that I've published to a subscriber instance that shows up in the subscriber's instance as federated data which prevents the subscriber from doing anything but creating cards from that dataset. There is one PDP policy set up on the excel dataset in the parent instance so that each subscriber is only viewing the data associated with their broker id. So, the dataset copy connector doesn't work as a work around for this issue as it doesn't factor in the PDP.

JohnMiller

Thanks, @lbg3222 !
Looks like we may have a feature switch that needs to get enabled in that instance. The easiest path is to connect with your CSM to have them get that enabled. We have the feature switch but have not enabled it for all Subscribers because there are use cases that prefer it disabled (vs enabled).

We can certainly help get that enabled on any instance you’d prefer it enabled on (and included in the template used for any future instances you may utilize, as well).

MarkSnodgrass

That's a good question and worth submitting an enhancement in the ideas exchange to allow subscribers to build ETLs off of the federated dataset.

Published datasets are federated because of how Domo is pushing the data around behind the scenes (from what I know.)

Your CSM could likely put you in contact with someone from the Domo Everywhere team to get a more thorough explanation of how it is set up.

GrantSmith

Simply put the publish dataset is being read from the main instance and not the publication instance. Same concept as a federated dataset but instead of a Domo database it’s pulling from a different database like snowflake

lbg3222

@MarkSnodgrass @GrantSmith We can't be the only use case for this. Does anyone have an idea for a work around? Our subscribers need to be able to use these datasets in an ETL or else we don't really have use for Domo Everywhere…

JohnMiller

Hey team

Great conversation.

Domo uses federated datasets in publish primarily for scale/speed, and cost. The scale/speed comes into play when delivering large amounts of data out to subscriber instances. Copying it to each would just take that much longer, especially as the number of subscribers increases. And any updates would get copied. As you’d imagine, this becomes especially problematic at scale.

The cost aspect is that it saves materializing those rows inside the subscribers, so is effectively an unlimited amount of data that can get used inside those instances.

Published datasets can get used in ETL inside the subscribers, and the resultant datasets are materialized. If you are experiencing issues in that happening, please let us know! The core product functionality is certainly to have a method to use those in ETL.

We have some other options available for copying and materializing data inside other environments that we can talk through, as well, but wanted to first see if this info helps.

Thanks, again!

lbg3222

@JohnMiller Thank you for your response. The only option available to the subscribers with the published dataset is to Open With Analyzer. The screenshot below shows a published dataset in the Subscriber instance that is originally an excel uploaded dataset in the main instance.

JohnMiller

Thanks, @lbg3222 !
Looks like we may have a feature switch that needs to get enabled in that instance. The easiest path is to connect with your CSM to have them get that enabled. We have the feature switch but have not enabled it for all Subscribers because there are use cases that prefer it disabled (vs enabled).

We can certainly help get that enabled on any instance you’d prefer it enabled on (and included in the template used for any future instances you may utilize, as well).

Mary_Moilanen

@lbg3222, did you get this figured out?

I have a similar issue. My subscribers do have the ability to create MagicETLs, but when I applied PDPs in the subscriber site (vs. just on what I was publishing to them), that ability went away. I was told that was expected behavior.

My other issue here is that my users can't save Views, meaning they have to create a MagicETL for every. single. little. transformation they're trying to do (e.g. simple joins, adding a calculated column). Looks like you had this problem too. That's annoying to my users, but the bigger issue is that it will cost us a fortune on consumption pricing. I sumitted a support case thinking it was a break/fix issue, since they can create the view in the first place (it just won't save), but was told this was working as expected and, since saving Views on Federated data sets was an "edge use case" they wouldn't be changing it. Since all published data sets are Federated for the subscriber, this doesn't exactly feel like an edge use case to me, lol!