Allow Virtual DataSets as Inputs to Redshift / MySQL Dataflows

user038578

We use the Virtual DataSets toolkit item to populate our secondary Domo instance with data from our primary Domo instance. Domo calls the resulting virtual dataset a federated dataset (the data provider is "domo-federateddata"). Unfortunately we cannot use the federated datasets as inputs to Redshift / MySQL ETL dataflows. Domo does not allow a federated dataset to be an ETL input unless it is Snowflake federated dataset.

This limitation makes it difficult to test ETL changes, which is the goal of our secondary / dev instance.

The recommended alternative is to use the DataSet Copy connector, but DataSet Copy can only update the destination instance on a schedule, whereas federated / virtualized datasets update the data in the destination instance immediately whenever the data in the source instance changes. Relying on DataSet Copy to mirror data between instances creates opportunity for a dataset to differ between instances, meaning an ETL that is identical between instances will yield different outputs due to differences in the input dataset. Depending on the frequency at which the ETL runs, the two instances may never produce the same output. Again, this limitation makes it difficult to test ETL changes, which is the entire goal of our secondary / dev instance.