Domo on handling tranactional data and dimensional changes type 1 and type 2?

CoreVest · December 2015

By using Domo’s built-in data capturing methods.
1. How does Domo handle transactional data?
  1. How does it handle duplicate data in regards to transactional data?
2. How does Domo handle dimensional changes (Type1 &Type2)?
  1. Type 1 - old data is overwritten by new data
  2. Type 2 – multiple records are created to track data historically.
3. We are concerned with creating daily snapshots of the data in our domo instances. By doing so, the data would start to grow exponentially.
  1. Is there a limit to the size of data we can store?
  2. Will there be performance issues?

jaromp · December 2015

Q1 - How does Domo handle transactional data?

A - I'm not sure exactly what is being asked here; all data is treated the same within Domo, and like any data within Domo, it is possible to adjust, combine, aggregate, and so on. Perhaps the answers to the rest of your questions will help clarify your question here.

Q1.1 - How does Domo handle duplicate data in regards to transactional data?

A - When importing data, Domo does not check for or remove duplicates. Duplicates can be removed from a dataset once it is in Domo using the data transformation tools within Domo (Magic ETL, Dataflow).

jaromp · December 2015

Q3 - We are concerned with creating daily snapshots of the data in our domo instances. By doing so, the data would start to grow exponentially.

Q3.1 - Is there a limit to the size of data we can store?

A - There is no set limit for datasets.

Q3.2 - Will there be performance issues?

A - Theoretically, the bigger the dataset, the longer it will take to process. The impact from large datasets is felt differently depending on which tools are being used, and how they are being used (indexing in dataflows, MySql dataflow vs Magic ETL, etc.). There are customers who are able to effectively use datasets with billions of rows.

jaromp · December 2015

The Domo append and replace functionality will work exactly the same on the Domo side for all connections. In other words, if "Append" is selected, Domo will take the new data supplied by the data source and append it to the existing dataset. If "Replace" is selected it will delete the old dataset and replace it with the new data supplied by the datasource.

The difference you are seeing could be a difference in the specifics that datasource; for example, a report from one datasource could return any records that have been updated in the last day, while a report from another datasource may return all the data each time it is called. Additionally, even two reports from the same datasource may be custom or configured to return the data differently.

This will vary so widely for each source system and customized reports within that system that we do not have any standard documentation covering those scenarios. When the datasource is created, the particulars for that datasource should be identified by whoever is setting up the connection. If there are questions about how the datasource is behaving, they can be directed to our support team.

nalbright · December 2015

Would you want these questions answered separately or do you want all of the various answers combined together?

CoreVest · December 2015

Separately please

jaromp · December 2015

Q1 - How does Domo handle transactional data?

A - I'm not sure exactly what is being asked here; all data is treated the same within Domo, and like any data within Domo, it is possible to adjust, combine, aggregate, and so on. Perhaps the answers to the rest of your questions will help clarify your question here.

Q1.1 - How does Domo handle duplicate data in regards to transactional data?

A - When importing data, Domo does not check for or remove duplicates. Duplicates can be removed from a dataset once it is in Domo using the data transformation tools within Domo (Magic ETL, Dataflow).

jaromp · December 2015

Q2 - How does Domo handle dimensional changes (Type 1 and Type 2)?

Type 1 - old data is overwritten by new data

Type 2 - multiple records are created to track data historically.

A - When uploading data to Domo, the user can choose to completely replace the dataset with new data or simply append the new data to the existing dataset.

Additionally, there are also ways to utilize dataflows to take the new data and append it conditionally (removing duplicates, etc) to a historical dataset that hold all the historical transactional data from that datasource.

CoreVest · December 2015

Thank you,

I just needed a documented response.

jaromp · December 2015

Q3 - We are concerned with creating daily snapshots of the data in our domo instances. By doing so, the data would start to grow exponentially.

Q3.1 - Is there a limit to the size of data we can store?

A - There is no set limit for datasets.

Q3.2 - Will there be performance issues?

A - Theoretically, the bigger the dataset, the longer it will take to process. The impact from large datasets is felt differently depending on which tools are being used, and how they are being used (indexing in dataflows, MySql dataflow vs Magic ETL, etc.). There are customers who are able to effectively use datasets with billions of rows.

CoreVest · December 2015

I am aware of the replace/append functionality. Can you provide any documentation on how each dataset behaves with replace/append. i have cases where two different data sources have the same settings set to append, one creates a daily copy of the data, the other only appends is data has changed. It is very difficult to configure datasets if there is no consistency.

CoreVest · December 2015

Thank you

jaromp · December 2015

The Domo append and replace functionality will work exactly the same on the Domo side for all connections. In other words, if "Append" is selected, Domo will take the new data supplied by the data source and append it to the existing dataset. If "Replace" is selected it will delete the old dataset and replace it with the new data supplied by the datasource.

The difference you are seeing could be a difference in the specifics that datasource; for example, a report from one datasource could return any records that have been updated in the last day, while a report from another datasource may return all the data each time it is called. Additionally, even two reports from the same datasource may be custom or configured to return the data differently.

This will vary so widely for each source system and customized reports within that system that we do not have any standard documentation covering those scenarios. When the datasource is created, the particulars for that datasource should be identified by whoever is setting up the connection. If there are questions about how the datasource is behaving, they can be directed to our support team.

Domo on handling tranactional data and dimensional changes type 1 and type 2?

Best Answers

Answers

Categories