Changing dataset update method from Replace to Partition doubles data
We've noticed that if we change the update method of an existing dataset to Partition from replace, we end up with two records in the dataset for every new one (and both of them look identical, down to the field we're using for partitioning): it was quite the shock to see a dataset I expected to have 91 million rows suddenly had 182 million. The obvious takeaway is that we probably should start from scratch when using partitions, but the benefits of converting existing ETLs is too strong a siren song for me to resist.
Two questions:
- Has anyone else noticed this? Both of us here got bit by this, so I want to know if it's a general bug, if it just affects our instance, or if we're doing it wrong and it's Working As Designed™.
- What would be a way around this? If I have other ETLs downstream of the dataset I don't want to delete the existing one and start from scratch unless I absolutely have to. My quick-and-probably-inefficient idea is to store the data to a secondary dataset (deduping if necessary), set the original to partition, and figure out how to use the even more beta feature to tell it to keep no partitions. Let it run once to clear out the dataset, then reimport the data from the secondary dataset and send it back to the original.
Any help would be appreciated.
Best Answers
-
Well that's annoying, @Jones01. I'll go ahead and put in a ticket and hope you're wrong, because doubling the number of instantiated rows could get expensive for everyone.
Thanks both of you for the response.
0
Answers
-
This sounds like a bug. I'd recommend logging a ticket with Domo Support.
**Was this post helpful? Click Agree or Like below**
**Did this solve your problem? Accept it as a solution!**0 -
Well that's annoying, @Jones01. I'll go ahead and put in a ticket and hope you're wrong, because doubling the number of instantiated rows could get expensive for everyone.
Thanks both of you for the response.
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 296 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive