Paritoning beta. Where to start?
I've got access to the partitioning beta and trying to figure out where to start. I have a number of legacy datasets (marketing platforms like Facebook, LinkedIn, Google Ads and Google Analytics) and dataflows that are clunky at best and go the long route to handle recursive data. I'm looking to simplify and make these dataflows more efficient but I am stuck. For example, do you rebuild your datasets especially if you are using connectors? If so how best to do that to leverage partitioning?
Best Answer
-
@dkonig It depends on the connector. The ones that support partitioning will have a "merge" method which uses partitions. If the connector you have been using does not have the merge method, there may be another version of the connector that does. When paired with MagicETL subset processing (which I assume is the beta you are referring to), Domo detects which rows in the input datasets have been added/updated and only processes those rows. In many cases this results in a much faster data pipeline.
Here are a couple other articles that talk about partitioning theory:
1
Answers
-
@user029082 I suggest checking out this recording from one of the previous Lunch & Learn streams. There are some great examples and use cases for partitioning!
1 -
Yes. I remember this video. I'll need to go back and re-watch since I saw it before I got beta access.
1 -
This video is from back in January. It definitely mentions the use cases I am interested in but I am still not sure exactly how to configure things. I'm still unclear on what to partition on especially when a dataset that are created with connectors without partitioning are used in the dataflow
1 -
@user029082 In that case I would recommend reaching out directly to Andrea or the beta team to help you with the your specific use cases.
1 -
Maybe I'm just not understanding how/why to use partitioning. With a connector that only has append and replace options can you even use partitioning to pull in new data and replace(update), let's say, the last 30 days?
1 -
@dkonig It depends on the connector. The ones that support partitioning will have a "merge" method which uses partitions. If the connector you have been using does not have the merge method, there may be another version of the connector that does. When paired with MagicETL subset processing (which I assume is the beta you are referring to), Domo detects which rows in the input datasets have been added/updated and only processes those rows. In many cases this results in a much faster data pipeline.
Here are a couple other articles that talk about partitioning theory:
1 -
Hmm. Ok. So if the connector is doesn't have merge then it's going to either replace or append no matter what. I only gain the benefit of partitioning in MagicETL if I need to use that data for other reporting. The partitioning allows me to update the partitions I want.
For example let's take Google Ads. There is usually some time for recent data to 'cure' so a recent range of dates may have data that changes from day-to-day. The connector would run every day and replace the entire dataset and then, via MagicETL, I could set up a dataflow that takes all the old existing data, that does not change, and just grab the newer data based on whatever settings I give it. This saves me all the reprocessing of the old data. Is that about right?
0 -
@MichelleH Does my previous comment make sense or am I still not grasping?
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 296 Workbench
- 6 Cloud Amplifier
- 8 Federated
- 2.9K Transform
- 100 SQL DataFlows
- 614 Datasets
- 2.2K Magic ETL
- 3.8K Visualize
- 2.5K Charting
- 729 Beast Mode
- 53 App Studio
- 40 Variables
- 677 Automate
- 173 Apps
- 451 APIs & Domo Developer
- 45 Workflows
- 8 DomoAI
- 34 Predict
- 14 Jupyter Workspaces
- 20 R & Python Tiles
- 394 Distribute
- 113 Domo Everywhere
- 275 Scheduled Reports
- 6 Software Integrations
- 121 Manage
- 118 Governance & Security
- Domo Community Gallery
- 32 Product Releases
- 10 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 108 Community Announcements
- 4.8K Archive