Appending large datasets: Magic ETL vs Redshift
Hi,
I have a need to append large datasets together with Domo, (Google analytics data in Big Query).
According to the documentation when you have inputs larger than 100m rows you should use Redshift to transform the data.
I compared doing large dataset appends in Magic ETL against Redshift and they both took a similar amount of time to complete. I was wondering what is the rationale behind the recommendation to use redshift when there doesn't seem to be an improvement in performance?
Thanks
Comments
-
Are you only appending these two datasets, or are you doing more calculations? If you only want to append them, you may want to consider using DataFusion since that's specifically designed for simple joins/appends on very large datasets.
0
Categories
- All Categories
- 1.8K Product Ideas
- 1.8K Ideas Exchange
- 1.6K Connect
- 1.2K Connectors
- 300 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 2.9K Transform
- 102 SQL DataFlows
- 626 Datasets
- 2.2K Magic ETL
- 3.9K Visualize
- 2.5K Charting
- 754 Beast Mode
- 61 App Studio
- 41 Variables
- 693 Automate
- 178 Apps
- 456 APIs & Domo Developer
- 49 Workflows
- 10 DomoAI
- 38 Predict
- 16 Jupyter Workspaces
- 22 R & Python Tiles
- 398 Distribute
- 115 Domo Everywhere
- 276 Scheduled Reports
- 7 Software Integrations
- 130 Manage
- 127 Governance & Security
- 8 Domo Community Gallery
- 38 Product Releases
- 11 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 110 Community Announcements
- 4.8K Archive