Snowflake Connector Column Limit?
We're currently testing our Snowflake-Domo (key pair) connector and running into issues when querying wide tables from our Snowflake instance.
The table in question is ~12M rows and 400+ columns. Currently we import this into Domo with the Amazon S3 connector. This works, though can be slow (2-6 hours) to complete.
We've established the Snowflake Key-Pair connector successfully, but when we query this table we get the error "Failed to execute import successfully." I've trimmed the SELECT to only include ~250 columns, but received the same error.
I was able to successfully import with 100 columns (in 18m 44s), so it would appear that the range for columns that we can import is between 100 and 250. While a large # of columns isn't ideal (and we're actively trimming them down), this is the first time we've experienced this issue.
Is there a known limit to the # of columns that can be imported via Snowflake, or is there a better solution or configuration we should attempt for wide tables?
Answers
-
Tried 150 columns and it successfully completed after 32m 18s.
Tried 200 columns and is failed after 47m 26s.
Sounds like the sweet spot is somewhere in between.
0 -
Can you break it up into two datasets an then join it back together in Magic ETL?
**Check out my Domo Tips & Tricks Videos
**Make sure to any users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
That's an option we'd explore as a last case scenario, though I'd be worried about runtime/job success when joining 150+150+100 column tables at 12M rows a piece.
Short-term trying to figure out a column limit and going from there. May break these into use-case specific datasets from a single source anyway.
0
Categories
- All Categories
- 1.9K Product Ideas
- 1.9K Ideas Exchange
- 1.6K Connect
- 1.3K Connectors
- 303 Workbench
- 6 Cloud Amplifier
- 9 Federated
- 3K Transform
- 104 SQL DataFlows
- 640 Datasets
- 2.2K Magic ETL
- 4K Visualize
- 2.5K Charting
- 769 Beast Mode
- 72 App Studio
- 43 Variables
- 718 Automate
- 185 Apps
- 462 APIs & Domo Developer
- 57 Workflows
- 14 DomoAI
- 40 Predict
- 17 Jupyter Workspaces
- 23 R & Python Tiles
- 402 Distribute
- 116 Domo Everywhere
- 277 Scheduled Reports
- 9 Software Integrations
- 135 Manage
- 132 Governance & Security
- 8 Domo Community Gallery
- 44 Product Releases
- 12 Domo University
- 5.4K Community Forums
- 40 Getting Started
- 30 Community Member Introductions
- 111 Community Announcements
- 4.8K Archive