Snowflake Connector Column Limit?

We're currently testing our Snowflake-Domo (key pair) connector and running into issues when querying wide tables from our Snowflake instance.
The table in question is ~12M rows and 400+ columns. Currently we import this into Domo with the Amazon S3 connector. This works, though can be slow (2-6 hours) to complete.
We've established the Snowflake Key-Pair connector successfully, but when we query this table we get the error "Failed to execute import successfully." I've trimmed the SELECT to only include ~250 columns, but received the same error.
I was able to successfully import with 100 columns (in 18m 44s), so it would appear that the range for columns that we can import is between 100 and 250. While a large # of columns isn't ideal (and we're actively trimming them down), this is the first time we've experienced this issue.
Is there a known limit to the # of columns that can be imported via Snowflake, or is there a better solution or configuration we should attempt for wide tables?
Answers
-
Tried 150 columns and it successfully completed after 32m 18s.
Tried 200 columns and is failed after 47m 26s.
Sounds like the sweet spot is somewhere in between.
0 -
Can you break it up into two datasets an then join it back together in Magic ETL?
**Check out my Domo Tips & Tricks Videos
**Make sure toany users posts that helped you.
**Please mark as accepted the ones who solved your issue.0 -
That's an option we'd explore as a last case scenario, though I'd be worried about runtime/job success when joining 150+150+100 column tables at 12M rows a piece.
Short-term trying to figure out a column limit and going from there. May break these into use-case specific datasets from a single source anyway.
0
Categories
- All Categories
- Product Ideas
- 2.1K Ideas Exchange
- Data Connections
- 1.3K Connectors
- 309 Workbench
- 17 Cloud Integrations
- Data & ETL
- 2.3K Magic ETL
- 120 SQL DataFlows
- 666 Datasets
- Visualize & Apps
- 90 App Studio
- 198 Pro-code Components
- 2.6K Charting & Analyzer
- 873 Calculations & Variables (Beast Mode)
- AI & Data science
- 23 Domo AI & AI Chat
- 4 Managing AI
- 18 Jupyter Workspaces
- Automate
- 122 Workflows
- Alerts
- Distribute
- 118 Domo Everywhere
- 284 Reporting
- Manage
- 145 Governance & Security
- 489 APIs
- 11 Add-ins & Plugins
- 13 Domo Community Gallery
- 49 Product Releases
- 13 Domo University
- Community Forums
- 41 Getting Started
- 31 Community Member Introductions
- 118 Community Announcements
- 5K Archive