2 Issues: MySQL SSH - Query Parameter and STREAM API
Sorry for combo post but I got problems.
1. I'm using the MySQL SSH Connector - I'm using the QUERY option to pull in data from a table. Currently, most are just daily imports but some tables are 1B rows and so doing a 'Select * from table;' is VERY inefficient. What I'd like to do is using a parameter to only pull in the most recent data (if the source table has a date field or if the source field has a PK ID that I could use). It looks like the query parameter option was intended to do that, but I don't see how it would store a variable so it could reference it when it imported. I know the other option here is to just do a recurse DF but it will slow me down so I'm trying to reduce waste where I can.
2. Am I right in thinking that this may be a good case for Stream API? It looks in the documentation that it only works for very large flat files, but that doesn't make sense to me...so I'm hoping I'm wrong with that. I'm not a programmer either so any helpful advice here is VERY much appreciated.
Thanks everyone
Matt
Best Answer
-
Answers:
1. no stored system variable - must find it myself using a date/id field and go from there....build recursive to just filter out the dupes.
2. Maybe - but you have to build code to export source data to a flat file to then import in your stream which sucks.
Alternative: There is an option for data assembler for big datasets -going to try it out (2b row table may need it). Also I got workbench working thru an SSH connector (putty/odbc/port forwarding) and that works sometimes ok...but seems to fail at the most inopportune times w/o good error messages. - closing this up.
0
Answers
-
Answers:
1. no stored system variable - must find it myself using a date/id field and go from there....build recursive to just filter out the dupes.
2. Maybe - but you have to build code to export source data to a flat file to then import in your stream which sucks.
Alternative: There is an option for data assembler for big datasets -going to try it out (2b row table may need it). Also I got workbench working thru an SSH connector (putty/odbc/port forwarding) and that works sometimes ok...but seems to fail at the most inopportune times w/o good error messages. - closing this up.
0
Categories
- All Categories
- 1.4K Product Ideas
- 1.4K Ideas Exchange
- 1.4K Connect
- 1.1K Connectors
- 282 Workbench
- 3 Cloud Amplifier
- 4 Federated
- 2.8K Transform
- 86 SQL DataFlows
- 548 Datasets
- 2.2K Magic ETL
- 3.2K Visualize
- 2.3K Charting
- 544 Beast Mode
- App Studio
- 26 Variables
- 566 Automate
- 134 Apps
- 411 APIs & Domo Developer
- 21 Workflows
- DomoAI
- 28 Predict
- 12 Jupyter Workspaces
- 16 R & Python Tiles
- 345 Distribute
- 87 Domo Everywhere
- 257 Scheduled Reports
- 1 Software Integrations
- 85 Manage
- 84 Governance & Security
- 8 Product Release Questions
- Community Forums
- 41 Getting Started
- 27 Community Member Introductions
- 81 Community Announcements
- 4.8K Archive