Extract 1 row / few rows from big dataset (>100M rows)
Comments
-
Hi,
It depends upon how your dataset is configured. For a standard dataset, the full dataset must load. Maybe if you can partition your dataset then you can speed this up.
Jarvis
0 -
Yeah this is a tough one...
In a nutshell any ETL tool (except Fusion) will have to transfer your data into the transformation engine, a SQL database or Magic's ... data processing environment, before you can transform it. Hence why your ETLs take so long.
Your goal should be to use VIEWS which go directly to the Adrenaline (our database layer) to transform / subset your data.
Up to about 5 months ago, your options for this were VERY limited... but ... stay tuned ...
There are updates to Product in the pipe that will make this story MUCH BETTER. Find your Domo Customer Success Manager (CSM) and ask them how you can create views in Domo. We've got new features coming to the UI that are really going to help you.
That said, if you're pretty nerdy you can use the JavaCLI to create a view without a pretty User Interface.
use get-schema to get the schema of an existing dataset in Domo, then use create-dataview... to ... create a dataview based on the schema you pulled with filters added.
Jae Wilson
Check out my 🎥 Domo Training YouTube Channel 👨💻
**Say "Thanks" by clicking the ❤️ in the post that helped you.
**Please mark the post that solves your problem by clicking on "Accept as Solution"2
Categories
- 10.5K All Categories
- 6 Connect
- 916 Connectors
- 250 Workbench
- 463 Transform
- 1.7K Magic ETL
- 69 SQL DataFlows
- 476 Datasets
- 190 Visualize
- 252 Beast Mode
- 2.1K Charting
- 11 Variables
- 17 Automate
- 354 APIs & Domo Developer
- 89 Apps
- 3 Workflows
- 20 Predict
- 5 Jupyter Workspaces
- 15 R & Python Tiles
- 246 Distribute
- 62 Domo Everywhere
- 243 Scheduled Reports
- 21 Manage
- 42 Governance & Security
- 172 Product Ideas
- 1.2K Ideas Exchange
- 11 Community Forums
- 27 Getting Started
- 14 Community Member Introductions
- 55 Community News
- 4.5K Archive