Currently, to filter data from a source like Google Sheets, Snowflake, or a database, a user must first ingest the entire dataset into Domo. Only after the data has landed in Domo can we apply filters using Magic ETL, SQL DataFlows, or Dataset Views.
This workflow has several significant drawbacks:
- Inefficiency: It consumes unnecessary time and resources to extract, transfer, and store large volumes of data that will be immediately discarded.
- Increased Complexity: It forces the creation of extra dataflows for what should be a simple filtering step, cluttering the data pipeline.
- Higher Costs: It inflates data storage and processing costs within the Domo platform.
- Delayed Insights: The time taken to import and then process large, unfiltered datasets delays the availability of fresh data for decision-making.
For example, if a Google Sheet contains 100,000 records but a user only needs the last two days' worth of data, they are forced to import all 100,000 rows and then build an ETL to filter them. This is a highly inefficient process.
