[BigQuery data] Delete data from certain days
Hi Guys,
I have a dataset that imports data from BigQuery everyday. Every morning a query is scheduled to get data from the previous day and append it to the data set.
Unfortunately, data from 1//26/2020 and 1/27/2020 was incomplete at the time the query ran and so the data I have in DOMO for these two days is wrong.
Is there a way to delete the data for these two days from the dataset so I can rerun the query and append the correct data to the dataset?
The dataset is very big so ideally I'd like to avoid replacing the whole data.
Thank you for your help!
Julien
Comments
-
When this happens to us, I will go into the dataflow that's appending the data and add a transform (SQL) to delete the rows I need to delete, or add a filter (ETL) to filter out the rows I want to remove. Then, after I run the dataflow once, I go in and delete the filter/transform. If the rows that came in for those dates would be duplicated if you reimported those dates and you're using SQL, using a UNION will remove the duplicate rows. I'm unsure about ETL, but I believe there's a remove duplicates function in there as well.
EDIT: Here's the SQL you'd use:
DELETE FROM <TABLE_NAME> WHERE <COLUMN_NAME> <CONDITION ie: = 1, <=5, IN ('list','of','things')>
0
Categories
- All Categories
- 1.7K Product Ideas
- 1.7K Ideas Exchange
- 1.5K Connect
- 1.2K Connectors
- 292 Workbench
- 4 Cloud Amplifier
- 8 Federated
- 2.8K Transform
- 95 SQL DataFlows
- 602 Datasets
- 2.1K Magic ETL
- 3.7K Visualize
- 2.4K Charting
- 697 Beast Mode
- 43 App Studio
- 39 Variables
- 658 Automate
- 170 Apps
- 441 APIs & Domo Developer
- 42 Workflows
- 5 DomoAI
- 32 Predict
- 12 Jupyter Workspaces
- 20 R & Python Tiles
- 386 Distribute
- 111 Domo Everywhere
- 269 Scheduled Reports
- 6 Software Integrations
- 113 Manage
- 110 Governance & Security
- 8 Domo University
- 30 Product Releases
- Community Forums
- 39 Getting Started
- 29 Community Member Introductions
- 98 Community Announcements
- Domo Community Gallery
- 4.8K Archive