Update a single row in dataset

Options
mcoblentz
mcoblentz Member
in Archive

I am uploading sales quotes into Domo for analysis.  Some of the rows in the quotes are new, some are updated in place.  I don't know how to change just the updated rows.  I'm not keen to continually reload the quote tables - it's 50M rows - so is there another technique that can change a row in place?  

 

Thanks!

Comments

  • kshah008
    kshah008 Contributor

    Hi all,


    Can anybody help @mcoblentz out?

    Thanks!

  • ASperry
    ASperry Contributor

    Matthew,

     

    If your dataset is doing a full replace instead of an append, then the only options are to re-upload the data or correct the one row within a dataflow. 

     

    If you your dataset is appending, it's possible for us to delete the append and then the append could be replaced with the desired row changes.

     

    Thanks,

    Aaron Sperry

  • mcoblentz
    mcoblentz Member

    Hi Aaron, 

    I'm trying to avoid doing a full data set replacement (via workbench) because that's 56M rows.  So if we do append, then for the INSERTS that's pretty easy.  But the UPDATES (300K of them) is not quite so simple - is it?  If we delete the rows then if the UPDATE/APPEND does not work, we have a damaged dataset.  

     

    If we wanted to delete the affected row(s), how would I do that in Magic ETL anyway?  

  • ASperry
    ASperry Contributor

    Matthew,

     

    Thanks for the update! I didn't realize you were talking about workbench initially. Can you tell me where is your source data coming from? Are you able to create a transform within workbench that filters out those specific rows? If so this would the easiest route.

     

    Essentially, with an ETL you would simply filter out those rows you are looking to delete. Your ETL output dataset would then exclude those rows. The only problem I can think of with this would be the amount of time it would take for the ETL to run. It might take just as long as doing a full upload.

     

    Thanks,

    Aaron Sperry

Categories