Does datafusion count against warehouse row count

jboettger
jboettger
in Magic ETL

Reading the documentation for datafusions:

DataFusion: Advantages and Disadvantages

  • Because a DataFusion is a just a view of the data, the output doesn’t have to be materialized and indexed into the Domo databases, which makes it so that data can be combined, visualized, and updated very quickly in Domo.

  • It makes data transformations fast when you’re working with a lot of data; more specifically, any DataSets with more than 20 million rows and >25 columns of data should be combined using a DataFusion.

  • You can leverage creative Beast Mode calculations within Analyzer to specify which data you want to display and how.

  • You can control naming of columns during initial setup of the DataFusion.

  • You can control which columns to include or exclude at any time.

  • You can only do joins or appends using DataFusion.

  • You cannot use DataFusions as inputs for DataFlows.

  • You have less backend control over renaming or editing your column names after the initial creation of the DataFusion.

The first bullet point uses the word "view".  To an Oracle developer it sounds as if the row count in the fusion would not actually count against the warehouse total row count (hints being "VIEW", "MATERIALIZED" and "INDEXED INTO DOMO").  When I create a 35M row fusion, my warehouse count increased by 35M.  Can anyone explain?

 

Thanks!

Comments

  • BrendanH
    BrendanH Domo Employee

     

    Even though the data has not been indexed every time the fusion runs it still has to store the data somewhere or you wouldn't be able to visualize the data in cards, so the data has 35M rows cached until the next time the datafusion runs.

  • jboettger
    jboettger

    Thanks 11057,

     

    But being stored (indexed) does not agree with the first bullet point:

    • Because a DataFusion is a just a view of the data, the output doesn’t have to be materialized and indexed into the Domo databases, which makes it so that data can be combined, visualized, and updated very quickly in Domo.

    To me, this says its virtual.

  • Microsoft-Dude
    Microsoft-Dude

    I to would love confirmation on this from Domo or from someone that has been with Domo for longer than a year where their record counts including Datafusions were over their record count limit, but if you omitted Datafusions, then the record count was within their contractual row count limits.  I was told by the technical salesperson that DataFusions don't count since they are not materialized (like a DB view).  This makes perfect since based on the documentation and is how I think it all works, but like I said.  I would love confirmation on this.  Without this, we would quickly go over our row count limit, but if we manage our data and ETL processes well, then we should be able to "view" our data in many different ways with DataFusion without it impacting our total row count.  If my assumption is correct and DataFusions don't count against your warehouse row count, then I would love to see how I can see my true warehouse row count.  I currently do this manually or grabbing the data through the API.

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    AFAIK ... depending on who you talk to at Domo the answer may flip flop depending on where the stat is collected from.  But I believe the correct answer should be Data Fusions DO NOT count.

     

    (I say this as a former employee of Domo, but you should check in with your CSM, and make sure they give you an answer in writing ? 

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • Microsoft-Dude
    Microsoft-Dude

    Thanks for the response Jae!  BTW - Love all the tutorials on your Youtube channel.

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    ? @Microsoft-Dude  ? subtle dig in the profile name?

     

    If you're a PowerBI guy, I'd love to have a conversation with you sometime, I need to do a YouTube video on "getting started in Domo for PowerBI developers" b/c building datasets in Domo is SO DIFFERENT.

     

     

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories