Fusion, how many rows is too many?

grimmers
grimmers Member
in Magic ETL

I have been watching @jaeW_at_Onyx on Youtube and am seriously excited by his data fusion methods. I would like to mimic his 
how to build a Trial Balance, Income Statement and Balance Sheet with no ETL but have 28 million rows in my GL. Would it be more efficient to split said 28 million into yearly chunks,  first using a row fusion and then add metric table in a column fusion or just have them in one huge dataset. I am currently using workbench to import and append recent transactions every 15 mins.

Comments

  • GrantSmith
    GrantSmith Coach

    @grimmers - Fusions should be able to easily handle 23M records. It should handle up to hunders of millions of rows just fine. Plus with the benefit of fusions updating near instantaneously they're a really great tool for the large data sets.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • grimmers
    grimmers Member

    @GrantSmith thanks for the reply. I am curious to know how Domo will handle this much data in a fusion with a cross join to a dimension table. I will be giving it a try today so fingers crossed!

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    you have to be careful if you're adding new data every 15 minutes.

     

    When you add new data, Domo has to reindex the data in Adrenaline.  If Adrenaline re-indexes the Fusions, I'm pretty confident it clears your cache.  If you clear your cache, all your pages and cards will never benefit from a warm cache.

     

    It should be fine as long as your cards are primarily dealing with low cardinality (few distinct values) data.  If you're pulling in long text strings (descriptions of transactions) that can impact performance as Adrenaline compresses your data.

     

    If you're worried about performance, cut long free text columns.  

     

    I don't think I would split up data into years.

     

    Let us know how it goes!

     

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • grimmers
    grimmers Member

    OK, after some testing with my 28 million rows and adding a cross join to a dimension table the blow out table went  to 340 million rows with no visible performance issues. even with 15 minute updates of the transaction table there have been no problems. I am now a Fusion convert!!

    Thanks to @GrantSmith , @jaeW_at_Onyx  for their input

     

     

     

  • jaeW_at_Onyx
    jaeW_at_Onyx Coach

    @grimmers  that's wicked!  glad to hear.  If your dashboard does slow down, consider dropping columns in your Fusion.  

     

    If any of the responses from @GrantSmith  or me helped, please feel free to mark them as a solution!  Thanks!

    Jae Wilson
    Check out my 🎥 Domo Training YouTube Channel 👨‍💻

    **Say "Thanks" by clicking the ❤️ in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"

Categories