Fusion, how many rows is too many?

I have been watching @jaeW_at_Onyx on Youtube and am seriously excited by his data fusion methods. I would like to mimic his
how to build a Trial Balance, Income Statement and Balance Sheet with no ETL but have 28 million rows in my GL. Would it be more efficient to split said 28 million into yearly chunks, first using a row fusion and then add metric table in a column fusion or just have them in one huge dataset. I am currently using workbench to import and append recent transactions every 15 mins.

Find more posts tagged with

Comments

GrantSmith

@grimmers - Fusions should be able to easily handle 23M records. It should handle up to hunders of millions of rows just fine. Plus with the benefit of fusions updating near instantaneously they're a really great tool for the large data sets.

grimmers

@GrantSmith thanks for the reply. I am curious to know how Domo will handle this much data in a fusion with a cross join to a dimension table. I will be giving it a try today so fingers crossed!

jaeW_at_Onyx

you have to be careful if you're adding new data every 15 minutes.

When you add new data, Domo has to reindex the data in Adrenaline. If Adrenaline re-indexes the Fusions, I'm pretty confident it clears your cache. If you clear your cache, all your pages and cards will never benefit from a warm cache.

It should be fine as long as your cards are primarily dealing with low cardinality (few distinct values) data. If you're pulling in long text strings (descriptions of transactions) that can impact performance as Adrenaline compresses your data.

If you're worried about performance, cut long free text columns.

I don't think I would split up data into years.

Let us know how it goes!

grimmers

OK, after some testing with my 28 million rows and adding a cross join to a dimension table the blow out table went to 340 million rows with no visible performance issues. even with 15 minute updates of the transaction table there have been no problems. I am now a Fusion convert!!

Thanks to @GrantSmith , @jaeW_at_Onyx for their input

jaeW_at_Onyx

@grimmers that's wicked! glad to hear. If your dashboard does slow down, consider dropping columns in your Fusion.

If any of the responses from @GrantSmith or me helped, please feel free to mark them as a solution! Thanks!