Limit on ranking tiles

dacorson
dacorson Member
in Magic ETL

I added 20 ranking tiles to me Magic ETL, but it fails stating out of memory. I removed 4 of them, and now it runs fine. Is there a limit to the number of ranking tiles you can use? The input is only 2 million records.

Tagged:

Answers

  • MichelleH
    MichelleH Coach

    Hi @dacorson I don't know of a specific limit on rank and window tiles. Could you please share a screenshot of the error you are seeing?

  • GrantSmith
    GrantSmith Coach

    @justin-millheim do you have any insight into this?

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**
  • [Deleted User]
    [Deleted User] Member

    @dacorson , Domo DataFlows Product Manager here. The only true limits we have in Magic ETL is whether or not we can complete the specified transformation steps in a 24 hour window. There are additional nuances to it but that's the most common limit one may encounter at one point or another when dealing with extremely large amount of data or complex DataFlows.

    For added context, the Rank and Window tile is one of the most "costly" actions in Magic ETL. What I mean by "costly" is that it is one of the most processing-time-intensive-actions we have as that action needs to evaluate every row of data against each other row within your ETL (similar to remove duplicates, but even more complex). Without having any additional information, the out of memory error you are experiencing seems normal to me as 2 million rows of data compared against each row of itself 20 times becomes an extremely large processing job very fast. If this DataFlow used to work with those specific transformations and stopped running successfully, or you feel there is a bug/error with Domo, then I encourage submitting a Support case as that will get reviewed and triaged to the appropriate engineering team for evaluation. With a support case and specific dataflow cited, Domo Support and Engineering can investigate further and provide additional help if you would like to go down that route.

    Thanks for the ping @GrantSmith

Categories