Many rows

mroker
mroker Member
in Magic ETL

I need to find a way to pull in many rows of data. I have two months worth of data which is essentially 110M rows. I need to pull in at least a year. Is there a way to do this? I tried the ETL partitioning method but it said 'too many rows of data'. Would workbench work with partitioning here or would I get the same error? Any suggestions?

Tagged:

Best Answer

  • MichelleH
    MichelleH Coach
    Answer ✓

    @mroker If this is a one time import of historical data, then appending would likely be simpler than partitioning.

Answers

  • MichelleH
    MichelleH Coach

    @mroker Where is your data currently stored? Depending on whether it's a spreadsheet, ODBC, cloud, etc. there are different connection options available. If importing all the rows at once is an issue, then you may need to upload in batches and appending the batches onto each other.

  • mroker
    mroker Member

    ODBC. So, I can append, was hoping to partition. I just wanted to see what the limitations are like on the ETL/Dataset side of things. At the end of the day, having a year's worth of data would be like 700M rows which to me is a massive dataset. Also, wanted to make sure partition would even work.

  • MichelleH
    MichelleH Coach
    Answer ✓

    @mroker If this is a one time import of historical data, then appending would likely be simpler than partitioning.

Categories