Is there a way to limit the number of input records on the ETL dataflow ?


I have 10 GB of data and I want to run a python script on that. But I am getting Out of memory issue because of the size of data.

I want to limit the number of records from the input dataset. How to do it? Thanks in advance for suggestions and solutions.


  • Unknown



    Are you using the Magic Transformation: ETL? Or how are you running python scripts?


    If you are using the ETL, what about using the Filter and just choose a subset of your data based on any condition you'd prefer?

    ETL FilterETL Filter