Trying to append data from databricks



My requirement is to pull the data - run a query 90 columns from databricks table for 12 month which is around 2M records of data and append it to the existing dataset via API

Since its the large volume of data, I'm getting error as '"status":400,"statusReason":"Bad Request","message":"Too many non-consecutive part ids. Make consecutive or upload multiple data versions'.

I then tried to run the query by month and append it to the dataset using



Now I get error 'Python int too large to convert to C long'.

Can someone help me here or provide the proper syntax to append the data daily to the existing dataset using python script that I can run from databricks?

Best Answer