API using partition

mroker
mroker Member
in Magic ETL

I have a question. I am using APIs now and I would like to know if it is possible to do partitioning using APIs. I know using Magic ETL and using Workbench you can do it, but can you do it using APIs? If I can't is the next step doing recursion?

The reason for asking is that I am uploading data daily and would like to add to it, and if any of the data is already there, would like to append to it, but not duplicate.

I don't see PyDomo having this feature so far.

Best Answer

  • ArborRose
    ArborRose Coach
    edited February 14 Answer ✓

    I don't believe you could do it in the API. As you state, you can implement that after receiving the dataset. I would think it would rely on the source of the API implementing the partition.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • ArborRose
    ArborRose Coach
    edited February 14 Answer ✓

    I don't believe you could do it in the API. As you state, you can implement that after receiving the dataset. I would think it would rely on the source of the API implementing the partition.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • mroker
    mroker Member

    Thank you @ArborRose for the answer. I will then use the replace daily and use partitioning on the MagicETL.

  • ArborRose
    ArborRose Coach

    @mroker - have a look in recent discussions for my question on "Effective Dating". This sounds exactly like what I am doing and the solution I created appears to work. My dataset is coming from an API scheduled daily. I append any records that are new or change to create an archive.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Categories