API Partitioning

Options
mroker
mroker Member
edited February 13 in Magic ETL

Hi all, so I want to use the Domo API's to do partitioning. That is, I want to send data daily, and I do not want the data there to be overwritten. I am trying this but I think it is being overwritten:

https://api.domo.com/v1/datasets/{dataset_id}/data?updateMethod=PARTITION

Is this the correct way to upload via partition?

Also, where can I find good information and documentation on this?

Tagged:

Best Answer

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    If I grasp the question correctly…

    If you are setting up an API, you want to go to the Domo AppStore and find the proper API connector. Type JSON in the search. Something like the JSON no code connector. In the connector you will input your settings like the URL you show. "updateMethod" looks like a parameter. You should find fields for entering parameters for header, body, or date.

    Complete the form. Set your data structure and schedule. It will return a dataset. Partitioning is normally done using an ETL after you have the dataset from the API.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • GrantStowell
    GrantStowell Domo Moderator
    Options

    Hi @mroker, could you further clarify the issue you are trying to solve for?

  • ArborRose
    ArborRose Coach
    Answer ✓
    Options

    If I grasp the question correctly…

    If you are setting up an API, you want to go to the Domo AppStore and find the proper API connector. Type JSON in the search. Something like the JSON no code connector. In the connector you will input your settings like the URL you show. "updateMethod" looks like a parameter. You should find fields for entering parameters for header, body, or date.

    Complete the form. Set your data structure and schedule. It will return a dataset. Partitioning is normally done using an ETL after you have the dataset from the API.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • mroker
    mroker Member
    Options

    Thanks @ArborRose . So yes, I didn't know it was that easy in the ETL. Basically I wanted to use the API endpoint to partition the way Workbench does. I wanted to push data into Domo and if there was overlapping data, for it not to be duplicated. Also, if there was additional data, to append. So now what I am doing to solve that is to just use the replace on the API endpoint and I've created a ETL where the output is using a partition. It works perfectly.

  • ArborRose
    ArborRose Coach
    edited February 15
    Options

    @mroker - Glad to hear it's working.

    After you get your API configured, I suggest making sure the data structure matches your data. The JSON connectors will try to guess at what your data is. If your field holds a date, make sure its a date. Not a string, not datetime…a date. Make sure joining identifiers come across as string or whatever you need. Otherwise, you'll kill yourself later trying to convert all those fields to cast them into something they should have been from the start.

    With the No Code oAuth Connector, you can adjust the schema after its predicted the field types.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Categories