Automate file upload

I was only introduced to using Python inside Jupyter Notebooks in the past few months. But it has worked well. Somewhere out there are demo pages and videos. Short and sweet - I go to "Data" and on the left side, there are three dots. Click there and you will find Jupyter Notebooks. Within your Python code you can edit, check responses, debug portions at a time, save and schedule like you do Magic ETLs. It didn't take me long after watching some video I found through Google, to muster up the courage and create a "sample" workbook. I created a Python page and hit the run icon and it did what I told it. Then I got a little more advanced.

My first attempt at your question would be something like:

pip install pandas domo-py glob2

This would install the pandas, domo, and glob libraries. Then something like:

import os

import glob

import pandas as pd

from domo import Domo



# Configuration for Domo API

DOMO_CLIENT_ID = 'your_client_id'

DOMO_CLIENT_SECRET = 'your_client_secret'

DOMO_API_HOST = 'api.domo.com'  # or use 'api.domo.com' for production



# Folder where CSV files are located

CSV_FOLDER_PATH = '/path/to/your/csv_folder'



# Domo dataset details

DATASET_ID = 'your_dataset_id'  # Replace with your actual Dataset ID


def read_csv_files(folder_path):



    # List all CSV files in the specified folder

    csv_files = glob.glob(os.path.join(folder_path, '*.csv'))


    # Initialize an empty list to hold DataFrames

    dataframes = []



    # Loop through the list of CSV files

    for file in csv_files:



        # Read each CSV file into a DataFrame

        df = pd.read_csv(file)



        # Append the DataFrame to the list

        dataframes.append(df)



    # Concatenate all DataFrames into a single DataFrame

    combined_df = pd.concat(dataframes, ignore_index=True)

    return combined_df



def upload_to_domo(df, dataset_id):



    # Initialize Domo client

    domo = Domo(DOMO_CLIENT_ID, DOMO_CLIENT_SECRET, host=DOMO_API_HOST)


    # Convert DataFrame to CSV for upload

    csv_data = df.to_csv(index=False)


    # Upload the CSV data to Domo

    domo.datasets.data_import(dataset_id, csv_data, content_type='text/csv')

    print(f'Data uploaded to Domo dataset {dataset_id} successfully.')



# Main execution

if __name__ == '__main__':



    # Read and combine CSV files

    combined_df = read_csv_files(CSV_FOLDER_PATH)


    # Upload combined DataFrame to Domo

    upload_to_domo(combined_df, DATASET_ID)