APIs & Domo Developer

APIs & Domo Developer

Automate file upload

Hello! I am receiving multiple CSV files via email, and I need to automate the process of uploading this data to Domo. Unfortunately, I can't use the email connector because they send many files in one email. However, I can upload all the files into a folder. Do you have a solution for this? Thanks!

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In

Best Answer

  • Coach
    Answer ✓

    I was only introduced to using Python inside Jupyter Notebooks in the past few months. But it has worked well. Somewhere out there are demo pages and videos. Short and sweet - I go to "Data" and on the left side, there are three dots. Click there and you will find Jupyter Notebooks. Within your Python code you can edit, check responses, debug portions at a time, save and schedule like you do Magic ETLs. It didn't take me long after watching some video I found through Google, to muster up the courage and create a "sample" workbook. I created a Python page and hit the run icon and it did what I told it. Then I got a little more advanced.

    My first attempt at your question would be something like:

    1. pip install pandas domo-py glob2

    This would install the pandas, domo, and glob libraries. Then something like:

    1. import os
    2. import glob
    3. import pandas as pd
    4. from domo import Domo


    5. # Configuration for Domo API
    6. DOMO_CLIENT_ID = 'your_client_id'
    7. DOMO_CLIENT_SECRET = 'your_client_secret'
    8. DOMO_API_HOST = 'api.domo.com' # or use 'api.domo.com' for production


    9. # Folder where CSV files are located
    10. CSV_FOLDER_PATH = '/path/to/your/csv_folder'


    11. # Domo dataset details
    12. DATASET_ID = 'your_dataset_id' # Replace with your actual Dataset ID

    13. def read_csv_files(folder_path):

    14.  
    15. # List all CSV files in the specified folder
    16. csv_files = glob.glob(os.path.join(folder_path, '*.csv'))

    17. # Initialize an empty list to hold DataFrames
    18. dataframes = []


    19. # Loop through the list of CSV files
    20. for file in csv_files:

    21.  
    22. # Read each CSV file into a DataFrame
    23. df = pd.read_csv(file)

    24.  
    25. # Append the DataFrame to the list
    26. dataframes.append(df)


    27. # Concatenate all DataFrames into a single DataFrame
    28. combined_df = pd.concat(dataframes, ignore_index=True)
    29. return combined_df


    30. def upload_to_domo(df, dataset_id):

    31.  
    32. # Initialize Domo client
    33. domo = Domo(DOMO_CLIENT_ID, DOMO_CLIENT_SECRET, host=DOMO_API_HOST)

    34. # Convert DataFrame to CSV for upload
    35. csv_data = df.to_csv(index=False)

    36. # Upload the CSV data to Domo
    37. domo.datasets.data_import(dataset_id, csv_data, content_type='text/csv')
    38. print(f'Data uploaded to Domo dataset {dataset_id} successfully.')


    39. # Main execution
    40. if __name__ == '__main__':

    41.  
    42. # Read and combine CSV files
    43. combined_df = read_csv_files(CSV_FOLDER_PATH)

    44. # Upload combined DataFrame to Domo
    45. upload_to_domo(combined_df, DATASET_ID)

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

Answers

  • Contributor

    @vaco take a look at the Amazon S3 Advanced connector. I think this will pickup multiple files.

    https://domo-support.domo.com/s/article/360043436353?language=en_US

  • Even though your e-mail has multiple files, you can set up multiple forwarding rules and use the dataset via e-mail connector to handle each file. You can use the attachment name expression field to differentiate which file you want to connect to. Check out the documentation in this KB article.


    https://domo-support.domo.com/s/article/360042931954?language=en_US

    **Check out my Domo Tips & Tricks Videos

    **Make sure to <3 any users posts that helped you.
    **Please mark as accepted the ones who solved your issue.
  • Thank you @Jones01 and @MarkSnodgrass ! I will try both methods.😀

  • I like @MarkSnodgrass suggestion. You could also use Python scripts using the imaplib, email, and os libraries to automate the download of attachments from emails. Alternatively, third-party tools could possibly automate the download process.

    Domo Workbench could probably do it, but you would need to handle the specifics for files, subfolders, and transformations.

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • Member
    edited September 2024

    Thanks @ArborRose! I will look into using python for this automation. I will probably ask some follow up questions on that.😁

  • Coach
    Answer ✓

    I was only introduced to using Python inside Jupyter Notebooks in the past few months. But it has worked well. Somewhere out there are demo pages and videos. Short and sweet - I go to "Data" and on the left side, there are three dots. Click there and you will find Jupyter Notebooks. Within your Python code you can edit, check responses, debug portions at a time, save and schedule like you do Magic ETLs. It didn't take me long after watching some video I found through Google, to muster up the courage and create a "sample" workbook. I created a Python page and hit the run icon and it did what I told it. Then I got a little more advanced.

    My first attempt at your question would be something like:

    1. pip install pandas domo-py glob2

    This would install the pandas, domo, and glob libraries. Then something like:

    1. import os
    2. import glob
    3. import pandas as pd
    4. from domo import Domo


    5. # Configuration for Domo API
    6. DOMO_CLIENT_ID = 'your_client_id'
    7. DOMO_CLIENT_SECRET = 'your_client_secret'
    8. DOMO_API_HOST = 'api.domo.com' # or use 'api.domo.com' for production


    9. # Folder where CSV files are located
    10. CSV_FOLDER_PATH = '/path/to/your/csv_folder'


    11. # Domo dataset details
    12. DATASET_ID = 'your_dataset_id' # Replace with your actual Dataset ID

    13. def read_csv_files(folder_path):

    14.  
    15. # List all CSV files in the specified folder
    16. csv_files = glob.glob(os.path.join(folder_path, '*.csv'))

    17. # Initialize an empty list to hold DataFrames
    18. dataframes = []


    19. # Loop through the list of CSV files
    20. for file in csv_files:

    21.  
    22. # Read each CSV file into a DataFrame
    23. df = pd.read_csv(file)

    24.  
    25. # Append the DataFrame to the list
    26. dataframes.append(df)


    27. # Concatenate all DataFrames into a single DataFrame
    28. combined_df = pd.concat(dataframes, ignore_index=True)
    29. return combined_df


    30. def upload_to_domo(df, dataset_id):

    31.  
    32. # Initialize Domo client
    33. domo = Domo(DOMO_CLIENT_ID, DOMO_CLIENT_SECRET, host=DOMO_API_HOST)

    34. # Convert DataFrame to CSV for upload
    35. csv_data = df.to_csv(index=False)

    36. # Upload the CSV data to Domo
    37. domo.datasets.data_import(dataset_id, csv_data, content_type='text/csv')
    38. print(f'Data uploaded to Domo dataset {dataset_id} successfully.')


    39. # Main execution
    40. if __name__ == '__main__':

    41.  
    42. # Read and combine CSV files
    43. combined_df = read_csv_files(CSV_FOLDER_PATH)

    44. # Upload combined DataFrame to Domo
    45. upload_to_domo(combined_df, DATASET_ID)

    ** Was this post helpful? Click Agree or Like below. **
    ** Did this solve your problem? Accept it as a solution! **

  • That's great! thank you @ArborRose! 😀

Welcome!

It looks like you're new here. Members get access to exclusive content, events, rewards, and more. Sign in or register to get started.
Sign In