Website Scraping

Lyndon Bye
Lyndon Bye Member
in Connectors

Hi Everyone

 

Has anyone ever used Domo and its connectors to web scrape a website? Jut wondering if any of the connectors have the ability

 

Thanks

 

LB

Comments

  • guitarhero23
    guitarhero23 Contributor

    I haven't experimented to see if a connector exists that does that but it sounds like something that's better suited for scripting out in a programming language and sending the results to Domo via the Domo SDK.

     

    Example: in Python use Selenium or Beautiful Soup to scrape the contents of a site and then format it as a CSV to upload to Domo. 

     

    Maybe other people have different use cases that might answer it better but I'm not sure any connector will do that. Do you have an example of a site you would be scraping?


    **Make sure to like any users posts that helped you and accept the ones who solved your issue.**
  • Lyndon Bye
    Lyndon Bye Member

    Hi

     

    Thanks for the response. Agreed Python is the way or Sceaming Frog, but a connector would be easier

     

    THx

  • Cartergan
    Cartergan Contributor

    I too am currently working on a project involving creating a card with info from scraped websites. I do not believe Domo has the capability to do this since it requires some scripting, but I do have some good resources/code I've been working on for basic scraping if you run into any issues! 

  • ahackett
    ahackett Member

    I'm looking into this, did anyone come up with a solution?

  • GrantSmith
    GrantSmith Coach

    @ahackett

    I'd recommend using a scripting language like Python (with the beautiful soup package). You could possibly utilize an ETL with some regular expressions but there's much more flexibility and power with a scripting language than you would have with Domo.

    **Was this post helpful? Click Agree or Like below**
    **Did this solve your problem? Accept it as a solution!**

Categories