How to Create a Dynamic Pivot in Dataflow ?

Options
VictorReyes
VictorReyes Contributor
in Magic ETL

Hi,

 

I have a dataset that I would like to pivot on a certain column value, development month, that introuduces new unique values each time the data is refreshed.

 

The issue that I am having is that this field can change any time the dataset is updated. This is a rather large data set (2 million rows) and I'd like to avoid having to go in every month and figure out if there were new values added. 

 

Also, the pivot columns should be a running total meaning that column 3 should be the sum of all values 1, 2, and 3. Column 6 should be the sum of all values from columns 1, 2, ,3 ,4 ,5 ,6. I would like this to be dynamic so that if I load the dataset today and there is new development month, say 84, then this is added as a column and the values from 1-84 summed as the value. 

 

Any ideas how to do this in DOMO? I know how to do this is MSSQL but not in MySQL or Magic ETL. 

 

Example:

 

 

Best Answer

Answers

  • RGranada
    RGranada Contributor
    Hi,

    Have you considered trying a Redshift dataflow, if you can do it in mssql it should work in Redshift with minor changes...

    Hope this helps.

    Ricardo Granada 

    MajorDomo@Lusiaves

    **If the post solves your problem, mark it by clicking on "Accept as Solution"
    **You can say "Thank you" by clicking the thumbs up in the post that helped you.
  • VictorReyes
    VictorReyes Contributor

    Hi,

     

    I don't see Redshift listed under dataflows. Do you know a link to an article that will point me in the right direction?

  • RGranada
    RGranada Contributor

    Hi,

     

    In the data center click after clicking new and selecting SQL, you should have the following choices :

     

    DOMO_SQL_DATAFLOWS.png

    This link should help:

    http://knowledge.domo.com?cid=datacenternew

     

    Don't hesitate to ask if you have more questions. 

     

     

     

    Ricardo Granada 

    MajorDomo@Lusiaves

    **If the post solves your problem, mark it by clicking on "Accept as Solution"
    **You can say "Thank you" by clicking the thumbs up in the post that helped you.
  • VictorReyes
    VictorReyes Contributor

    Hi,

     

    I don't have that option and we've opted into the beta program. My SQL dataflow defaults to MySQL. 

  • VictorReyes
    VictorReyes Contributor

    This is what I have: 

     

    2017-08-16_17-33-12.png

  • RGranada
    RGranada Contributor

    Have you selected "SQL"?

     

    After that, you should have what I showed you.

     

     

     

    Ricardo Granada 

    MajorDomo@Lusiaves

    **If the post solves your problem, mark it by clicking on "Accept as Solution"
    **You can say "Thank you" by clicking the thumbs up in the post that helped you.
  • VictorReyes
    VictorReyes Contributor

    Hi,

     

     

    I called DOMO and they have to enable Redshift in the instance, apparently this is not a default option. 

     

    Wouldn't have known that there is a 2nd option otherwise. A bit frustrating to have to hunt down all of the different features available. Don't understand why that wouldn't be enabled from the start. 

  • AS
    AS Coach

    An insider's view on that question:

    When customers get started with Domo, there's generally a discussion between the customer and Domo about how much data will be transformed regularly.  The rule of thumb is if the largest datasets are less than 5 million rows or so, MySQL is the recommendation.  More people are familiar with MySQL anyway, so that's usually not a concern.

    If datasets tend to be larger, the recommendation is Redshift.  But Domo likes to keep things as simple as possible for the business user, so they don't automatically unlock all features.  I think it's actually more rare, to @RGranada's point, that customers have or would want access to both.  It's a bit of a drag to slip in and out of different variants.

     

    For some like you, there are criteria aside from data size that should inform the MySQL vs Redshift discussion, and there are instances where one can't do what the other can.  So very few might really want access to both.  All that said, I think for Domo it's really a question of capability vs simplicity, and their scale tends toward simplicity.  Domo has all sorts of tools in the nonpublic toolchest.

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • VictorReyes
    VictorReyes Contributor

    Hi,

     

    Ah thank you AS This helps put everything into perspective.

     

    Our organization's BI is lacking and incredibly slow so we're starting to create our own marts and views in DOMO and then pass those back to our internal teams to be re-created internally.

     

    We'd prefer to have every single tool that DOMO has to offer. Would you be able to point me in the right direction si I can get everything in their non-public toolset enabled? Often times we're under the gun to deliver so I'd prefer to be proactive and have everything at our disposal.  

  • RGranada
    RGranada Contributor

    Great view on the subject @AS . I agree with you entirely, furthermore, DOMO still treats RedShift Dataflows as Beta.

     

    Here on our platform 98% of our dataflows are redshift and they were enabled since inception, and it's also my personal choice over Mysql, so I do tend to get a little biased.

     

    @VictorReye sorry for not pointing that out,  you can contact support or your success manager in order to get redshift dataflows enabled.

     

    Best regards,

     

     

     

     

    Ricardo Granada 

    MajorDomo@Lusiaves

    **If the post solves your problem, mark it by clicking on "Accept as Solution"
    **You can say "Thank you" by clicking the thumbs up in the post that helped you.
  • AS
    AS Coach

    Much of what is nonpublic is still considered beta or even alpha.  

    There are hundreds of API connectors unseen by the public eye.  Admin tools.  Excel plugins for PDPs, etc. And I'm sure tons of stuff I don't know about.  They have a vibrant skunkworks operation.  That's how the data warehouse and data lineage features started, as well as the Powerpoint and Excel plugin in tool downloads.

    A great way to find out about it is to participate in the beta program.  And to become good friends with your account executive and success managers.  I got a jumpstart by being coworkers with mine first, but that's not the only way ?

    Aaron
    MajorDomo @ Merit Medical

    **Say "Thanks" by clicking the heart in the post that helped you.
    **Please mark the post that solves your problem by clicking on "Accept as Solution"
  • VictorReyes
    VictorReyes Contributor

    Hi,

     

    @ASThat is good to know. We are enrolled in the beta program. We had the new charts and analyzer in advance.

     

    I'll touch base with our relationship manager. 

     

     

    Thank you

  • [Deleted User]
    [Deleted User] Member

    @VictorReyes,

     

    Please mark one of the replies as a solution if this addressed your question.

    Thanks!

     

  • bankuda
    bankuda Member

    Hi there,

     

    Has anyone been successful in finding a solution using MySQL or Magic ETL?

     

    Thank you,

     

    Brandon

  • Unknown
    Unknown
    Answer ✓

    Hi, @bankuda,

     

    Stumbled upon this thread today - I hope you already found a solution, but if not, check out this reference in the Knowledge Base: https://knowledge.domo.com?cid=dynamicreversepivot

     

    That article explains how to pivot; a similar approach can be used to unvpivot. 

Categories