Migrating Cards to a New DataSet: Tips and advice

DataSquirrel

I'm researching what's involved in migrating many cards to a series of new DataSets and have been doing a bit of reading and experimenting. I already posted some information to an old thread on this, but it seems like a big enough subject for its own topic.

 

For background, my current task is to shift three DataSets to a new import method. Right now, we push out an Excel file, import it with Workbench and all is well. We went with this route as it's easy to set up and experiment with. Going forward, we've got too many different machines contributing data (and too much data) to continue this way. Instead, we're centralizing all of the data in Postgres and pulling it from there. All of that's fine.

 

When we started out, we didn't appreciate that when you great a DataSet you are locked into that delivery method forever. Our new and old DataSets are identical, it's only the connection method (Postgres pull instead of Workbench push) that changes.

 

Question: Is it really true that there is no way to change how the data is supplied? That's all we need to change. If there is a way to switch the method behind the scenes, that would solve all of our problems and the rest of my post is meaningless. But I don't think so. But I can hope!

 

Okay, so these three DataSets have a lot of cards, 73, 29, and 34. I don't want to migrate them and blow things up. I've tried this out in the past and once a card is broken, it can be impossible to fix. I don't know of any way to safely clone or back up Cards, so I'm approaching migrating our 136 cards....nervously.

 

Okay, now a bit on findings and question to match. Any and all comments, suggestions, best practices, tricks, magic incantations, etc. are most welcome. And, for a bit more background, I've already written a tool that uses the Domo APIs to pull all accessible information down about our DataSets, etc. I use this to automatically detect name and type problems, and it should help comparing user access and PDP rule settings too. Unfortunately, there is no way to find the cards associated with a DataSet. (Please vote for this feature Card API? List cards that use a DataSet?)

Findings and questions broken down by area below. Did I miss any areas? I didn't even think of Drill Paths or filters when I first sat down with this.

------------------------------
Column Names
------------------------------
These have to match exactly and case-sensitively. I don't get the case-sensitive part, but case-sensitive mismatches bring up the warning message.

 

I think that there's no problem with a DataSet having extra columns, but it can't be missing anything from the original.

 

------------------------------
Column Data Types
------------------------------
These have to match exactly. I've got this covered, so far as I can tell.

Good: Domo checks for column name and type name conflicts during migration, but it does not give you any details about what exactly is wrong. (Please vote for this feature Please add details to the warning dialog when moving a Card to a new DataSet)

 

------------------------------
Beast Modes
------------------------------
Beast Modes seem to be recreated atuomatically when you migrate. Nice! There's something in the docs about this behavior, but I don't know what the nuances are.

 

Any warnings or words of wisdom?

 

Idea: Speaking of Beast Modes, they're hard to detect. They're invisible to the API (Please vote for this here API: Please expose Beast Modes), and there's no UI in Domo to see them in one place. Unless I've missed something, there's also no way to distinguish cards or DataSets with calculated columns.

 

Idea: I'd like to be able to migrate calculated fields to normal fields when changing DataSets. Please vote for this feature here: Please allow Beast Modes to be mapped to full columns when changing a card's DataSet.

------------------------------
Drill Paths
------------------------------
It sounds like filters and Drill Paths are not migrated. There's a long-standing thread on this subject. It's marked as "solved", but it doesn't sound like there's a working solution.

 

The Domo APIs include no trace of Drill Path details. The child chart don't appear, there's nothing about them. Same wth Beast Modes, they're invisible in the API. You seem to have to go in and review every single card by hand. It would be great to have simpler UI tools for finding, managing, and editing calculated columns. For the minute, I'd be happy to have an API to call. Please vote for this idea here: Please add Drill Path details to the APIs.

 

------------------------------
Filters
------------------------------
I read that they break. I'm not sure what that means. I'm building up a checklist of steps to take to pre-flight each card and to then verify the card after the move. What would be suggest I put down for filters?

 

------------------------------
User Access
------------------------------
Haven't looked closely. I think we'll end up having to do it all by hand.

 

------------------------------
PDP Rules
------------------------------
Haven't looked closely. I think we'll end up having to do it all by hand.

 

Thanks!

Jarvis

Hi DataSquirrel,

 

Migrating Cards to a New Dataset is very simple and can be completed in a number of ways. There are some guidelines to follow when migrating cards. All current cards that are attached to a dataset could break if the column names are not identical. Column name changes that aren't identical can affect BeastModes, Filters, Color rules. Cards can be easily fixed by just adding the renamed column in replace of the old one. 

 

Here's a little more information if you need it:

 

http://knowledge.domo.com?cid=switchingcards

 

Have a good one!

DataSquirrel

Thanks for the link! I should have made it clear that I've already been through the documentation and tried searching through past Dojo theads.

 

As I've probably said somewhere, I've written a tool to pull the DataSet definitions via the API so that I can then have a script compare the new and old DataSets. This lets me pre-flight the new DataSet for name and type conflicts. So I should be 100% okay on names going forward.

 

Do you have any experience on the Drill Path and Filter prolblems that I've seen mentioned in the past?

 

Thanks

user14313

Upvoting as I think there are some unresolved issues / suggestions in here.

 

For example, instead of just saying "Warning, fields don't match", why not let us map non-matching fields to new ones?

 

E.g. I just migrated 300 cards over from one dataset to another, in which we had renamed a field (same data) in response to user feedback over clarity. It doesn't seem hard to say "hey old_column no longer exists in the new dataset, so replace all instances of old_column with new_column".

 

Ideally this would go into Beast Modes and make the change there, too.