Identifying what data came from which connector on a card.

Parkerains · 2024-06-03T15:24:42+00:00

There was an error rendering this rich post.

Hello all, and thank you for any help. I have my magic ETL flow built out so a report is pulling information from two different connectors. There does not seem to be a batch ID or file path as options for me to select, which I thought would be the method to identify which system they pull from. Does anyone know how to filter on a card by connector? Thanks any help would be appreciated.

Find more posts tagged with

Connectors

Cards

Magic ETL

Data Sources

Card Builder

Accepted answers

Jonathan53891

Hello @Parkerains, this will depend on how you have the Magic ETL dataflow set up. One solution to this would be to use the "Add Constants" ETL tile in your dataflow if you are appending your data sources together. Once you have finished creating the dataflow logic for each of your respective data sources, you can place a constant tile before the sources get appended together.

For this, you would need to use the same column name for each ETL tile (I use "Source" as the column name for this example). Then, you can input the unique name of the data sources they came from (i.e., Source = Facebook, Source = Instagram Business, etc). Since they are all appended to the same column, you can use this column in a filter on your dashboard to toggle between the various data sources in addition to having a clear distinction for where the data came from.

Please also note that the batch IDs for the cloud-based connectors provide a reference for how many times the dataset has updated. This can be utilized in a variety of ways, but the _BATCH_ID_ column does not provide any descriptive information about the connector itself.

DEV - Social Media Consolidated (EXAMPLE).png

DEV - Social Media Consolidated Add Constants Tile (EXAMPLE).png

ColemenWilson

@Parkerains I re-read your original question and if you are trying to apply a filter for a certain input then the "Add Constants" tile approach is definitely the way to go. My method is for identifying which field came from which source, not which rows.

All comments

ColemenWilson

There are a few options for this, all of which we employ at my company.
1. Use field naming conventions. Example: `Salesforce.Account.ID`
2. Use field schema descriptions or tags

Would these work for you? If so I could share more info on them if that's helpful.

Jonathan53891

DEV - Social Media Consolidated (EXAMPLE).png

DEV - Social Media Consolidated Add Constants Tile (EXAMPLE).png

Parkerains

Thanks @Jonathan53891 and @ColemenWilson - won’t get to try these until tomorrow when I am back at my desk. But thank you for the quick response and helpful info. I will come back and post to this tomorrow with an update after I have had a chance to try both of these methods. Thanks again!

ColemenWilson