Defect? New Source Columns missing in DataFusion Result until after rebuild

Kurbz

Hi Folks,

you might need to replicate and confirm the following before marking it as a defect.

 

Take two source ds, sA & sB.

sA has 3 columns, a,b,c.

sB has 3 columns, x,y,z.

In a DataFusion sA is joined to sB to create R, the result ds.

After the R is created, it contains a,b,c,x,y,z.

 

Next i want to extend sA, i've added a new column in sA call d.

sA now contains a,b,c,d.

 

After R is rebuilt, in the preview of the ds, we see a,b,c,d,x,y,z, but d is greyed out.

d also dose not appear on the cards.

 

To resolve this (unexpected behaviour), we need to edit our DataFusion R.

Graphically remove the to source ds, sA and sB, which in turn removes the joins.

Then add back sA, sB, and add the same join again.

Now when previewed, d is no longer greyed.

Additionally d is abailable in the cards.

 

Rgds,

PodiumMason

Hey @Kurbz I can confirm that this is a bug that I've seen as well. If the schema is changed at all in any of the input datasets for the joins, those columns won't be made available without recreating the joins. (Also column names can't be changed after creating the fusion initially, not sure if this is a bug or more of an enhancement need.)

Kurbz

Thanks for the comment and confirmation @PodiumMason

 

I regard it as unexpected behaviour, just my opinion, hence i'd classify it as a bug.

If the source is changed, ie schema, it should push into relevant downstream items and refresh, can imagine this could get complex.

IDEA:

would be nice to see the push from sA down stream, with an alert stating "updating this datasource will effect the following items, 1,2,3,... We recomend you review the impacted items to ensure your items preform as you require"...

rgds,