Hi there,
I am trying to rewrite a Redshift Dataflow into Magic ETL. I ran across a specific function that was available in Redshift that I dont think is possible in Magic ETL. I wanted to get the community's input and advice to see what you guys think.
This is the function found in Redshift:
LEAD(CASE WHEN t2."channel" NOT IN ('Operational Email','Direct') THEN COALESCE(t2."source",'blank') END, 1) IGNORE NULLS OVER (PARTITION BY t1."domo_id" ORDER BY t2."sessionid" DESC)
In Magic ETL, there is the Rank & Window tile that allows for the LEAD function, but it does not behave the same as the Redshift since the Redshift has the IGNORE NULL clause.
There are 20 cases of this LEAD IGNORE NULL function in the Redshift, so my first thought is that the method would be 20 filters + 20 Rank & Windows in Magic. I'm afraid that will affect performance for larger datasets.
Should I leave the Redshift Dataflow as Redshift?
What are some ways that you guys think I should handle this?
The current Redshift Dataflow runs in 4 hours. The purpose for moving into Magic, is the hope that it would reduce the run time significantly.
Appreciate any help!
Thank you in advance