Multiple singe day events grouped as single consecutive day events

jvanis

I am trying to figure out a way, in Magic ETL, to take multiple single day events and group them when there is break in consecutive days.

Example, if the results show the 1st, 2nd, 3rd, 5th, 6th, 9th, 10th and 11th, I need three results, not 8. I want to see begin and end dates of 1st-3rd, 5th-6th, and 9th-11th.

Any thoughts?

ColemenWilson

Walked through how to do it here: https://www.loom.com/share/5abc1ddf7dfa4e39a39472d274240f64?sid=337d1499-14b3-4766-b18f-18d509408ba3

This is modified from what I previously suggested and doesn't require lag and lead functions. Big thanks to @AustinHarrison for the idea of a more simple way to do this using row numbers!

ColemenWilson

Classic gap and island analysis. Here is an article introducing the topic and how to accomplish what you are trying to do: https://www.red-gate.com/simple-talk/databases/sql-server/t-sql-programming-sql-server/introduction-to-gaps-and-islands-analysis/

An alternative way in MagicETL would be to use a lag function and have some logic that if the previous value is not the current value minus one then that current value must be a new start to an island (the 5th in your example). Similarly you could do a lead function and anytime the future value isn't the current value plus one then it is the start of an island (the 6th in your example).

I think the linked article is the cleanest way to do this in a MySQL dataflow.

jvanis

Thank you @ColemenWilson, but could you expand on the MagicETL method? I've been trying to use the Rank & Window with the previous date (lag), but not able to get to the grouping I'm hoping for. I need to somehow carry a constant down the rows until it sees that the datediff is >1, then a new constant until >1, etc.

ColemenWilson

Walked through how to do it here: https://www.loom.com/share/5abc1ddf7dfa4e39a39472d274240f64?sid=337d1499-14b3-4766-b18f-18d509408ba3

This is modified from what I previously suggested and doesn't require lag and lead functions. Big thanks to @AustinHarrison for the idea of a more simple way to do this using row numbers!