Calculating Distance In Magic ETL

RyanIllum

I have around 400 locations with different coordinates. I need to find if the distance between any the locations is above 1 mile. Should I use ETL or another tool?

GrantSmith

https://www.domo.com/blog/improving-supply-chain-logistics-with-geographic-proximity-matching/

@MichelleH has written up a great article on the community blog you can reference on how she did geospatial distance calculations using a Redshift dataflow.

MarkSnodgrass

Here is a video walkthrough of how to do it:

https://youtu.be/xzi-fhEmN-Y

ColemenWilson

@MarkSnodgrass laid it out exactly how I did it. Keep in mind that your data can grow rapidly. For example, right now you have 400 rows, 400 x 400 = 160,000. Bumping that up to 500 takes your total rows to 250,000. So be cautious when using this type of self join.

GrantSmith

https://www.domo.com/blog/improving-supply-chain-logistics-with-geographic-proximity-matching/

@MichelleH has written up a great article on the community blog you can reference on how she did geospatial distance calculations using a Redshift dataflow.

ColemenWilson

Agreed with the above. If you want to do it in Magic ETL and just need to see any distances greater than 1 mile you could do this:

1. Join all 400 rows to each of the 400 rows (note this would be 160,000 rows of data)
2. Compare each location to each other location using the DISTANCE() function
   1. You'll need the coordinates of each location to do this.
   2. You'll need to convert from KM to Miles by mulitplying the result by 0.621371
3. Use a filter tile or filter in a card to only show those where distance > 1 mile

MarkSnodgrass

Here is a video walkthrough of how to do it:

https://youtu.be/xzi-fhEmN-Y

RyanIllum

@ColemenWilson how do I compare each location for every location without having to write a formula for every location

MarkSnodgrass

You will want to change your join since you are comparing it to the same data set. Here is how I typically do it to get a row for every combination.

Use the add constants tile to create a column called JoinKey and set it to a value of 1. Do that twice. Connect them to the join tile and join on the JoinKey with an inner join. Rename columns so that you have your first and second coordinates. You will then have a row for each combination and then you can use your distance function.

ColemenWilson

@MarkSnodgrass laid it out exactly how I did it. Keep in mind that your data can grow rapidly. For example, right now you have 400 rows, 400 x 400 = 160,000. Bumping that up to 500 takes your total rows to 250,000. So be cautious when using this type of self join.

RyanIllum

Thank you guys I was able to figure it out.