Best Of

We absolutely hear this signal on MySQL 5.6. Here are a few thoughts on how we see things:

1. The expressive power of SQL is great. We know some of you (or others in your org) prefer SQL to MagicETL tiles. (Often we do, too!) We don’t see the two as direct competitors, but rather as different tools for different people or use cases.
2. To that end, we are working on a SQL tile for MagicETL. This brings the expressive power of SQL to MagicETL (beyond the existing formula tile). We anticipate starting a beta of this feature later this year. We are working on an “Open with MagicETL” option for MySQL and Redshift DataFlows to help you easily port any existing flows to MagicETL, as desired. (And also a “Save as…” option so you can easily create a copy for testing before migrating your production DataFlow.)
3. The SQL support being added to MagicETL will introduce non-equi joins to MagicETL as well as richer windowing function support. Common table expressions (CTEs) will also be available. These address some of the main areas where MagicETL is not a viable option today. A JSON tile is also coming soon to MagicETL. We will be evaluating the opportunity to expose the power of that tile within the SQL syntax, as well.
4. MagicETL has the benefit of running all of your transform steps in a streaming/parallel fashion as frequently as possible. This is one of the many reasons you see faster processing times with MagicETL. A full database like MySQL or Redshift carries additional overhead that isn’t necessary with MagicETL. So, if we can hit the mark with SQL in MagicETL, we feel like you can end up with the best of both worlds. Also, with MagicETL, the automatic optimizations within MagicETL will continue to work when using the SQL tile in MagicETL. That means current or future optimizations to reduce processing time will be available automatically without any additional effort on your part.
5. For any remaining use cases that cannot be solved in MagicETL, we feel that Adrenaline DataFlows may be your best next stop. That said, we know there may always be a spot where MySQL best suits your needs. In that case, we think exposing a modern MySQL engine within MagicETL will provide a powerful experience. While moving data into a MySQL instance for tasks that can be accomplished using the native MagicETL SQL tile will typically perform more slowly (and likely disable automatic optimizations), we’d like to make it easy for you to make that choice and seamlessly move your data into or out of MySQL to accomplish any processing steps that you’d prefer to run there.
6. In order to avoid breaking existing pipelines, if we were to provide a modern MySQL version in the current experience, it would likely be exposed as a version picker within the existing MySQL DataFlow UI. Our hope, though, is to simplify the experience by moving towards fewer engines and UIs to choose from for data transform needs.
7. On the subject of security, every MySQL DataFlow today gets its own MySQL instance running in a sandboxed environment without network access. We recognize that MySQL 5.6 has reached end of life, but have taken steps to keep your data pipelines secured and running without interruption.

A toggle next to "Try the new Magic ETL" would read "Documentation Mode" and would:

• Put an opaque film over the whole Magic ETL map
• Disallow changes to the tiles
• Enable the ability to draw and color-code sections of the ETL
• Be able to notate what each section is doing

An additional item in the Save menu would be "Save and Run. Update documentation." and would allow you to make any quick edits to the documentation pane needed before saving and running.