Veristorm’s vStorm Enterprise vs. Legacy ETL Solutions for Big Data IntegrationSubmitted by admin on Wed, 2015-12-30 04:29
The revolution of Big Data and analytics technologies are driving changes throughout the business intelligence stack, enabling the integration of unstructured data that either didn’t exist or wasn’t previously collected.
In the drive to become more customer-centric, organizations are seeking to improve data-driven decisions. This had created the need for self-service, on-demand access to data as opposed to legacy ETL software development projects.
The vStorm Enterprise software is a new solution that was created specifically to address data integration challenges in the modern Big Data ecosystem. It provides a secure, efficient process to serve mainframe and distributed data into big data platforms for self-service analytics and visualization. Data analysts choose vStorm Enterprise because it addresses three primary concerns when accessing enterprise data from mainframe servers: Speed, cost, and security.
With a design-time web interface, vStorm Enterprise provides point and click availability to browse, filter, migrate, and schedule integration of mainframe and distributed data sources. This interactive way of exploring data, without the need to write software, empowers data analysts to access the data they need without the traditional ETL software development process.
vStorm Enterprise has a very rich UI to create and manage migration of data, without the clutter of ETL solutions.
Data sources supported include: VSAM/QSAM, DB2 for z/OS, JDBC data sources, SMF/RMF data, IDMS, Datacom and mainframe SYSLOG/OPERLOG data.
For DB2 for z/OS data, vStorm Enterprise uses unload mechanisms that are four to five times faster than SQL methods. The unload methods have much smaller CPU footprints as well, saving on MIPS charges.
Through its no-programming, small CPU and no-storage footprint due to direct network streaming, vStorm Enterprise results in lower costs to migrate data. No SMP/E install is required for mainframe data sources. In most cases, binary data is moved to the Linux side to reduce CPU costs on the mainframe.
The end-to-end migration capability requires no programming. In contrast, the leading legacy ETL solution requires data preparation on the mainframe that adds to CPU and intermediate storage costs. This also means there are multiple copies of the data that must be managed.
The vStorm Enterprise software also provides discovery of DASD objects, including VSAM and QSAM, on the mainframe. Legacy ETL solutions don’t have this ability, forcing this discovery to be performed manually or through the expense and delay of creating and maintaining more software.
Is moving the data enough?
Moving VSAM or relational data is much more useful if you move the data and the metadata. The metadata describes the columns and fields and preserves the relationships between records. With automatic migration of metadata, vStorm Enterprise powers analytics and visualization tools to do more, without programming or manual intervention.
For example, moving a VSAM file into Hive makes the data available through HQL (a SQL-like language) since the Hive table will be created after the migration.
Another advantage of vStorm Enterprise is that it can insert data and metadata into HDFS or Hive directly, without using MapReduce or staging. This saves on storage costs and eliminates another potential security challenge.
Enterprise data stored on mainframe servers is generally very sensitive and subject to strict governance. The vStorm Enterprise multi-tenant GUI enhances security and governance by limiting user access to the data according to their defined RACF/ACF2 profiles. After the approved data is selected to be moved, the transfer is protected using SSL.
Consider this quick checklist when comparing vStorm Enterprise to legacy ETL software.