How to Overcome the Big Data Skills Shortage

How to Overcome the Big Data Skills Shortage

By its very nature, Big Data is tough to deal with. Typically, there are numerous data sets, many of which are incomplete or contain inaccurate material. It’s a complicated process to quantify all this information and create meaningful metadata that can be easily searched and analyzed.

On top of this, there's a serious supply-and-demand problem in data science. There is a growing demand for people who can manage and manipulate Big Data, but not enough skilled data scientists.

How Bad Is It?

In late 2013, when Boston-based business intelligence and analytics software company Lavastorm Analytics surveyed 425 analytics professionals, C-suite executives, managers and business analysts, 83% said analytics was critical to their businesses' success – but 65% said they were unable to hire enough workers with data skills.

A number of respondents went further, saying that the skills gap had prevented their organization from becoming more data-driven in its decision-making, and caused executives to question the importance of analytics findings.

Government Circles

At administrative level, the overriding emphasis on storing and managing data means local authorities have little time or resources to invest in data analytics, let alone in developing the analytics skills of their IT personnel.

Training targeted at bringing people up-to-date with the latest Big Data techniques is one potential solution. But with tight budgets, authorities need to justify the investment by presenting a compelling business case as to why this will bring greater value than alternative spending plans.

This year, a government-funded Administrative Data Research Network (ADRN), has been established across the UK. The initiative brings together universities, government, national statistics authorities, investors and researchers, to maximize the benefits offered by reusing UK administrative data for research intended to benefit the public.

Too often, government organizations are held back from taking advantage of Big Data opportunities by public concerns over data security. And local authorities sometimes make expensive mistakes by purchasing the wrong technology, or systems that aren't capable of delivering the kind of analytics capability they were expecting.

In helping to address the public sector skills gap, ease of use for data storage and analytic systems and solutions is key. There has been a push to provide ease of access mechanisms via the Web, with portals onto Big Data, and online delivery of analytical servers.

Corporate Views

Many businesses lack the skills to organize and analyze data in meaningful ways. The shortage is particularly pressing for marketing firms, which have not traditionally employed many people with data skills. They are now facing intense competition to become more data-driven.

Organizations are looking to compensate for this skills shortage by developing automation and visualization technologies. Automation and visualization analyses can make data easier to use, even among individuals working in different functions. Visualization also lets users look at data in a more accessible way - with graphics, rather than columns of numbers. Though implementing the technology may be challenging, it can prove less costly in the long term.

The Banking Sector

92 of the top 100 banks rely on System z for their core functions, according to a recent American Banker article. So the mainframe skills shortage is hitting financial services pretty hard. In some cases, institutions may only have one or two employees who fully understand how the technology works.

As mainframe programmers hit retirement age, banks, with the help of some IT industry initiatives, are working to foster and recruit young talent. It's a challenge, as both banks and the mainframe struggle with an image problem. The good news is that recent graduates with mainframe skills are now in high demand.

Self-Service Options

One way around the skills shortage problem is by implementing self-service analytics tools. These may require significant technical expertise to set up, but once running, can be used by business people who lack data or technical skills.

Several emerging self-service analytics vendors – like Lavastorm, Tableau, Qliktech and Logi Analytics – are producing software intended to ingest data from a range of sources and graphically organize information, without the user having to write code-based queries.

Rather than putting in a request to a help desk and waiting for an analyst to address it, business users can do their own analyses, which reduces the time needed to gain valuable insights. Marketing professionals can analyze trends and operations in their own time.

Of course, self-service tools can't do everything that more robust analytics systems can. For complex operations, data scientists must still be called upon for their expertise.

The Road Ahead

Within the public sector, we are seeing a growing number of projects that successfully combine high-performance computing (HPC) and Big Data, many of which deliver cloud interfaces that provide ease of access for the non-expert.

At enterprise level, SHARE’s zNextGen community offers peer-to-peer learning, as well as technical skills training for young mainframe professionals. Members can take advantage of online courses from Marist College and Interskill to enhance their skills. zNextGen’s mentorship program pairs emerging IT professionals with mainframe veterans, so they can learn how to carve out a successful career.

Enhanced recruitment techniques, using psychometric assessments such as cognitive testing, emotional intelligence testing and motivational assessment are now being promoted to increase the chances of landing the best Big Data talent.