AI is helping to identify skills gaps and future jobs. An expert explains how

“Believe it or not, if you take a forensic accountant and teach them some cyber, they can become a cybersecurity expert,” says Erik Brynjolfsson, Professor at the Stanford Institute for Human-Centred AI.

“There's a lot of skill overlap in those two professions, even though you wouldn't have thought of it without looking at the underlying data.”

Brynjolfsson and his colleagues at the Stanford Digital Economy Lab are using AI to sift through millions of job postings and identify skills gaps and skill “adjacencies” that can help companies retrain team members to meet the demands of the future of work.

The World Economic Forum’s Future of Jobs Report 2023 says that 60% of workers will require additional training by 2027, with the biggest priority being analytical thinking.

Technology-related roles dominate the report’s list of fastest-growing jobs, with AI and Machine Learning Specialists coming top. And LinkedIn research conducted for the Future of Jobs Report 2023 identifies the 100 “Jobs on the Rise” – those that have grown “fastest, consistently and globally” over the past four years – with technology and IT-related roles making up 16 of these.

Skills and AI will be among the hot topics up for discussion at the World Economic Forum’s Growth Summit on 2-3 May in Geneva.

AI is becoming more important to all jobs, as Brynjolfsson knows, and his Work2Vec project is helping companies retrain employees for emerging roles by identifying similarities between existing and future skillsets. Below are excerpts from an interview with him about the impact of AI on the world of work:

We're in the early days of a real revolution in terms of how AI is affecting work and employment. The technology's growing exponentially, but our skills, our organizations and our institutions are not adapting nearly as quickly.

About 10 years ago, I started looking at this more closely. One project involved coming up with a set of criteria for the types of tasks that are suitable for machine learning. It’s clear that while AI is very powerful at human level or even superhuman level for many tasks, there are many other things where humans continue to have a big advantage, and that's going to continue to be true for for quite some time.

I don't predict any sort of a job apocalypse or mass unemployment, but I do think there's going to be massive restructuring of work as AI starts doing more tasks that previously were only done by humans.

We came up with a set of criteria for which tasks were most suitable for machine learning and which ones were not. We wrote that up in an article in Science magazine, and then we applied that rubric to about 18,000 tasks in the US economy from a dataset called O*NET.

Most jobs have many distinct tasks in them. For instance, if you are a radiologist, you don't just look at medical images, you also consult with patients and you coordinate care with other doctors and so forth. Radiologists do about 26 distinct tasks, and what we found was that machine learning could affect many of those tasks. But in none of the 950 occupations we looked at did machine learning turn the table and do all of the different tasks.

In each case, there were parts of the job that were best done by humans and others where machines could help out. For instance, in the case of radiologists, machine learning was very good at looking at medical images and increasingly good at diagnosing different pathologies. However, it's not good at consoling patients or talking to them after a diagnosis, or coordinating care with other doctors. So that was what we found for bus drivers, for economists, for nurses, cashiers – every job. There were parts of them that machine learning could do more of and others they could do less of. So going forward, I think we'll continue to see some significant restructuring of work, but not mass unemployment due to AI.

Exactly. A lot of the tasks that were not well done by machines included some of the softer skills – relationships, emotional intelligence, also creativity, coming up with new questions to work on, new kinds of innovations that scientists or creative people or artists or entrepreneurs work on. So there are a broad scope of things where humans continue to have an advantage.

One of the cool things about working in this field is not only that we're studying how AI's changing work, but we can use AI and particularly machine learning tools to understand work itself. So we have a big project, we call it Work2Vec, that takes jobs and converts them mathematically into vectors and relates how the jobs are similar to each other or different to each other.

My colleagues at the Stanford Digital Economy Lab and I are scanning through hundreds of millions of job postings. Each of those job postings describes not just the title of the job, but the tasks that they do, the skills that are needed, so you can see how similar jobs are, how different they are, using natural language processing techniques.

By sifting through these jobs, we're beginning to see patterns and understand how work is changing over time, how the skills are different in, say, Miami versus Denver, or what happens if you add a new skill like Python expertise to a job that previously didn't involve that. What does it mean for the compensation? These are the kinds of analyses you can do with Work2Vec. It's been wonderful to have not just a huge amount of data, but also some very powerful machine learning tools to really get at what's happening with the workforce.

We have data going back for about a decade and we can see over time that there's been a shift. Obviously, one of the most important shifts is that AI has become dramatically more important in many jobs, not just in core engineering jobs, but it's also beginning to creep into parts of medicine, in business and law and elsewhere.

We're also beginning to see changes in the kinds of skills that are needed, where there's a lot more management and leadership skills that are becoming important, and relationship skills. Some of those are probably also due to the changes in the underlying technology that we're seeing. We’ve seen that low-wage workers, those with high school education or less, have not been keeping up in terms of wages with more educated workers. In the past couple of years, that's reversed and low-wage workers, at least in the United States, have seen bigger wage increases, bigger compensation increases than those in the middle or the top of the spectrum.

These are the kinds of analyses that we can do for the first time. It's like a new kind of a tool, a microscope that allows us to peer into what's happening in the workforce in a way we never could have before. So it's really interesting times to be a researcher like me trying to understand the future of work.

One of the great things you can do with these tools is you can now give advice to hiring managers, to HR managers, to CEOs. I have a company called Workhelix that commercializes some of these tools and allows companies to peer into not only their own workforce, but the whole marketplace. That way they can make better decisions about compensation. They can understand which of their existing employees might be in danger of being hired away because the marketplace has increased the value of their skills. They can understand better how to recruit the right kinds of people.

They can also skate to where the puck is going to be. And by that I mean invest in the skills that are going to be needed in the next two to three to five years instead of just what's needed today, because we have some visibility into the future by looking at the types of jobs that machine learning is able to help with versus the ones that it's not affecting as much. The companies that have that kind of forward-vision radar for talent are able to succeed more than the companies that are flying blind.

I think companies do need to be talent creators. It's not enough to simply try to hire them from the outside. In many cases, these companies already have a lot of the talent they need inside the organization, but they don't know it. Or perhaps there's sort of a near miss, we call them skill adjacencies, where they don't have exactly the person that they need, but there's somebody very similar. Work2Vec will describe how those skills overlap and maybe by adding a little bit of additional training, online courses or other tools, maybe in as little as 60 to 90 days, you can convert someone who's not a perfect fit to somebody who's exactly what you need.

My company Workhelix is working with Amazon, which needs to hire a lot of people for their data centres. They need to hire more machine learning experts. They need to hire cybersecurity people. They need to hire people who can pull fibre. All three of those categories are very difficult to find. However, it turns out that if you take a data scientist and you teach them some Python and other skills, they can do a lot of the machine learning work.

And you can take an electrician who's been working with copper and teach them how to work with fibre. So in each case you can find somebody who you have a lot more supply of and, with a little bit of training, convert them to that scarcity that companies like Amazon need.

Whole new professions, whole new types of occupations, are emerging. Sometimes you see that in new titles. The concept of prompt engineer is something we never would have heard of before. A prompt engineer is somebody who works with new tools like Chat GPT and writes prompts that are useful in drawing out the knowledge and the insights from that tool so they might enable it to write a better piece of advertising copy – or a better job posting for that matter. Sometimes it takes a little bit of ingenuity to write the right prompt.

Increasingly in the coming decade, it's going to be important not simply to have good answers, but also to ask the right questions. And that's something that humans are especially good at. So we're able to look not just at the job titles, but at the content of the jobs underneath and understand where it is that people are adding value by asking the right questions.

The big challenge for the next decade is not going to be a job quantity problem, it's going to be a job quality problem. That is to say that right now we have almost record-low unemployment in the United States and many other countries. So there are a lot of jobs out there, but they don't always pay well, they don't always have good work-life balance. They aren't always the kind of jobs that people enjoy doing.

One of the challenges for our society going forward is to work to have better quality jobs. I don't think it’s inevitable that we'll do it right or wrong. It's something that we need to work hard on and consciously – CEOs, union leaders, policy-makers, technicians and technologists all need to work on thinking about how we can improve the quality of jobs and create work that is not just productive and valuable to customers and stockholders, but also fulfilling for the people doing the work. If we do that, I think we're going to have higher wages and better work-life balance.

As I've been working more and more with AI technology, I'm impressed by its power, but I'm also concerned that it could be used in ways that are destructive, that could undermine the shared prosperity that we’d all like to see. In particular, there's a concept in AI called the Turing Test, which is that the test of whether or not AI's intelligent is if it can perfectly mimic a human, so you can't tell which answers are given by the human and which are given by the machine. I think it's a very evocative vision, and I can see how it's inspiring. As an economist, it also turns out to be exactly the wrong vision in terms of creating value.

If you have a machine that does exactly the same things that the human does, then you're going to drive down wages and drive down the value of human labour. But there's another path, and that is to make machines that augment humans, that allow humans to do something new that they've never done before, to make better-quality or more creative or new products and services. AI that does that kind of innovation is more likely to raise wages and create widely shared prosperity.

So my call to arms for technologists is think more about how we can make AI that augments humans rather than simply automating labour. The same mission is what entrepreneurs and managers should be thinking of – come up with business models that build on humans and enhance the value of human labour and keep humans in the loop, or better yet, humans in charge. Policy-makers need to think about whether they are setting government policy in a way that encourages this kind of shared prosperity or whether they are destroying it.

For example, tax policy in the United States and most other advanced countries currently favours capital over labour. There are significantly higher taxes on labour than there are on capital, and that basically steers innovation away from helping labour and towards trying to concentrate benefits to capital owners. I don't think that's a sensible policy. At a minimum we should have a level playing field so that people could make whichever decisions create the most value, and not have policy-makers put the thumb on the scale steering value over to capital owners. If we do these three things for technologists, for managers, entrepreneurs and for policy-makers, then we're going to be able to keep humans in the loop, keep humans in charge, and have the kind of AI that benefits us all broadly.

I think a lot of people are concerned about the power of the most recent breakthroughs in generative AI or foundation models like ChatGPT, but also the ones for images like DALL-E and Stable Diffusion. These are strikingly powerful technologies and I share the amazement that many laypeople and technologists have at how effective these tools have become – far beyond what a lot of people expected.

The concern is that they're going to replace jobs. But I think there's another approach, which I talked about in The Turing Trap article, which is using them to enhance creativity and enhance jobs. The next decade could be a decade of human flourishing where we create more and better art, more and better writing, more and better creativity on lots of different dimensions, even invention and scientific discovery by using these generative AI tools to help us do new things that we couldn't have done before.

Rather than trying to stop the tools and, for instance, telling students not to use them, in my class I’m telling students to embrace them – but I expect their quality of work to be that much better now they have the help of these tools. Ultimately, by the end of the semester, I'm expecting the students to turn in assignments that are substantially more creative and interesting than the ones last year’s students or previous generations of students could have created.

With Work2Vec, we're able to hone in and understand much better where some of the gaps are in the workforce, where there is need for new types of skills, where the workforce already has a lot of the skills that we're looking for and how we can reallocate them. Going forward, I think we're going to be much more quantitative and scientific about allocating the most valuable asset on the planet, and that asset is human capital.

In the United States alone, there is about $220 trillion worth of human capital, but it's really badly managed because it's really badly measured. So we're working on understanding better what are the real skills in the workforce so that people can make better decisions. Ultimately it's going to be beneficial not just for the companies and for society, but for the individuals themselves. Most people are happier when they're doing a job that really fulfils their potential rather than when they're doing something that isn't exactly well suited to their skills and capabilities.