Can AI governance drive innovation?

Nigel Cassidy (NC): Ensuring AI is used ethically and accountably and getting better results into the bargain. How good AI governance is an enabler, not a blocker. I'm Nigel Cassidy and this is the CIPD podcast.

Now a century ago, the writer GK Chesterton may not have dreamt of AI, let alone how to govern it. But he did come up with a belter. Art needs a frame. He also said, don't take a fence down until you know why it was put up.

We get it. Constraints shouldn't hold back creativity. They're essential to doing great work. Yet somehow AI is allowed to jump the fences. Careless use undermines trust, can cause reputational damage and leave you facing litigation. Non-human labour deployed unaccountably spells trouble. Data breaches, IP infringement, discrimination by algorithm, factual hallucinations, you name it. So what role should HR play in establishing AI guardrails? And how can you show that governance isn't a brake on innovation, but essential for sustainable success?

Ray Eitel-Porter is an acclaimed advisor on navigating AI risk. He works with global banks, retailers and health providers alongside academic work in Cambridge, Oxford and beyond. He's a best-selling, indeed award-winning author and led the design of Accenture's internal AI compliance program.

Hi, Ray.

Ray Eitel-Porter: Hello.

NC: Sharon Hill is a director of people with first-hand experience growing AI capability at her organisation, Pay.UK, which moves digital money between bank accounts. She's also part of the Innovate UK Bridge AI programme, speeding research and adoption. Hello, Sharon.

Sharon Hill (SH): Hello.

NC: And Helen Dutter is a lawyer who's helped shape the CIPD's stand on some of this. She advises boards and individuals on governance, compliance and risk. She's a senior leader at DAX, a non-profit representing the rights of 180,000 artists worldwide. Hi, Helen.

Helen Dutta (HD): Hello, good to be here.

NC: So, Ray, I mean, on the face of it, I mean, this is about governing humans to ensure they use AI responsibly. I mean, that sounds simple enough, but I mean, it is more complicated, isn't it? Because now machines also do some of the work, so they need to be governed too.

RE-P: Absolutely. That I think is going to be the real next frontier, if you like, is governing AI agents, which are the talk of the town right now. These agents are much more powerful in what they allow us to do with AI because rather than simply returning recommended text or an image, they can actually execute things. They can actually go out and buy a ticket for you. They can go and make a transaction of any different type if allowed. And so putting in place real proper controls to govern how these agents behave is something that many people are working on at the moment and is frankly essential for any organisation.

NC: So Helen, I mean, we can look at what form governance might take in a minute and why we might need it. But I just want to ask you first, in your experience, do we actually need better AI governance now? I mean, has it fallen behind adoption and does that matter?

HD: Yeah, I mean, I think the aim of effective AI governance is to really ensure that AI is used responsibly and that it operates in the way we intend. I think it's very different to other tech, which we kind of, we can think about in adoption, we can think about introducing a new technology and perhaps knowing what is going to happen, how to train your staff and what guardrails to put in place.

I think AI is different because it's changing all of the time. As Ray has said, we're really looking now at agents and the governance around agents. And I think something about governance is that it has to be fluid. It can't be static. We have to engage with the workforce. We have to look at how AI is being used on an ongoing basis and adapt those guardrails and adapt those governance frameworks as we go. So it's very much a consultative process.

NC: And has the governance run behind what the AI can do?

HD: I think it depends in terms of different organisations. I think when we talk about governance, we have to kind of go out and think, well, what aspects are we looking at?

I think the main aspects are having clear guardrails. So how do we look at and define the legal and the ethical boundaries of AI use? We need to look at people and accountability. So establishing who participates in decision making around AI and who is accountable for that. And also at the processes and technology, which is about putting in place the right processes, controls and technological safeguards to manage AI effectively.

Now, I think traditionally you've seen tech adoption as more of a kind of top-down approach. With AI governance, I think to ensure that we're not behind the curve, we need to really make sure that we have proper accountability and that we ensure that we're continually engaging with our staff. So we're continually finding out the new risks that we may not have anticipated at the beginning of our adoption process. And I think this is an ongoing conversation.

NC: And this is big, isn't it, Sharon? As we're seeing more of these agents, more of this decision-making capability, which Ray has talked about, you know, who is authorised to use this AI? What decisions can it make? What data does it have access to? You know, what records does it keep? I mean, I could go on.

So to ensure your governance is right, Sharon, you need quite an understanding of all your AI deployment, I mean, both now and in the future. And I wonder whether some organisations even have that.

SH: Indeed. And I mean, we're at the beginning of this journey, aren't we? I think for a lot of us working in HR functions within organisations today, we don't know what we don't know yet. And you know, we're in the early stages of this journey. But one thing's quite clear so far, we've heard it in what we've spoken about already in this podcast, is it's going to continually change the landscape.

So in terms of the continuous requirements to review all of the governance aspects, there's going to be an ongoing and continuous change to the world of work and the human impact that this brings as well. So that's in relation to the jobs, the tasks, the workflows, the accountability, and then the skills required to cope with that. So the governance, if you like, in answer to your earlier opening remark, Nigel around HR's input to that is in my mind, you know, to help ensure the structures in place to understand those changes and identify those changes as they evolve so that we can react and respond to them and provide the appropriate mechanisms as we move forward and create the right environments for us to continue to innovate in a safe and ethical way.

NC: So Ray, I mean, you've worked through this process with a number of organisations, still do. And you've been tasked, say, with questioning and boosting your AI governance. Where do you start?

RE-P: It's a very good question. I think actually going back to what you were talking about earlier, knowing where AI is used is actually the starting point. Finding a way to build an AI inventory. And I think most critical is identifying what I call checkpoints within the organisation.

Typically this comes where funds are involved. So if you're going to either build new AI internally or purchase an external AI program or service, typically you're going to need resources, you'll need funding for that. So that's a good place to ask a question. Is this money going to be spent on something that uses AI? And if it is, then you want to capture that in your inventory and that allows you to do a risk triage and then a risk assessment.

So I think to the point of innovation, it's really important that you establish first of all, is this a very risky use of AI or potentially risky use of AI or not? Because if it isn't, you don't want to sort of overburden it with lots of risk assessments and so on.

I'll give you an example. Typically, we think about AI usage in defence as being quite high risk because obviously what you immediately think of with defence would have very serious consequences. But let's say you're using AI to generate the weekly menu in the canteen. That probably doesn't need a lot of AI governance because it's a pretty straightforward use of AI. So this risk triage, I think, is very important up front to identify the risks that are potentially large and then apply your detailed risk assessment to those use cases.

NC: And that seems to speak to some of those questions that I mentioned, like what work is the AI doing, who's allowed to use it, what decisions can it make, what data does it have access to do and so on. Helen, you use working groups, I think a fair bit in your organisation. Just explain how you actually carry out that process of examining and checking.

HD: So similar to Sharon, our organisation was part of the Innovate UK program as well, which provided a really good framework to work through with the workforce to look at where do they see the risks are and where do they see the opportunities are in terms of AI use and really think together about actually what are the red lines? What do we think we don't want to use AI for in our organisation because we feel they're too risky or we feel it's unethical. And in that journey of really working across the organisation and having groups to surface this, we came to some red lines around recruitment. So our workforce didn't feel comfortable in terms of using AI for selection in recruitment because of the inherent bias that there could be in that data. And then we've worked together with them, when we got to the end of the process, we set up a working group to really carry on that work to decide where AI would be used, what we consider high risk and what isn't.

And the working group we set up really has representatives from all departments in the organisation across different functions. So we have myself for governance and HR, we have an IT specialist, legal people working in operations and business processes to really come in and map what is the benefit that we're going to get from AI here and is this a justifiable use?

So from that, we've really kind of made clear, as Ray pointed out, we've kind of looked at the risk factors and our guidelines make clear that AI use is very different for things like drafting, research, summarisation, ideas generation. So that can be used in the LLM. We can carry on with that use without reverting to the working group. But for higher risk areas, so if were thinking about higher volume workflows, for instance, or automated processes or agents, they have to go through the AI working group so that we can really manage that risk. But, managing that risk involves looking at the risk from multiple perspectives. So, we can't just take a legal lens. We can't just take an IT lens. We want to look at how it operates. So, we really have those conversations and challenge each other to decide, is this a good use of AI in the organisation?

NC: Now, Sharon, I kind of, I get all that, what we heard from Helen. I understand the nature of these conversations that go on, but I still struggle to kind of understand what governance looks like in practice. I mean, is it rules? Is it, you know, in your own organisation, obviously when you do your training, your compliance, how does the governance rear its head, if you like.

SH: Interesting perspective because like Helen's already described in her organisation, we've approached our AI journey, if you like, at two levels, really. We have a data and AI council, which provides the structure and the governance and the technical controls in place to make sure that we review AI use cases and pilot them and deliver them and implement them safely and in a controlled manner. But we also challenged ourselves to make sure that whether we were governing the organisational consequences with the same discipline, if that makes sense. So we therefore made people and organisational impact quite explicit with that work. And to the point that Helen's made, any sort of high-risk activities that we need to look at, we look at in that framework.

and then we didn't have working groups, for example, we delivered a number of prompt-a-thons as they became labelled, which caught on because it was around using the tools like co-pilot and effective prompting, et cetera, and giving colleagues, employees, the confidence and the knowledge to know how they can kind of use AI in their day-to-day activities. But we set frameworks around that.

So for example, the governance included, you know, you couldn't be, you couldn't receive access to a co-pilot license unless you've done your online ethical use training and you'd read all of the guardrails around there. You know, we've talked about guardrails and having those in place and staff realise where they can and can't use it and all of the usual or common sort of checks and balances that you'd have in place to make sure that they're not letting the AI do the activity, that there's the human checks and the accountability and judgment that goes in there.

So that's how it's kind of manifested in that respect.

NC: But just before we leave this, I mean, Ray, there's an awful lot here and we're looking at this sort of partly through an HR lens. There's an awful lot for people to get their heads around, isn't there? Stuff like understanding data privacy, you know, how that discrimination or bias we were hearing about earlier happens, detecting hallucinations, regulatory compliance and all that. I mean, how on earth can people all slightly new to this, who are used to managing people who are unpredictable enough, [laughter], now we're managing the machines. How have you found organisations cope with all that?

RE-P: I think there are two things that you need to do really. One is obviously training, absolutely critical. And I'm a strong believer in training across the whole organisation. So let's think of it as awareness training if you like. I think everyone needs to have that basic level of training about AI, both what it's capable of, its potential, but also the risks. And then you'll have more specialist training for people who are using particular types of AI in their day-to-day work.

But I think the other thing to make it straightforward for people is actually to put workflows in place or, you know, to enforce checkpoints or stage gates and so forth so that as people go through a process of using AI, they're automatically asked questions and kind of reminded, if you like, of their obligations and how they need to behave.

The most challenging area, frankly, is something like a co-pilot where it rapidly is becoming embedded in everyone's every day and they can do more or less anything with it. the use cases are very broad. You could put all kinds of different types of information in and so on. So that's where we do largely come back to relying on people to do the right thing, to understand what is sensible to do and what is not sensible to do.

NC: Sharon, in many ways, more of this seems to be about personal responsibility. I know you wanted to talk earlier a bit more about what you do in practice with people. Maybe this is a good point to do that.

SH: Absolutely. I mean, we've talked about having, you know, workflows and stage gates and experimentation. And, you know, I think we've already touched on the points around personal responsibility. And I've referenced, you know, when we've rolled out some of our tooling, we've place an expectation on colleagues that they will adhere and stick to these guardrails, or there will be consequences, of course. And there's lots to consider around reputational damage and brand damage as well, if someone doesn't act responsibly and ethically.

I think from an HR lens and perspective, I could give you an example of how we practically run a co-pilot agent into the organisation, put that into production, where it's provided, lots of opportunities for colleagues that might have previously searched across meeting notes, briefing packs, all of that sort of activity and then that agent's done that work for them.

And so the governance that we've wrapped around that really is to ensure that that's all remained permissions based and those individuals that are using that agent are really clear about what their responsibilities are. And that's all grounded in the source material. The colleague reviews the output and owns the judgment. But that in turn, of course, is changing the way that work is being carried out.

And so from an HR point of view, where we're involved in that activity is as those agents go live, we need to listen to the users and test whether that's genuinely, A, reducing their effort and improving the work and delivering the benefits it was intended to bring when it was rolled out, but also identify those skills and capabilities that are required in order to feel safe and operate in that environment. and make sure that psychological safety is in place for everyone and they feel comfortable calling out if they believe there's risks or they're not comfortable with that.

To the point around governance, I think there's a really important element of colleague voice that has to be built into that and have that continuous feedback and that loop in the process to go back and check, check-in with people, you know, how has it been? How has it worked in practice? What's the impact it's having on you personally? And that's continuous. We seem to come back to this continuous approach really, rather than just, it's rolled out, it's been deployed, fantastic, carry on sort of thing, if that helps.

RE-P: I was going to actually pick up on a point that you indicated earlier about sort of overload, employee overload. And I think that there are two areas where HR has a really important role to play in shaping the way that we use AI going forwards.

So one is this topic of overload because I talk to a lot of people and see instances where people feel that the AI is going so fast that they can delegate things or they can do things with AI, but they all come back to the individual then to be checked or to be evaluated or incorporated into their other work or what have you. So all roads lead back to the individual, but they're leading back very quickly and in volume. And a lot of people are starting to feel a bit stressed and overloaded by this.

And then the other area where I think HR really needs to help is in terms of what I call cognitive atrophy. So, how much do we outsource to AI? How do we ensure that humans remain critically thinking, using skills, don't forget how to do things. And that applies, I think, both at an individual level, but also at an organisational level. You know, an organisation can lose critical capabilities if they're handed over wholesale to AI. Is that safe? Is that what the organisation wants? And also, what does the organisation kind of stand for? How does it want to work with people? These, I think, are really important questions going forward. And I think HR has a critical role to shape them.

HD: I can relate to what the lot of what Ray is saying there. I think on the employee overload point, we have seen at certain times this kind of overproduction of documentation. So, AI can generate a lot of information very quickly. and then it can pass the burden to others in the organisation to check it. So if you think systemically, you can think, oh, okay, there is a productivity gain by the person creating the document because they think they have worked faster, they've created something of use quicker, but then you have a productivity loss somewhere else in the system by virtue of this checking burden. And this can overload employees as well.

NC: So maybe the government should say one sheet of A4 only.

[Laughter]

RE-P: Yep

HD: Well, I think, in a similar way to if you're producing a board report, you need to really look at what are the core messages you're getting across, what are the decision points, and really make sure that you have put your own judgment in there. So not judging quality on the volume of documentation, but on clarity on the main points and really drawing that from AI and your own judgment.

NC: Now, I'd like to ask all three of you something which we hope will be central to this discussion, and that is the question of governance ultimately being an enabler and not a blocker. And I wondered whether, Sharon, you could explain what you can think of any examples of how a strong governance framework actually supports the kind of ongoing innovation that organisations need to meet their long-term goals.

SH: Absolutely. And it's interesting, isn't it, when we talk about the cognitive load on individuals and the impact that AI is having on individual wellbeing in organisations, because I think that's relative here as well.

So I think the way that we at Pay.UK have rolled out our governance framework really has been designed at a pace and a speed in order to help support that continuing innovation moving forward. So for example, we have been very structured in our identification of where we want to bring in AI, the use of AI in the organisation so that it doesn't become overbearing for people and it's proportionate and we know exactly what value it's going to bring rather than just bringing in AI for the sake of it.

We have applied a governance structure and a framework to assess use cases and then pilot those and roll them out in the organisation and then test that. So by having that framework and that governance in place and doing that in a structured and measured way, it does mean that the safety rails and the guardrails can be put in place so that we can deploy that technology and the individuals that are working alongside and developing that technology can be supported and that can be delivered in a sustained way.

NC: And is that your experience, Helen, that if you set clear expectations, if you've got decision making processes and the boundaries you were talking about earlier, it does actually create space for people to experiment or do they tend to get a bit scared?

HD: Yes, yeah. I mean, I think the idea that AI governance hampers innovation is misfounded. So I think good AI governance in the form of clear guardrails provides people with the confidence to innovate. So we found in our organisation, when we first started the process, we created a set of legal guidelines that they were created more by the legal department and the SLT. They weren't necessarily understood by staff. They didn't have real life examples. And when we went through the process of engaging with our staff and really looking at AI uses, looking at red lines and considering this together, we then reformulated our guidelines based on this with very clear red lines, very clear behavioural expectations about what we expected in terms of checking outputs, being accountable, being transparent. And what we found was once we had the clear guardrails, actually innovation increased because previously we had what we call a compliance anxiety. So this compliance anxiety, the worry of doing something wrong, was actually stopping people from innovating. And this was one of the kind of outcomes that came out of the Bridge AI program that we were in. So if people are interested in looking on CIPD website, you can see those case studies and that was from MemberOrg, which is our organisation.

NC: And Ray, from your own experience, does governance enable longer term benefit for organisations to spot and take up opportunities without big mistakes? Absolutely.

RE-P: And that's actually the reason that I wrote my book was that I was working with a lot of organisations and I was seeing nervousness, particularly on the part of senior leaders, holding back their willingness to use AI. And they weren't sure how to go about it and sleep at night knowing that the right kinds of controls and governance was in place. And so they were tending to put the brakes on. And that's where I think once you've got a good governance program in place, it actually gives people the confidence to use it. They know the right questions to ask. They know the boundaries. They know the tools they're allowed to use, the ones they're not allowed to use, these kinds of things. And that takes away, I think, Helen, you used the word anxiety. It takes away that anxiety from, from colleagues and they feel, they feel more confident and actually liberated.

NC: So just trying to put this all together, Sharon, if we think about the sort of big risks, the data privacy, the security breaches, the bias people talked about earlier, IP infringement, hallucinations, how does the governance ultimately help?

SH: I think the governance helps us because to the points already made, it creates that structure and that framework and it gives people a sense of safety that they can have the space to experiment, use the tools that are being made available to them, be supported in that process in understanding the tools and the techniques that are available and the skills and capabilities that they need to develop in order to work well in those environments.

So an example that I could give in that respect is around our talent and succession management exercises and activities that we have within the organisation. We're doing a lot around capability mapping and identifying what the future roles and capabilities will need to be in an AI enabled environment. So for example, succession planning is no longer around replacing the role. It's about replacing some of the capability and skills these days. And so just to sort of understand whether you've got a successor role is not enough anymore. It's understanding what that role and that succession requirement and the skills gaps associated with that will look like in the future.

So it's around providing that holistic space and taking a holistic approach with the governance to make sure that as the technology is deployed and rolled out, we've got a good framework and structure to identify the impact that's having and the change that brings in the organisation, whether that be the technological change, the job changes, the working processes, the accountability and the workflows. And people feel, I guess, confident, to Ray's point, that they can operate and thrive in that environment because it's a journey, isn't it? It’s a learning process for everyone and it's a journey for everyone and everyone needs to feel supported that they can cope with that change because ultimately it is a change management process. It's not just one and all and delivering a technology solution. It's a constantly evolving situation.

HD: I completely agree with Sharon. It is a change management process. And I think key to that as well is developing trust in the organisation. So I think that's another way that governance can enable innovation. So if staff know that there's clear accountability for outputs, if they know that there's transparency, it develops trust and trust ultimately encourages engagement. And I think when we think about implementing AI solutions, a lot of the time we need to look at it systemically again. So we need to think about how does an AI implementation affect the individual? How does it affect the department and the organisation as a whole? And individuals may be best placed at times to come up with ideas around how AI can be used to support workflows better. But if they're going to have certain concerns, like perhaps concerns about being displaced by AI, are they going to bring those ideas up?

So I think that is another way if we've got that trust, we've got clear accountability, clear transparency that can only support innovation.

NC: And Ray, I know you had to think about all this a lot for your book. What do you think people find hardest when it comes to AI governance and how can they best be helped to see and understand?

RE-P: I think the hardest part is the change management, as Helen and Sharon were alluding to. People don't like change in general. And in an environment where there's a lot of uncertainty around AI and future of jobs and this sort of thing in particular, they're nervous. So I think that is the biggest single barrier. And I think it's about coaching. I'm a strong believer in actually going in and working closely with individuals. Yes, it's helpful to have an online training program, you know, the half hour, the hour, whatever it is. But actually what really works is taking AI champions or people with a sort of a more technical background and having them go in and sit down with colleagues, sort of good old-fashioned one-to-one contact or get a few people around the table, understand what their job entails, and then explain to them how maybe AI could be used and help them to understand the potential value and to reassure them, sort of hand-holding, as it were, through the process. And once you do a little bit of that, then I think people get that initial confidence and then, off they go and they're sort of liberated to do it on their own.

SH: Yes, I was just going to build on that point actually, Ray, because we found that approach worked brilliantly because, you made a point earlier around looking at the thinking and the education required at different levels of the organisation. So the approach we adopted, which worked really well, was looking at the teams and the functional levels and to what levels we needed to develop that AI awareness and that AI training. So we've rolled out some general awareness to support the organisation overall. And yes, of course, a lot of that has been online for practical reasons. But then with the prompt-a-thons that we brought in, we did bring experts in and we sat around the table and they, and we encouraged staff and colleagues to come along with their own little case study or use case that they said, right, this is what I would really love to be able to do in this scenario. How can I do that safely? And it was amazing because as that started, it built colleagues' confidence around that. And before we knew it, we were getting requests for use cases, every single week to say, oh, this is amazing. How can we do this? So that helps with that. challenge as well in my experience.

NC: And I can see that might help winkle out those people who might have held back on AI applications because they thought maybe it would hurt their employment prospects or all kinds of reasons why people have, including bosses, we were hearing much earlier.

HD: I think there's something about how you sustain the change as well. So, you've gone through a big change management process to get AI implemented. It doesn't stop there. The changes keep happening with AI. There are changing use cases and we need to keep abreast of that as well.

So I think something that we've done is, so we've had peer-to-peer learning sessions where we've supported and had spaces for experimentation, but keeping that going. So having kind of drop-in groups, we have regular drop-in groups where people can come, there'll be members of the AI working group there to just say, “I've got this idea” and to start thinking together with colleagues, but just seeing it as an ongoing process. The change doesn't stop when the implementation is there.

NC: Great. I think between you, you really made the case for strong AI governance, given its profound impact on us human workers as well as the business itself. Because I mean, the alternative of machines managing the humans by algorithm, well, that doesn't bear thinking about, does it?

So, thanks very much to Ray Eitel-Porter, Sharon Hill and Helen Dutta for their excellent insight. CIPD has freely available online resources on this. There's a guide to ethical, legal and sustainable use of AI and there's some advice on the wider HR role in AI governance. But until next time, from me, Nigel Cassidy and all of us on the podcast, it's goodbye.