GenAI, CoPilot and related technologies are generating a lot of buzz. IT senior managers and executives are either full of expectations or under pressure to harness such technologies and realize business value. The type of business value hoped for - and often robustly demanded by CFOs, CEOs and company boards – are increases in efficiency and effectiveness.

Bottom line: do more with less!

The market seems to be a kind of large sea boiling and bubbling. It is not easy for executives to get their head around this specific topic: How to increase IT department efficiency by adopting GenAI (and related technology) at scale?

The purpose of this brief article is to share some quick insights

CoPilot or ChatGPT?

These two technologies might appear very similar to be almost considered synonyms but indeed, right now, they are quite different in their purpose.

This regardless the fact that, back in November 2023, CoPilot Chat was updated to use GPT-4 model underneath.

But first a bit of (very) fast-forward history.

GitHub platform began on 19th Oct 2007 by providing code repository services. It became popular and grew along the open-source movement. Initially Microsoft was only a big user of GitHub. However, eventually Microsoft acquired GitHub on 26th Oct 2018 for a reported US$7.5 billion!

In 2021, GitHub released CoPilot. Shortly after, it was announced CoPilot would be available as a subscription-only based service. It was June 2022.

GitHub was initially powered by OpenAI Codex, that could be considered a “production version” of GPT-3 (2020), additionally trained on 54 million public GitHub repositories.

Not long after, and more precisely in Jan 2023, Microsoft had invested 10 billion US $ into OpenAI Global LLC. A significant move with far reaching implications, still panning out.

Not surprising, in November 2023 CoPilot Chat was updated to use GPT-4 model, as already said, and very recently there has been the OpenAI announcement of its intention of becoming a for-profit company.

However, regardless all of the above, right now, GitHub CoPilot is positioned as an AI-powered coding assistant. Similar concept has now been introduced also in the MS Office space.

That is: A personal productivity tool. In CoPilot terminology a trusted pair programmer.

This means, in my mind, that CoPilot should not be considered a substitute for automatically executing significant programming tasks on its own.

The latter could instead be the “role” of GenAI (e.g., ChatGPT or similar generative AI tools).

In the remaining of this article, I will maintain this distinction since I find it useful to bring some clarity.

In other terms: CoPilot is to be considered a pair programmer tool. An helper for allowing a programmer to perform his software development tasks a bit quicker and more reliably. In one word: a personal productivity tool.

GenAI tools are instead to be considered as tools aimed at enabling a potentially very high degree of automation within the activity of an IT department. A good metaphor for that is a robotized conveyor belt that enables mass production compared and contrasted with handcrafted production, as the software development activity has been so far.

An important consideration first.

Before going any further, I would like to pause to clarify an important overarching consideration.

These GenAI tools (CoPilot included to some extent) do not make junior people more senior. Actually, junior people could potentially be hampered in their need of growing their knowledge whilst on the job.

This is nothing different from the perils of Internet. Indeed, to be able to effectively search for knowledge and for relevant quality-rich information, it requires the possession of a great deal of preliminary knowledge and awareness.

So, what is true, contrary to the prevailing marketing narrative, is that the need of very senior people, with a broad and deep technical competency, is still pretty much the case. Perhaps, far more than before, since these people will now need to complement their technical expertise with GenAI skills.

What actually GenAI helps with, is that it allows their expertise to scale-out exponentially, at least potentially.

And the reason is simple. As stated before, ChatGPT, or similar GenAI tools, is more similar, at least in principle, to what an automatic conveyor belt is for a manufactory company. They are there to automatize mass-production.

However, to design and set up a new production-line for a mass-produced good (like cars, washing machines, televisions, etc.) it takes upfront time, investment, etc. And most importantly skills! Therefore, very senior people.

All these upfront investments are then regained only once the break-even is reached.

This metaphor has profound implications on how to enable the usage of GenAI within an IT department.

What does it take to adopt CoPilot at scale?

So, adopting CoPilot at scale within an IT department is a different kettle of fish respect to comprehensively embedding GenAI driven automation within the same.

At risk of appearing blasphemous, CoPilot can well be considered as a powered-up IDE (Integrated Development Environment) and therefore rolling it out its usage at scale is a relatively straightforward exercise: it takes some planning, the setting up of the relevant organization and then the execution of the required training.

That is, as a title of example, instructor led sessions, the curated selection and gathering of training materials (courses, books, etc.) to enable self-training, a suitable approach, for instance, train first the trainer and avoid a big bang approach, institute on-going mechanisms for coaching, institute the right incentives and effectively measure adoption progress, etc.

The benefits to be expected are some improvements in productivity which, though, might well remain potentially unmeasurable, since too small to be noticed

What does it take to adopt GenAI at scale?

To the contrary, adopting GenAI (or related technology) at scale within an IT department is to be seen as aiming at introducing an unprecedented level of automation within its currently handcrafted operating model.

In order to do it properly, it is important to pivot such adoption on very senior people, experts of GenAI for sure, but also with a deep and broad expertise of the underlying software technologies which are the target of GenAI usage, e.g., coding languages, libraries, design patterns, frameworks, underlying data model, network protocols, etc.

This because to effectively interact with such a sophisticated tool and fully leverage its responses, often only partially corrected, what is required is the ability to read between the lines of GPT’s wrong or incomplete responses to foster creative intuitions about fine-tuning the prompt and or the approach to achieve reliable automation.

Other important elements which need be taken in consideration before articulating some suggestions about a possible approach for GenAI adoption are:

• Prompt engineering is of paramount importance.

ChatGPT is not deterministic. It means that the quality of its responses, their value and relevance to the problem at hand, can be exponentially better with a slightly more ‘accurate’ prompt.

To this regard, it is important for managers to be aware of a common antipattern quite widespread amongst companies: the usage of technical terms to mean something different from their precise meaning.

Often people learn the technical terms by kind of guessing their meaning from the conversational context where they have heard them as used by their colleagues.

This, some time, generates imprecision in their usage which may take root within organizations or departments. Their usage within the prompt with a meaning in mind that is not quite correct obviously might contribute to hallucinated responses by GPT.

• Not all projects are equally suitable to be automatized with GenAI.

One of the most important points to keep in mind is that not all projects are amenable to be taken up with GenAI. There isn’t a “YES” or “NO” answer: Often projects are somewhere in a continuum between these two extremes.

However, generally speaking, the most suitable projects to be tackled with GenAI have some of the following traits:

• The legacy code languages and target code languages are (or used to be) common. That is: there is a large number of projects using such languages in GitHub.

• The project is large (e.g., with a lot of code lines) but somehow with repetitive patterns. For instance, projects that are made by a significant number of large modules each of which exhibits similar patterns in its code structure. Such a things would allow a successful

fine-tuned automation approach enabled by GenAI to be applied and scaled out to the other similar modules.

• Source code is preponderant compared to configuration logic, potentially coded via configuration scripted in XML, XSLT, etc.).

• The more the business logic is coded in a ‘self-contained’ way (i.e., with little or no usage of frameworks, configuration files enabled by platforms, etc.), the better is. This is typically the situation of old legacy systems where the preponderant part of the business logic was coded by programmers bottom up.

• Finally, the more a project code structure is ‘partitionable’ (e.g., can be easily dealt with chunks by chunks) the better is, given prompt size might be capped.

• Obviously, projects with an homogeneous syntax are easier to be ‘understood’ by GenAI respect to those who mixes a lot of different languages.

• A project execution can be automatized in a number of (often) unexpected ways. Exactly as in the case of designing a conveyor belt, which thing implies the detailed understanding and conceptualization of each task required to produce a specific good (together with their relationship, dependencies, etc.), a software project can be potentially automatized in many different areas and ways, sometimes unexpected. So first, it is important to have a team able to conceptualize the project, in the sense of determining all its tasks to properly structure the search of GenAI-driven ways to automatize it.

Having said all that, it should appear clear that to adopt GenAI at scale it is not just a matter of rolling out a training programme as in the case of CoPilot. Its successful adoption MUST BE pivoted on sophisticated knowledge: traditional technical knowledge and GenAI specific knowledge.

It is the nurturing, spreading and on-going assessment of the quality of such knowledge that should be the underlying engine to support its adoption. Given that rolling out a training programme is not enough, my suggestion is to set up a specific governance.

A possible governance to support GenAI adoption at scale. I will conclude this brief article illustrating some traits of such governance.

• It should be impinged into a carefully designed CoE (Centre Of Excellence) Since knowledge is key it cannot be mass-produced at once, therefore it needs to be nurtured via a CoE. But, in order to do so effectively, such CoE should not become a remote ivory tower that dispenses good high-level pieces of advice, since too busy with too many projects. Rather, it should be very project-focused and provide a real hands-on leadership to projects. Here the model is more similar to university PhD supervisors. In other terms the success or failure of a specific project is also a direct measure of the effectiveness of such CoE.

• Objective criteria to measure results are a MUST. So, it is the comprehensiveness of data collection.

This also implies the need to carefully design a very rigorous and meticulous approach to collect relevant and objective data to enable the selected metrics to yield meaningful and truthful measurements.

It is important to understand that in order to enable this trait, the prevailing culture should not mandate success, but only transparency!

Scientific and technological advances are achieved upon the ability to learn from the lack of success, little by little.

• Put in place the right mechanisms to manage and further develop the acquired knowledge.

It is important to understand that GenAI is not a mature technology whose usage has been reliably engineered. It is still pretty much in an early R&D stage (though developing tumultuously), so are its business applications. In that respect carefully recording the lessons learned at a meaningful level of details, is of paramount importance to nurture knowledge and spreading it.

• Have any project initially vetted by the CoE.

Not all projects are equally suitable to be treated with GenAI. An upfront assessment of the most promising ones is important to show business value early.

Tackling a software project with the help of GenAI to achieve a high degree of automation, is like designing and building a robotized conveyor belt for mass production. It is important that the project is such to allow to break-event the initial effort (and cost) invested upfront to experiment and understand how it can be automatized with the help of GenAI.

Of course setting up a company-wide governance to accelerate GenAI adoption across an IT department is a complex task that doesn’t have one-size fits-all solution, given the culture of the IT department, its organization and its technology landscape also plays an extremely important part.