Tech-focused conference Libraries Rewired takes place on 10 November at London’s Code Node. Machine Learning Librarian at Hugging Face Daniel van Strien will be joining Digital Curator at the British Library Mia Ridge for a session on Harnessing AI and Machine Learning.

Here Daniel looks at the relationship between libraries, AI and machine learning and how to ensure the technology delivers for your service. You can book your place for Libraries Rewired at www.librariesrewired.org.uk

Almost every industry asks, “How will artificial intelligence impact us?”. Libraries (and related institutions) are also frequently asking this question. Indeed, a recent report commissioned by CILIP was titled The impact of AI, machine learning, automation and robotics on the information professions (www.cilip.org.uk/researchreport).

That libraries are grappling with the impact of AI on the sector is not so surprising when we think about what people claim AI, particularly Large Language Models, can provide, namely a way to access high-quality information efficiently. This is also one of the primary goals of many libraries and librarians.

As a result, librarians might worry that this new technology has the potential to make their work less relevant or even make it obsolete altogether. However, I believe there are many areas where libraries and librarians can take a more proactive stance in determining where and how machine learning might potentially enhance the work of libraries, as well as taking a more active role in using and developing appropriate machine learning based systems. Whilst libraries are right to ask what is the impact of AI on libraries, they should also be turning this question around and asking: “What is the potential impact of libraries on AI?”.

What can AI do? A lot of the recent excitement about AI has been focused on one branch of AI: generative AI. Generative AI includes tools like ChatGPT and Midjourney. As the name suggests, Generative AI refers to systems that generate open-ended outputs, often based on an input prompt.

For example, when you ask ChatGPT, “What is the role of libraries?” you are providing a ‘prompt’, in this case, a sequence of words. The objective of a generative system is to continue to generate text based on this initial text. One of the appeals of generative systems is the flexibility they exhibit. A single system can respond to a wide range of prompts and produce a wide range of responses. However, this is also one of the significant limitations of generative AI. Since the responses they produce are so flexible, they can generate a text which has the appearance of being a sensible response to a prompt but could contain subtle or not-so-subtle errors.

Where can information skills help? There are various ways in which you can try to ‘steer’ the outputs of a generative model. For example, with a system like ChatGPT, you can manipulate the prompt in various ways, with the aim of receiving a ‘better’ response. These manipulations of prompts include asking for a response in a specific format, providing examples to the system, or adding encouragements such as “Take a deep breath and work on this problem step-by-step” to prompts.These strategies all fall under the heading of “prompt engineering”.

This is one area where traditional information retrieval skills held by librarians can potentially play a huge role. Many information professionals have experience in generating fairly complex searches. The skills required to do this are not dissimilar to many skills one needs to become good at prompting a model to generate the desired response.

The fact that some of these prompts appear to address the system as a living thing, i.e. one that can take a deep breath, should not be misunderstood. Under the surface, these systems are all essentially producing a set of probabilities based on an input response, i.e. based on the training data for a model; it will return the series of words with the highest probabilities based on the given prompt.

A full version of this article will appear in the October/November issue of Information Professional, which is free for all CILIP members. Daniel will discuss these issues in more detail at the forthcoming Libraries Rewired conference on 10 November, 2023 - book now.

Join us today to support us in our mission, gain a boost in your career, or to start making a bigger impact in your industry.