Agentic AI: A comprehensive guide to building reliable AI Agents

AI is no longer science fiction—it’s already changing the way businesses operate day to day. Among themost game-changing developments? Agentic AI. These systems run on Large Language Models (LLMs) and are built to take the wheel—figuring things out and getting stuff done on their own, all based on what the user asks. Despite their impressive capabilities, they also bring new challenges to the table. If we want them to be reliable and actually useful, we need to be mindful of their limitations and tackle them.

This article explores the concept of agentic AI, its architecture, common pitfalls, and strategies for building robust systems. By the end, you’ll understand not only what agentic AI is, but also how to mitigate its challenges.

‍What is Agentic AI?

Let’s picture a scenario: an AI agent named Giovanni receives a simple command:
“Giovanni, mow the lawn.”

Being able to understand natural language, Giovanni easily grasp this request. Of course, simply understanding the command is not enough—he needs to figure out how to get it done and do it. This involves reasoning and planning each step required to fulfill the request. Giovanni breaks the task down into a sequence of actions:

• Walk to the garage
• Locate and take the lawn mower
• ...

But planning alone doesn’t make Giovanni an agent. Once he has mapped out the steps, he needs to act. This means interacting with the world around him and using the appropriate tools. In this case, he powers on the lawn mower and uses it to cut the grass.

And that’s the essence of an Agent: an AI model that doesn’t just passively respond, but actively reasons, plans, and interacts with its environment to achieve a goal. The term agentic comes from the word agency — the ability to make decisions, take initiative, and execute actions autonomously. An agent is not limited to providing information or generating outputs; it’s designed to think, plan, and do.

More formally, agentic AI refers to a system that combines reasoning, planning, and execution to autonomously achieve specific goals. These agents rely on advanced AI models to interpret user instructions, think, make decisions, and perform actions. Unlike static software programs, agentic AI is dynamic and adaptive, capable of interacting with its environment in real-time.

Agentic AI systems are composed of two main parts:

• The Brain: This is typically an LLM like OpenAI’s GPT-4 or Meta’s Llama. It handles reasoning and natural language understanding.
• The Body: This includes tools that allow the agent to interact with its environment. For instance, tools might enable the agent to send emails, retrieve data from APIs, or generate images.

By integrating these components, agentic AI can perform a wide range of tasks—from automating customer service to generating personalized content.

‍AI Agents architecture

To grasp how agentic AI works, it’s essential to delve into its architecture. In other words, it’s essential to understand the respective roles of LLMs and tools.

Large Language Models (LLMs) are advanced AI models trained on vast datasets to understand and generate human-like language. Built on transformer architectures (which leverage mechanisms like "attention"), LLMs excel at predicting the next word in a sequence (for efficiency reasons they actually predict sub-words known as token). This capability makes them ideal for tasks such as conversation modeling and text generation.

These models serve as the "brain" of agentic AI systems, enabling them to interpret user instructions and reason about tasks.

LLMs are powerful probabilistic models capable of generating a wide range of outputs but are unable to directly perform actions in the real world.  This is where tools come in. Tools are external functionalities provided to LLMs that enhance their capabilities. Examples include:

• A calculator tool for arithmetic operations.
• A web search tool for retrieving real-time data.
• APIs for interacting with external systems like databases or email platforms.

However, it is not sufficient to provide the LLM with tools to create a functioning agent. We also need to: 

• Instruct the model on the tools they are equipped with and how to use them.
• Guide their behavior to suit our needs.

We can achieve these tasks using system prompts: persistent instructions read by the model before the user input (or prompt). They contain information about the tools available to the LLM and how to invoke them, but also about the way the agent should think, interact with the user and structure its responses. This is crucial in order to obtain an agentic AI tailored to our needs. For instance, a simplified version of a system prompt could look like this: 

You are a helpful, friendly AI assistant specializing in answering questions about data science and machine learning.
Your goal is to provide accurate, concise, and easy-to-understand responses to user inquiries. When asked to write code, provide well-commented, functional code snippets in Python.
You have access to a calculator tool for performing mathematical calculations. To use it, enclose the calculation within <calculator> tags. For example: <calculator> 2 + 2 </calculator>.
Be polite, professional, and avoid generating speculative or biased information. If you are unsure about an answer, say "I don't know, but I can try to find out."

‍What are AI hallucinations?

Agentic AI can be incredibly powerful—but let’s be honest, it still has plenty of bugs to iron out.  Whoever build agentic AI must be aware of these challenges and implement strategies to mitigate them.

Perhaps the most concerning issue is the tendency of LLMs to generate false but plausible information – a phenomenon known as hallucination. This can severely undermine the agent’s credibility and it’s a significant risk when deploying LLM-driven systems in critical applications.

In agentic AI, hallucinations occur when the system generates responses that:

1. Are not grounded in its input data or knowledge base.
2. Contain fabricated or incorrect information.

There are different types of hallucinations, which are more or less harmful and which can be better understood by categorising them.

Before proceeding it is important to understand that the model output is based on source material that depends on the task we want to perform. For example, while answering a question, the source material will be given by the knowledge learned by the model during training. This is known as world knowledge.
On the contrary, on summarisation tasks we want the agent to solely rely on the information provided by the user, the .

In their relationship with the source material hallucinations can be categorized into two main types:

• Intrinsic Hallucinations: These are manipulated information that contradict the source material. For example, when asked who won the gold medal in the 100 m race at the 2022 Summer Olympics, the model could answer “Usain Bolt”. This is not true, as the 2022 Summer Olympics never took place. Note that the model might hallucinate while giving a truthful answer. For example, the model might disregard source knowledge when summarizing and rely solely on world knowledge, which contradicts the information it should use instead.
• Extrinsic Hallucinations: These involve additional information not supported by the source material. For instance, an AI might invent a non-existent feature of a product during a customer interaction.

We can also classify the agent output by its degree of truthfulness and justification. By combining these two parameters we will have four possible type of answers:

• Justified and True: This is what we want, and the only case in which the model does not hallucinate, providing accurate information supported by the source material.
• Justified but False: The AI's response is based on outdated or incorrect information, leading to a false conclusion despite sound reasoning. This indicates a need for improved or updated data sources. 
• Unjustified but True: The AI's response is factually correct but not based on the information it was given. This could indicate "lucky guesses" or reliance on knowledge outside its designated scope. For example, the AI telling the customer that they should arrive at an airport 2 hours before a flight departure. If this was not grounded in, say, a knowledge base article, then this information is technically a hallucination even if it is true.
• Unjustified and False: The worst-case scenario. The AI's response is both incorrect and not supported by any available information. This represents a clear failure of the system and requires immediate attention. These are harmful hallucinations that could require an organization to reach out to the customer to fix the mistake.

‍Sources of hallucinations in AI agents

There are a lot of possible sources of hallucinations and improper answer 

1. Stochastic Nature of Generative Models: LLMs generate outputs probabilistically, which can lead to variability and errors. This is especially true given their huge output space.
2. Incomplete or Outdated Data: If the system relies on outdated knowledge bases or APIs, it may produce inaccurate responses.
3. Complex Instructions: Ambiguous or poorly structured prompts can confuse the model.
4. Parametric Knowledge Bias: LLMs store vast amounts of information within their parameters, which can lead to over-reliance on "memorized" knowledge rather than external sources. This is especially true for very large models.
5. Reasoning failures: When the model fails to provide the correct answer due to logical errors.
6. Tools usage errors: When the model poorly exploits the tools at its disposal
7. Exposure bias: The LLM generates one token at a time. Therefore, when generating longer and longer answers, the output will increasingly depend on synthetic text generated in previous iterations. This may lead to degradation in the quality of the answer.
8. Ambigous user input: The input text might be misunderstood by the encoder thus providing the wrong embeddings to the decoder.

‍How to reduce Hallucination in agentic AI

Luckily for us there exists many possible strategies to deal with hallucination problems. Broadly speaking they can belong to two categories: strategies that prevent the hallucination from happening in the first place, and strategies that catch them once generated.
Here we will focus on some strategies of the former type.

External knowledge base

The most straightforward way to deal with hallucinations is to feed the agent with a clear, unambiguous prompt that is tailored to the user needs.
But what can we do while developing our agents? We can preprocess user requests to ensure clarity and filter out unclear, unwanted or dangerous queries (such as ones asking how to build a bomb).
Then, it is important to note that every LLM has a knowledge cut-off date and cannot know things after this date (but might confidently answer nonetheless). Besides mitigating the problem by using updated LLMs, we can solve it by providing the model tools to access an external knowledge base and/or communicate with APIs giving access to real-time data. Sometimes it might be important for the user to get outputs based on source knowledge rather than world knowledge. We can enforce more adherence to the information provided by strengthening our prompt. Let's see an example:

‍

By asking the model directly if humans live on Mars, the information we provided—“Since 2025 humans live on Mars”—is completely disregarded (parametric knowledge bias). However, with the second, stronger prompt we get what we wanted. 

Great. Now, what happens when we do not provide the context? The model will rely only on its knowledge that might be outdated or miss the information we need. To attach the relevant context to each query, we can use an external knowledge base with a retrieval mechanism. The latter is very important since it allows the model to find the relevant information based on the user prompt. This is known as retrieval augmentation. 

The main idea is to store our knowledge in a database and then provide the model with the entries that are closely related to the user query.

Let's see how this can be done in practice.

First, we need to identify a reliable and updated source of knowledge (e.g., a collection of scientific papers). Then we convert it into a vector representation. In other words, each piece of text will be converted into a collection of numbers giving its coordinates in a very high-dimensional space. The peculiarity of this vector space is that texts with similar semantic meanings will be close to each other. Once our vector space is populated by all the external knowledge base, we can use it to provide the model with the right source knowledge (also known as contexts). In order to do this, we convert our query to a vector and position it in the vector space. Then, we find the N closest points to the query, convert them into texts and feed them to the LLM alongside the user query.

Discourage guessing

LLMs are prone to guessing while often not knowing their limitations. For example they hate admitting when they are wrong or do not know something.
This can happen because they miss the needed information or because they are unable to do something (like solving math problems).
Here is where is important to provide the model with the right tools (to retrieve information from an external knowledge base as we saw or to solve math problem).
Another step to ensure the answer is appropriate is by using system prompts to encourage using the proper tools when needed, and admitting to not knowing something. An example could be the following:

"You are an AI assistant. If you do not know an answer, lack sufficient information, or cannot confidently respond, you must clearly admit this. Avoid making assumptions or guesses. Use external tools for fresh, niche, or location-based information, and for complex calculations. If you choose not to use a tool, ensure you have a clear reason. Always prioritize honesty and clarity."

Avoid logical breakdowns

I have 16 balls. 8 tennis balls and 8 golf balls. Half of the tennis ball are blue. How many blue tennis ball do I have?

Wrong intuitive answer:
8

What we want:
Here's how to solve the problem:

• Golf balls: You have 16 total balls - 8 tennis balls = 8 golf balls.
• Blue tennis balls: Half of the tennis balls are blue, so you have 8 tennis balls / 2 = 4 blue tennis balls.

Answer: You have 4 blue tennis balls.

‍

One of the most problematic features of hallucinations is that they seem plausible. It is like when asked a question for which the intuitive first answer is wrong. This can lead to a logical breakdown that leads to agents making poor decisions or fail at multi-step tasks.

How do we lead the agent to answering the question like in the example? We can use prompting techniques such as the Thought-Action-Observation cycle and the Re-Act approach. By using both we can shape the way the agent thinks, and obtain more reliable answers.

First, the agent is instructed—via the system prompt—to work iteratively until the solution is found by using a Thought-Action-Observation approach. This problem-solving cycle is composed of three phases:

1. Thought (planning): The agent reasons about the task and decides on an action.
2. Action (Execution): The agent executes the action using a tool.
3. Observation (Feedback): The agent reflects on the outcome and adapts its strategy if necessary. The agent reflects on its answer, identifies any discrepancies between the expected outcome and the actual outcome, and generates a new thought based on this feedback. The cycle is repeated until the problem is solved or the task completed, then the final answer is provided to the user.

This cycle ensures that agents can tackle complex tasks while learning from their interactions. A new cycle commences when the user provides further input. This way the user feedback is integrated in the thought process and the LLM can dynamically adapt to the new situation.

A simple extension of this mechanism is the Stop and Parse approach, where the LLM is explicitly instructed to stop after each action and parse the results before proceeding to the next step. This adds an extra layer of control and prevents the LLM from making assumptions based on incomplete information.

We can further improve the performance of the system by adopting the Re-Act (Reasoning + Action) approach that simply instruct the model to think “step-by-step”. In this way the agent is encouraged to generate a plan, rather than a final solution. In other words, a complex problem will be decomposed in smaller, more manageable, sub-tasks. Each step will then be considered in more detail and the agent will be less prone to make mistakes.

Evaluating your AI agents to detect hallucinations

If we want to keep AI agents from going off the rails, we need to work on AI safety and reliability. That’s where we come in. At Giskard, we provide specialized solutions for testing and evaluating AI systems. We help companies with:

• Automated and customizable testing frameworks tailored for LLM agents to identify potential sources of hallucinations
• Continuous red teaming solutions that ensure consistent performance across diverse business use cases

We apply a thoughtful approach to AI agent evaluation by leveraging the resources used by the agents themselves to generate targeted test cases. These adversarial tests are designed to assess both the agent’s outputs and its reasoning process. For instance, we create test cases that reveal issues like incorrect tool selection or improper input parameters, contributing to more robust and reliable AI systems.

By relying on our expertise and tools, businesses can significantly enhance the reliability and trustworthiness of their AI agents, ensuring they deliver accurate and valuable information to users.

Conclusion

Agentic AI represents a paradigm shift in how businesses interact with technology. By leveraging LLM-driven architectures, these systems can autonomously reason, plan, and execute tasks across various domains—from customer support automation to dynamic content generation. However, challenges such as tool usage errors, reasoning failures, and hallucinations must be addressed proactively.

For businesses looking to integrate agentic AI into their operations:

1. Adopt robust frameworks that emphasize continuous improvement.
2. Partner with experts like Giskard for advanced testing and monitoring solutions.

With the right strategies in place, companies can unlock the full potential of this transformative technology while minimizing risks—paving the way for smarter automation and enhanced operational efficiency in today’s competitive landscape.

For more information about Giskard’s solutions for building reliable agentic AI systems, you can reach out to us!