iclbench.agents package

Submodules

iclbench.agents.base module

class iclbench.agents.base.BaseAgent(client_factory, prompt_builder)

Bases: object

Base class for all agents in the ICLBench framework.

The BaseAgent class serves as a foundational class for creating agents that interact with environments using a client for language model interactions and a prompt builder to manage context. It provides methods for performing actions based on observations, updating the prompt with observations and actions, and resetting the prompt builder.

Variables:

• client (Client) – An instance of the client created by the client factory.

• prompt_builder (PromptBuilder) – An instance of the prompt builder for managing prompt context and history.

__init__(client_factory, prompt_builder)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

reset()

Resets the prompt builder to its initial state.

This method clears the prompt history and any other stored information in the prompt builder, allowing for a fresh start in a new episode or task.

update_prompt(observation, action)

Updates the prompt builder with the latest observation and action.

This method incorporates the current observation and the action taken into the prompt history, allowing for context to be maintained across interactions.

Parameters:

• observation (Observation) – The current observation from the environment.

• action (Action) – The action taken by the agent.

iclbench.agents.chain_of_thought module

class iclbench.agents.chain_of_thought.ChainOfThoughtAgent(client_factory: LLMClientWrapper, prompt_builder, config)

Bases: BaseAgent

__init__(client_factory: LLMClientWrapper, prompt_builder, config)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs, prev_action=None)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

iclbench.agents.dummy module

class iclbench.agents.dummy.DummyAgent(client_factory, prompt_builder)

Bases: BaseAgent

For debugging.

__init__(client_factory, prompt_builder)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs, prev_action=None)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

class iclbench.agents.dummy.LLMResponse(model_id, completion, stop_reason, input_tokens, output_tokens, reasoning)

Bases: tuple

completion

Alias for field number 1

input_tokens

Alias for field number 3

model_id

Alias for field number 0

output_tokens

Alias for field number 4

reasoning

Alias for field number 5

stop_reason

Alias for field number 2

iclbench.agents.dummy.make_dummy_action(text)

iclbench.agents.icl module

class iclbench.agents.icl.ICLAgent(client_factory, prompt_builder)

Bases: BaseAgent

__init__(client_factory, prompt_builder)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs, prev_action=None)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

cache_icl()

get_icl_prompt() → List[Message]

update_icl_action(action: str)

update_icl_observation(obs: dict)

wrap_episode()

class iclbench.agents.icl.Message(role: str, content: str, attachment: object | None = None)

Bases: object

__init__(role: str, content: str, attachment: object | None = None)

iclbench.agents.naive module

class iclbench.agents.naive.NaiveAgent(client_factory, prompt_builder)

Bases: BaseAgent

__init__(client_factory, prompt_builder)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs, prev_action=None)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

iclbench.agents.self_refine module

class iclbench.agents.self_refine.SelfRefineAgent(client_factory: LLMClientWrapper, prompt_builder, max_iterations=3)

Bases: BaseAgent

__init__(client_factory: LLMClientWrapper, prompt_builder, max_iterations=3)

Initializes the BaseAgent with a client factory and a prompt builder.

Parameters:

• client_factory (Callable) – A factory function to create a client instance.

• prompt_builder (PromptBuilder) – An instance of the prompt builder.

act(obs, prev_action=None)

Abstract method for performing an action based on the given observation.

This method must be implemented by subclasses, as the action logic will depend on the specific agent type. For instance, derived agents may store action-observation histories when acting.

Parameters:

obs (Observation) – The current observation from the environment.

Module contents

class iclbench.agents.AgentFactory(config)

Bases: object

A factory class for creating various types of agents based on the provided configuration.

The AgentFactory class encapsulates the logic for creating agents used in the ICLBench framework. It generates agents of different types, each tailored for specific tasks and functionality.

Variables:

config (Config) – The configuration object containing settings for agent creation, including agent type and client configuration.

Parameters:

config (Config) –

The configuration settings for the agent, which must include: - agent.type (str): The type of agent to create. Supported types include:

• ”naive”

• ”icl”

• ”cot”

• ”self_refine”

• ”dummy”

• client (dict): The client configuration for the agent.

create_agent()

Creates and returns an instance of the specified agent type.

__init__(config)

create_agent()

Creates an instance of the agent based on the configuration settings.

Returns:

An instance of the specified agent type, which could be:

• NaiveAgent

• ICLAgent

• ChainOfThoughtAgent

• SelfRefineAgent

• DummyAgent

Return type:

Agent

Raises:

ValueError – If the specified agent type in the configuration is not recognized.