iclbench package

Subpackages

• iclbench.agents package
  • Submodules
  • iclbench.agents.base module
    • BaseAgent
      • BaseAgent.__init__()
      • BaseAgent.act()
      • BaseAgent.reset()
      • BaseAgent.update_prompt()
  • iclbench.agents.chain_of_thought module
    • ChainOfThoughtAgent
      • ChainOfThoughtAgent.__init__()
      • ChainOfThoughtAgent.act()
  • iclbench.agents.dummy module
    • DummyAgent
      • DummyAgent.__init__()
      • DummyAgent.act()
    • LLMResponse
      • LLMResponse.completion
      • LLMResponse.input_tokens
      • LLMResponse.model_id
      • LLMResponse.output_tokens
      • LLMResponse.reasoning
      • LLMResponse.stop_reason
    • make_dummy_action()
  • iclbench.agents.icl module
    • ICLAgent
      • ICLAgent.__init__()
      • ICLAgent.act()
      • ICLAgent.cache_icl()
      • ICLAgent.get_icl_prompt()
      • ICLAgent.update_icl_action()
      • ICLAgent.update_icl_observation()
      • ICLAgent.wrap_episode()
    • Message
      • Message.__init__()
  • iclbench.agents.naive module
    • NaiveAgent
      • NaiveAgent.__init__()
      • NaiveAgent.act()
  • iclbench.agents.self_refine module
    • SelfRefineAgent
      • SelfRefineAgent.__init__()
      • SelfRefineAgent.act()
  • Module contents
    • AgentFactory
      • AgentFactory.create_agent()
      • AgentFactory.__init__()
      • AgentFactory.create_agent()
• iclbench.environments package
  • Subpackages
    • iclbench.environments.baba_is_ai package
      • Submodules
      • iclbench.environments.baba_is_ai.base module
      • Module contents
    • iclbench.environments.babyai_text package
      • Submodules
      • iclbench.environments.babyai_text.clean_lang_wrapper module
      • Module contents
    • iclbench.environments.craftax package
      • Submodules
      • iclbench.environments.craftax.env module
      • Module contents
    • iclbench.environments.crafter package
      • Submodules
      • iclbench.environments.crafter.env module
      • Module contents
    • iclbench.environments.minihack package
      • Module contents
    • iclbench.environments.nle package
      • Submodules
      • iclbench.environments.nle.base module
      • iclbench.environments.nle.progress module
      • iclbench.environments.nle.render module
      • iclbench.environments.nle.render_rgb module
      • Module contents
    • iclbench.environments.textworld package
      • Submodules
      • iclbench.environments.textworld.base module
      • Module contents
  • Submodules
  • iclbench.environments.env_wrapper module
    • EnvWrapper
      • EnvWrapper.__init__()
      • EnvWrapper.actions
      • EnvWrapper.check_action_validity()
      • EnvWrapper.get_instruction_prompt()
      • EnvWrapper.get_stats()
      • EnvWrapper.get_text_action()
      • EnvWrapper.max_steps
      • EnvWrapper.reset()
      • EnvWrapper.step()
  • Module contents
    • Strings
      • Strings.__init__()
      • Strings.contains()
      • Strings.map()
      • Strings.sample()
    • make_env()
• iclbench.prompt_builder package
  • Module contents
    • create_prompt_builder()

Submodules

iclbench.client module

class iclbench.client.ClaudeWrapper(client_config)

Bases: LLMClientWrapper

__init__(client_config)

convert_messages(messages)

generate(messages)

class iclbench.client.GoogleGenerativeAIWrapper(client_config)

Bases: LLMClientWrapper

__init__(client_config)

cache_icl_demo(icl_demo)

convert_messages(messages)

extract_completion(response)

Extracts and returns the completion from the response safely.

generate(messages)

get_completion(converted_messages, max_retries=5, delay=5)

class iclbench.client.LLMClientWrapper(client_config)

Bases: object

__init__(client_config)

generate(messages)

class iclbench.client.LLMResponse(model_id, completion, stop_reason, input_tokens, output_tokens, reasoning)

Bases: tuple

completion

Alias for field number 1

input_tokens

Alias for field number 3

model_id

Alias for field number 0

output_tokens

Alias for field number 4

reasoning

Alias for field number 5

stop_reason

Alias for field number 2

class iclbench.client.OpenAIWrapper(client_config)

Bases: LLMClientWrapper

__init__(client_config)

convert_messages(messages)

generate(messages)

class iclbench.client.ReplicateWrapper(client_config)

Bases: LLMClientWrapper

__init__(client_config)

generate(messages)

iclbench.client.create_llm_client(client_config)

Factory function to create the appropriate LLM client based on the client name.

iclbench.client.process_image_claude(image)

iclbench.client.process_image_openai(image)

iclbench.dataset module

class iclbench.dataset.InContextDataset(config, env_name, original_cwd)

Bases: object

__init__(config, env_name, original_cwd) → None

check_seed(demo_path)

demo_path(i, task, demo_config)

demo_task(task)

icl_episodes(task)

load_incontext_actions(demo_path)

override_incontext_config(demo_config, demo_path)

iclbench.dataset.choice_excluding(lst, excluded_element)

iclbench.dataset.natural_sort_key(s)

iclbench.evaluator module

class iclbench.evaluator.Evaluator(env_name, config, original_cwd='')

Bases: object

Class to evaluate an agent on a set of tasks in a given environment.

The Evaluator class is responsible for orchestrating the evaluation of agents across multiple tasks within a specified environment. It manages the setup of the environment, runs episodes, logs results, and can execute evaluations in parallel or sequentially.

Variables:

• env_name (str) – Name of the environment in which the agent operates.

• config (Config) – Configuration object containing evaluation parameters.

• tasks (list) – List of tasks for the specified environment.

• num_episodes (int) – Number of episodes to run for each task.

• num_workers (int) – Number of parallel worker processes to use.

• max_steps_per_episode (int) – Maximum number of steps per episode.

• dataset (InContextDataset) – Dataset object for managing in-context learning tasks.

__init__(env_name, config, original_cwd='')

Initializes the Evaluator with environment name and configuration.

Parameters:

• env_name (str) – Name of the environment.

• config (Config) – Configuration object with evaluation parameters.

• original_cwd (str, optional) – Original current working directory. Defaults to “”.

load_in_context_learning_episode(i, task, agent, episode_log)

Loads and executes an in-context learning episode for the specified task.

Parameters:

• i (int) – Index of the in-context learning episode.

• task (str) – Name of the task to be evaluated.

• agent (BaseAgent) – The agent being evaluated.

• episode_log (dict) – Log to record episode results.

run(agent_factory)

Executes the evaluation process either sequentially or in parallel.

Parameters:

agent_factory (AgentFactory) – Factory to create instances of the agent.

Returns:

Summary of the results for all tasks.

Return type:

dict

run_episode(task, agent, process_num=None, position=0)

Executes a single evaluation episode for the specified task.

Parameters:

• task (str) – Name of the task to be evaluated.

• agent (BaseAgent) – The agent being evaluated.

• process_num (int, optional) – Process number for logging purposes. Defaults to None.

• position (int, optional) – Position for progress bar. Defaults to 0.

Returns:

Log of the episode results including trajectory, action frequency,

and performance metrics.

Return type:

dict

iclbench.utils module

iclbench.utils.load_secrets(file_path)

iclbench.utils.setup_environment(openai_tag: str = 'OPENAI_API_KEY', gemini_tag: str = 'GEMINI_API_KEY', anthropic_tag: str = 'ANTHROPIC_API_KEY', replicate_tag: str = 'REPLICATE_API_KEY', organization: str = None, original_cwd: str = '')

iclbench.utils.summarize_env_progressions(results_summaries: defaultdict, config) → float

iclbench.utils.wandb_save_artifact(config)

Module contents