Agents

Overview

Enterprise h2oGPTe agents enhance the functionality and versatility of Enterprise h2oGPTe to execute a broader range of tasks autonomously. In other words, this setting allows the large language model (LLM) to perform actions such as running code, generating plots, searching the web, conducting research, developing and preparing models, and more.

Enterprise h2oGPTe agents are equipped with a diverse suite of tools and features designed to optimize workflows, enhance productivity, and simplify complex tasks.

note

To learn how to use agents for dataset analysis and model development and preparation, consider the following tutorials:

• Tutorial 5: Dataset analysis with Enterprise h2oGPTe agents
• Tutorial 6: Model development and preparation with Enterprise h2oGPTe agents

Tool requirements

Enterprise h2oGPTe includes multiple agent tools, each with specific requirements. Basic agent functionality is enabled by default, but many tools require additional configuration.

• Check the requirements for each tool you intend to use
• Configure the necessary environment variables in your deployment
• Ensure any required external services or credentials are set up properly
• Verify that hardware requirements (such as GPU for image generation) are met
• Replace all <your-*-key> placeholders with actual keys/tokens

This basic agent configuration enables several tools by default:

ALLOW_API: "1"
H2OGPT_OPENAI_LOG_PATH: /workspace/save/h2ogpt_openai/
TRANSFORMERS_CACHE: /workspace/.cache

Instructions

To use agents in Enterprise h2oGPTe (enable an agent in a chat session) and configure the type of tools the agent can use, consider the following steps:

1. In Enteprise h2oGPTe, click + New chat.
2. Click Use agent.
3. Click settings Customize.
4. Click the Configuration tab.
5. By default, in the Tools setting (under the Use agent setting), all available tools for an agent are presented.

note

To learn about each available tool for an agent, see Tools.

Tools

Data analysis capabilities

1. H2O Driverless AI Data Science: Leverage AI capabilities for automated machine learning, data analysis, and model generation to extract actionable insights from complex datasets.

   Requirements:

   For DAI Cloud:

   ENABLE_DAI: "1"
   DAI_INSTANCE_TYPE: cloud
   DAI_CLIENT_ID: hac-platform-public
   DAI_ENGINE: <your-dai-engine>
   DAI_ENVIRONMENT: <your-dai-environment-url>
   DAI_TOKEN: <your-dai-token>
   DAI_TOKEN_ENDPOINT_URL: <your-token-endpoint-url>

   For DAI on-premises:

   ENABLE_DAI: "1"
   DAI_INSTANCE_TYPE: onprem
   DAI_ADDRESS: <your-dai-address>
   DAI_USERNAME: <your-username>
   DAI_PASSWORD: <your-password>

2. Python Coding: Execute Python code snippets for custom data computations, machine learning tasks, or to automate data preprocessing.

   Requirements:

   • Enabled with basic agent configuration:

     ALLOW_API: "1"

3. Advanced Reasoning: Perform context-aware reasoning to enhance decision-making processes.

   Requirements:

   • Requires models that support reasoning capabilities.

   • Configure in Helm chart:

     agent:
     enabled: true
     overrideConfig:
     visible_reasoning_models:
     - meta-llama/Llama-3.1-8B-Instruct

Information retrieval and research

4. Scholar Papers Search: Locate scholarly papers, research articles, and academic content.

   Requirements:

   • Requires the Internet Access tool to be enabled.

5. Google search: Access Google's search capabilities to gather timely and accurate information.

   Requirements:

   • Requires the Internet Access tool to be enabled.
   • Optional:
     • Configure the Google API key:

       GOOGLE_API_KEY: <your-google-api-key>

6. Bing search: Access Bing's search capabilities to gather timely and accurate information.

   Requirements:

   • Requires the Internet Access tool to be enabled.
   • Optional:
     • Configure the Bing API key:

       BING_API_KEY: <your-bing-api-key>

7. Internet Access: Utilize general internet access for a wide range of tasks, from data retrieval to live updates.

   Requirements:

   • Basic agent configuration:

     ALLOW_API: "1"

8. Wolfram|Alpha Math Science Search: Solve scientific and mathematical problems using the computational knowledge engine for complex computations.

   Requirements:

   • Requires the Internet Access tool to be enabled.
   • Optional:
     • Configure the Wolfram|Alpha API key:

       WOLFRAM_ALPHA_APPID: <your-wolfram-alpha-appid>

9. Wikipedia Articles Search: Retrieve information from Wikipedia for learning or to support data analysis and research.

   Requirements:

   • Requires the Internet Access tool to be enabled.

10. Wayback Machine Search: Access archived versions of websites for research, historical context, or to reference past web content.

    Requirements:

    • Requires the Internet Access tool to be enabled.

11. Web Image Search: Find relevant images from the web to support visual analysis, presentations, or content creation.

    Requirements:

    • Requires the Internet Access tool to be enabled.

12. Ask Question About Documents: Extract or query information from uploaded documents for detailed insights.

    Requirements:

    • Basic agent configuration:

      ALLOW_API: "1"

Content generation and transformation

13. Image Generation: Create custom images based on textual descriptions. Integrate image generation capabilities from different providers, including local models, OpenAI, Azure OpenAI, and Black Forest Labs.

    Requirements:

    • Requires GPU in agent pod.
    • Shell Scripting tool must be enabled as a prerequisite.
    • Configuration in Helm chart:

      h2ogpt:
      config:
          externalLLM:
          enabled: true
          secret:
              ALLOW_API: "1"
              
              # Black Forest Labs Configuration
              BFL_API_KEY: <your-black-forest-labs-api-key>
              
              # OpenAI Configuration
              IMAGEGEN_OPENAI_BASE_URL: https://api.openai.com/v1
              IMAGEGEN_OPENAI_API_KEY: <your-openai-api-key>
              
              # Azure OpenAI Configuration
              AZURE_OPENAI_BASE_URL: openai.azure.com
              AZURE_OPENAI_API_KEY: <your-azure-openai-api-key>
              
          
              agent:
              enabled: true
              additionalConfig:
                  function_api_key: <your-function-api-key>
                  enforce_h2ogpt_api_key: false
                  enforce_h2ogpt_ui_key: false
                  enable_image: true
                  visible_image_models:
                  - "flux.1-schnell"  # Local model for image generation
      
                  # Optional: Additional image generation models
                  # Uncomment and replace with actual model names when using external providers
                  # - "openai-model"  # OpenAI model
                  # - "azure-model"   # Azure model
      

14. Audio-Video Transcription: Convert spoken content from audio or video files into text.

    Requirements:

    • Basic agent configuration:

      ALLOW_API: "1"

    • Optional:

      • Configure the OpenAI base URL and API key for STT:

        STT_OPENAI_BASE_URL: <your-stt-openai-base-url>
        STT_OPENAI_API_KEY: <your-stt-openai-api-key>
        
        # Additional Configuration:
        asr_model: "distil-whisper/distil-large-v3"  # Model for converting audio/video transcription
        enable_stt: true  # Enable speech-to-text for audio/video transcription
        note

        Ensure you have selected Agents Only when uploading.

15. Convert Document to Text: Transform scanned or digital documents into editable text formats.

    Requirements:

    • Basic agent configuration

      ALLOW_API: "1"

16. Screenshot Webpage: Capture screenshots of webpages.

    Requirements:

    • Requires the Internet Access tool to be enabled.

17. Mermaid Chart-Diagram Renderer: Generate flowcharts and diagrams.

    Requirements:

    • Enabled with the default configuration.

Automation and scripting

18. Shell Scripting: Automate system-level tasks to streamline operations.

    Requirements:

    • Basic agent configuration:

      ALLOW_API: "1"

Specialized AI features

19. RAG Text: Enhance text-based tasks with retrieval-augmented generation.

    Requirements:

    • Uses default models in ModelLock.
    • Basic agent configuration:

      ALLOW_API: "1"

20. RAG Vision: Apply retrieval-augmented generation for vision-based tasks.

    Requirements:

    • Uses default vision models in ModelLock.

Multimedia processing

21. Download Web Video: Download web videos.

    Requirements:

    • Requires the Internet Access tool to be enabled.
    • Shell Scripting tool must be enabled as a prerequisite.

22. Ask Question About Image: Query and analyze visual data.

    Requirements:

    • Requires a vision model configured in ModelLock.

Software Development Tools

23. Aider Code Generation: Automatically generate code snippets to accelerate development tasks, particularly for data science and machine learning models.

    Requirements:

    • Models are fetched from ModelLock.
    • No additional configuration required.

24. GitHub: Integrates with the GitHub platform for version control, code repository browsing, and collaboration.

    Requirements:

    • Requires configuring GITHUB_TOKEN as a secret:

      GITHUB_TOKEN: <your-github-token>

25. Evaluate Answer: Provides automated evaluation of responses generated by agents.

    Requirements:

    • Basic agent configuration:

      ALLOW_API: "1"

26. SWEBench Evaluate Patch: Evaluates software patches by running automated tests, quality checks, and performance benchmarks.

    Requirements:

    • Basic agent configuration:

      ALLOW_API: "1"

note

To use tools that require internet access, such as GitHub, Wikipedia Articles Search, Web Image Search, etc., ensure that the Internet Access tool is selected.

---

Feedback

• Submit and view feedback for this page
• Send feedback about Enterprise h2oGPTe to cloud-feedback@h2o.ai