AI Skills Library

GenAI Hub Prompt Evaluation

Run GenAI evaluations on SAP AI Core: benchmark LLMs and prompts against a test dataset using system-defined and custom (LLM-as-judge) metrics, then retrieve aggregate results. Supports all three evaluation types — prompt registry, orchestration registry, and grounding (RAG) orchestration registry. WHEN: user wants to evaluate or benchmark an LLM / prompt / orchestration config, run a GenAI evaluation, create or list evaluation metrics, define a custom LLM-as-a-judge metric, create a prompt template (prompt registry) or an orchestration-registry config (with or without grounding) for evaluation, create an evaluation configuration or execution, or fetch evaluation results / scores for an execution. DO NOT USE FOR: agent evaluation via the offline aeval service — use `aeval-offline-evaluation`; building a production orchestration/RAG pipeline for inference — use `genai-hub-orchestration`; generic executions, deployments, or artifacts unrelated to evaluation — use `aicore-lifecycle-management`; credentials, resource groups, or object stores — use `aicore-admin-resources`.

Benchmark LLMs and prompts with GenAI evaluations on SAP AI Core

Details

AuthorSAP

Categoryplatform

Classification

PlatformSAP

Trust Level

SAP Certified

Tags

sap-ai-coreevaluationbenchmarkingllm-as-judgegenai-hub

RepositorySAP/ai-skills-library

LicenseApache-2.0

Checks

Installation

Add to Joule Work Desktop

Add to Joule Work WebComing soon

Install via CLI

npx skills add SAP/ai-skills-library --skill genai-hub-prompt-evaluation

Part of plugin

AI Core Skills

SAPSAP

Skills for SAP AI Core.

Plugin6 skills

Similar Skills

AI Core Admin Resources

SAPSAP

Verify and configure SAP AI Core credentials, retrieve auth tokens, and manage resource groups. Runs a setup health check, resolves authentication errors, and lists, creates, updates, or deletes resource groups. Use when checking whether AI Core is set up, configuring credentials, getting a Bearer token, or managing resource groups.

SkillPlatform

AI Core Lifecycle Management

SAPSAP

Manage the full SAP AI Core deployment lifecycle: scenarios, executables, configurations, and deployments. Lists and inspects scenarios and executables, creates configurations, creates, lists, stops, deletes, or patches deployments, fetches deployment logs, and removes duplicate LLM deployments. Use when managing AI Core deployments and their configurations.

SkillPlatform

GenAI Hub Foundation Models

SAPSAP

List and explore foundation models available in the SAP AI Core Generative AI Hub: browse providers, model names, versions, capabilities, and executable IDs, and get curl inference examples for deployed models. Use when discovering available models, finding an executable-id or model-name for deployment, or generating inference examples such as chat completions, embeddings, or image generation.

SkillPlatform

GenAI Hub Orchestration

SAPSAP

Configure and use the SAP AI Core Orchestration service: build prompt templates, add content filtering, set up document grounding (RAG) pipelines, and call models via the Harmonized API. WHEN: user asks about the orchestration service, Harmonized API, prompt templating with variables, content filtering, grounding, RAG pipeline, document grounding, or building a pipeline that wraps an LLM. DO NOT USE FOR: creating an orchestration deployment — use `aicore-lifecycle-management` for that; credential setup — use `aicore-admin-resources`; browsing or deploying foundation models directly without a pipeline — use `genai-hub-foundation-models`; direct LLM inference without a pipeline — use `aicore-sdk-java` or `aicore-sdk-js`.

SkillPlatform

Link Disclaimer

Link Disclaimer

GenAI Hub Prompt Evaluation

Link Disclaimer

Link Disclaimer

How to add to Joule Work Web

Link Disclaimer

Checks — GenAI Hub Prompt Evaluation — SkillSpector