Conversational LLM Evaluations in Minutes with NVIDIA NeMo Evaluator Agent Skills

Built on the NVIDIA NeMo Evaluator library, it allows developers to configure, run, and monitor evaluations directly inside Cursor, or another preferred agentic development tool. All through interaction with the agent and never manually creating YAML files or shell commands.

The Problem: Configuration Overhead

Running a single LLM evaluation means making dozens of interconnected decisions:

• Execution: Local Docker or SLURM cluster?
• Deployment: vLLM, SGLang, NVIDIA NIM, NVIDIA TensorRT-LLM, or external endpoint? What number of nodes?
• Model: What temperature? What context length? Does it use reasoning tokens?
• Benchmarks: Tau2-Bench, MTEB, GSM8K, AIME, GPQA, LiveCodeBench, RULER, more? All the above?
• Export: Local files, CSV, Weights & Biases, or MLflow?

Each selection spawns sub-choices. Using vLLM? Configure tensor parallelism. Running reasoning models? Parse pondering tokens. Multi-node SLURM? Arrange HAProxy load balancing. The result’s an overhead of working through complex YAML configs which might be easy to get fallacious and hard to debug.

execution:
  backend: slurm
  params:
    account: ???  
    partition: ???
    time_limit: "24:00:00"
deployment:
  backend: vllm
  params:
    tensor_parallel_size: ???  
    max_model_len: ???  

model:
  sampling_params:
    temperature: ???  
    top_p: ???

This configuration burden turns an easy evaluation into unnecessary complexity.

The Solution: Agent Skills for LLM Evaluation

Agent skills have change into the usual technique to give agents deep domain expertise. A “skill” packages instructions, scripts, and resources into portable folders that any compatible agent can discover and use.

Within the coding assistants space, the ecosystem has exploded covering all the things from Git workflows to API integrations. The nel-assistant is an agent skill that transforms Cursor, Claude Code, Codex, and other agentic developer tools into an evaluation expert. As a substitute of manual YAML authoring, you will have a conversation. The skill researches model cards to seek out optimal temperatures, top_p values, and context lengths, and identifies the optimal tensor parallelism on your specific GPU setup.

One conversation. Zero manual YAML debugging. Production-ready config.

You: "Evaluate NVIDIA Nemotron-3-Nano-30B-A3B on standard benchmarks
      using vLLM locally. Export to Weights & Biases."

Agent: I'll enable you set that up. First, let me check your environment...
       ✓ NeMo Evaluator 26.01 detected
       I'll need 5 quick answers to construct your config:
       1. Execution: local (detected out of your request)
       2. Deployment: vllm (confirmed)
       3. Export: wandb (confirmed)
       4. Model type: chat or base? [chat]
       5. Benchmark category: standard (confirmed)

       Now let me research the model card...
       ✓ Found: temperature=0.6, top_p=0.95, context=128K
       ✓ Optimal TP=8 on your 2x H100 setup

       Generated config: Nemotron-3-Nano-30B-A3B.yaml
       Able to run!

How It Works: Three Phases

Phase 1: Configure

The skill starts by asking five targeted questions to ascertain context:

1. Execution environment: Local or SLURM?
2. Deployment backend: vLLM, SGLang, NVIDIA NIM, NVIDIA TensorRT-LLM, or external?
3. Export destination: None, MLflow, or Weights & Biases?
4. Model type: Base, chat, or reasoning?
5. Benchmark categories: Standard, code, math, safety, or multilingual?

From these answers, it calls:

nel skills build-config 
  --execution local 
  --deployment vllm 
  --model-type chat 
  --benchmarks standard

This deep-merges modular YAML templates into tested, schema-compliant fragments that compose into structurally valid configs and minimizes syntax errors. With the skill alongside, the agent never generates free-form YAML, eliminating syntax errors.

Next, the agent robotically analyzes the model card and applies optimal configuration parameters.

Give the agent a HuggingFace handle NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 or checkpoint path, and it uses WebSearch to extract:

• Sampling params: Temperature, top_p
• Hardware logic: Optimal TP/DP settings based in your GPU count
• Reasoning config: System prompts, payload modifiers (e.g., enable_thinking for o1-style models)
• Context length: Max model length for vLLM --max-model-len

Developers not need to go looking through model cards to seek out the appropriate settings. The agent reads the model details and applies the proper parameters robotically.

Without the skill, this often means jumping between Hugging Face, blog posts, and documentation. It takes time and breaks focus. With the skill, the setup happens in seconds.

Phase 2: Validate and Refine

The skill identifies the remaining ??? values within the YAML:

• SLURM details: Account names, partition names, deadlines
• Export URIs: WandB project names, MLflow tracking URIs
• API keys: Environment variables for deployments

You may interactively:

• Add/remove tasks: Browse nel ls tasks and pick exactly what you would like
• Override per-task settings: “Use temperature=0 for HumanEval but 0.7 for MMLU”
• Configure advanced scaling: For >120B models, arrange data-parallel multi-node with HAProxy load balancing
• Add reasoning interceptors: Strip tokens, cache reasoning traces

Phase 3: Run and Monitor

The agent proposes a three-tier staged rollout: Dry run, Smoke test, and Full run.

nel run --config nemotron-3-nano.yaml --dry-run


nel run --config nemotron-3-nano.yaml 
  -o ++evaluation.nemo_evaluator_config.config.params.limit_samples=10


nel run --config nemotron-3-nano.yaml

Once submitted, progress will be monitored directly in Cursor using commands for status, detailed metrics, and live logs. You never leave your coding environment!

> Please, check the evaluation progress.

# Agent runs: nel status nemotron-3-nano-20260212-143022 && nel info ...

Status: RUNNING
Progress: 3/8 tasks accomplished
- ✓ mmlu: 65.2% accuracy (5 hours)
- ✓ hellaswag: 78.4% accuracy (2 hours)
- ✓ arc_challenge: 53.8% accuracy (1 hour)
- ⏳ truthfulqa_mc2: 45% complete...
- ⏳ winogrande: In queue
- ⏳ gsm8k: In queue
- ⏳ humaneval: In queue
- ⏳ mbpp: In queue

Technical Details

Template-Based Generation

As a substitute of generating YAML from scratch, nel-assistant merges modular templates for execution, deployment, benchmarks, and exports. This deep merge ensures structural validity.

Model Card Extraction Pipeline

1. Cursor or your agentic IDE fetches the HuggingFace model card via web search.
2. Extraction via regex identifies parameters and chat templates.
3. Hardware logic calculates optimal TP/DP based on model size and available GPU memory.
4. Reasoning detection checks for keywords like “reasoning” or “chain-of-thought.”
5. Values are injected directly into the config YAML.

Generic LLMs hallucinate YAML syntax. They mix incompatible backends. They create flags that do not exist.

As a substitute of generating YAML from scratch, nel skills build-config merges modular templates:

templates/
├── execution/
│   ├── local.yaml          # Docker execution
│   └── slurm.yaml          # SLURM execution
├── deployment/
│   ├── vllm.yaml           # vLLM backend
│   ├── sglang.yaml         # SGLang backend
│   └── nim.yaml            # NVIDIA NIM
├── benchmarks/
│   ├── reasoning.yaml      # GPQA-D, HellaSwag, SciCode, MATH, AIME
│   └── agentic.yaml        # TerminalBench, SWE-Bench
│   ├── longcontext.yaml    # AA-LCR, RULER
│   ├── instruction.yaml    # IFBench, ArenaHard
│   ├── multi-lingual.yaml  # MMLU-ProX, WMT24++
└── export/
    ├── wandb.yaml          # W&B integration
    └── mlflow.yaml         # MLflow integration

Deep merge = structural validity. You may’t produce invalid YAML once you’re composing pre-validated fragments.

The nel-assistant uses build-config to merge tested templates. Every config is structurally valid by construction. The agent composes YAML like a type-safe compiler, not a text generator.

Configuration Should Not Be a Bottleneck

LLM evaluation already involves essential decisions — choosing benchmarks, interpreting results, and comparing models. Configuration should support that process, not slow it down.

The nel-assistant skill makes it invisible. You describe what you would like in natural language, and the agent handles the remainder: researching model cards, generating configs, validating setups, staging rollouts, and monitoring progress.

No more 200-line YAML files. No more hunting through documentation. No more syntax errors.

Just: “Evaluate this model on these benchmarks.”

Resources

The nel-assistant skill is open-source and ships with NVIDIA NeMo Evaluator 26.01+.
Contributions welcome on GitHub!