Run detail

06549906899b1593

Dataset

beir-v1.0.0-dbpedia-entity

Method

mugi

Model

gpt-4.1-nano

Retriever

BM25 (lexical)

params_hash

293d5e12

Queries

400

Metrics

ndcg_cut_10	0.4085
recall_100	0.5161

Reproduce this run

Three steps: (1) reformulate the queries with QueryGym's example pipeline, (2) run retrieval with Pyserini, (3) evaluate with trec_eval.

1. reformulate

python examples/querygym_pyserini/pipeline.py \
    --dataset beir-v1.0.0-dbpedia-entity \
    --method mugi \
    --model openai/gpt-4.1-nano \
    --steps reformulate \
    --temperature 1 \
    --max-tokens 128 \
    --method-params '{"num_examples":4,"train_split":"train"}' \
    --output-dir outputs/reproduce

2. retrieve (BM25)

python -m pyserini.search.lucene \
  --threads 16 --batch-size 128 \
  --index beir-v1.0.0-dbpedia-entity.flat \
  --topics outputs/reproduce/queries/reformulated_queries.tsv \
  --bm25 --k1 0.9 --b 0.4 \
  --output run.txt \
  --hits 1000

3. evaluate

python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \
  beir-v1.0.0-dbpedia-entity-test run.txt

Artifacts

Config

config.json

{
  "method_params": {
    "num_examples": 4,
    "train_split": "train"
  },
  "llm_config": {
    "temperature": 1,
    "max_tokens": 128
  },
  "dataset_config": {
    "topics": "beir-v1.0.0-dbpedia-entity-test",
    "index": "beir-v1.0.0-dbpedia-entity.flat",
    "num_queries": 400
  },
  "retrieval": {
    "retriever_id": "bm25",
    "paradigm": "lexical",
    "params": {
      "k1": 0.9,
      "b": 0.4
    },
    "implementation": "pyserini:LuceneSearcher"
  }
}