mugi

All results produced by QueryGym · fully reproducible!

12 model × retriever configurations for this method across BEIR, MS MARCO DL, and DL-HARD.
Click any row or the + button to expand. Tabs switch dataset context. The three steps (reformulate → retrieve → evaluate) update accordingly.

	Model	Retriever	ArguAna		DBPedia		FiQA		SciFact		COVID		News		BRIGHT — AOPS		BRIGHT — Biology		BRIGHT — Earth Science		BRIGHT — Economics		BRIGHT — LeetCode		BRIGHT — Pony		BRIGHT — Psychology		BRIGHT — Robotics		BRIGHT — Stack Overflow		BRIGHT — Sustainable Living		BRIGHT — TheoremQA Questions		BRIGHT — TheoremQA Theorems		DL-HARD		DL 2019		DL 2020
	Model	Retriever	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100																									nDCG@10	R@1k	nDCG@10	R@1k	nDCG@10	R@1k
	Qwen2.5-72B-Instruct	BGE-base-en-v1.5	0.6194	0.9900	0.4342	0.5318	0.4192	0.7526	0.7453	0.9700	0.7972	0.1425	0.4732	0.5298	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3948	0.8548	0.7512	0.9071	0.7122	0.8894
methodmugi llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id c75e8a83747970d6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id df4b60507d728672 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 20ac8806eccf8139 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 6c9a9f1190dbd1f1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id bd29a2cdc356222a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 8060dace1f95b43e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d5c74c4a59b0f66f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 50f0e78bf714634c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id e78fee0e18c2bd8d · open full run detail →
	Qwen2.5-72B-Instruct	BM25	0.3868	—	0.4103	0.5296	0.2435	—	0.7203	—	0.6927	0.1694	0.5009	0.5921	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3609	0.8122	0.6911	0.9055	0.6268	0.9015
methodmugi llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 6444a5e04bcb9013 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 898543f1681a8169 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id 831cb39c41716188 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id fb35e4c85286196d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 74ad89762275ca4e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 20e68a265b0a0f15 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 74b697f39b901162 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id b8d5bfc95f0c4b50 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 2c11d2c0e20cc0cb · open full run detail →
	Qwen2.5-72B-Instruct	SPLADE++	0.5031	0.9787	0.3735	0.5044	0.3023	0.6787	0.6951	0.9493	0.6639	0.1105	0.4394	0.4972	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3260	0.8098	0.6746	0.9275	0.6419	0.9165
methodmugi llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id c5cd4658373a61c8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id bbed6bd9f822ca4b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 5328e402f08642b8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 9f31850bdb47fef0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id d5d4c66acd86484c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 6afab8749e67e7e2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 00665889c6151a55 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d5b50b564ccc29e8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 2b1068b944df9dec · open full run detail →
	Qwen2.5-7B-Instruct	BGE-base-en-v1.5	0.6213	0.9922	0.4106	0.5195	0.4130	0.7456	0.7449	0.9767	0.8071	0.1406	0.4648	0.5142	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3619	0.8495	0.6869	0.8781	0.6888	0.8823
methodmugi llmQwen2.5-7B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 8a090846f802a1b9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 7efc74d3d7b42740 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 32a6aaa4f19280f2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id fbae23c6b1d00855 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id ed4abc42a91d4fc6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id ba319ae357c14074 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 7087d436683edb25 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d808a5c2486f3fcb · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id c56c984559f3e98f · open full run detail →
	Qwen2.5-7B-Instruct	BM25	0.3926	0.9381	0.4006	0.5114	0.2368	0.5652	0.7063	0.9627	0.6771	0.1628	0.4436	0.5767	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3173	0.7707	0.6394	0.8732	0.6069	0.8882
methodmugi llmQwen2.5-7B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 546ff3396d14cb08 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 08d0f86de18246a2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 687e67dc791532ef · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 357679b380ca5c0f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 31d81c8575bfaaad · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 88d0e4a8540d28d0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 646e8a471d7c7a31 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 0bd45569e0452bd6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id ed52ae7274db797c · open full run detail →
	Qwen2.5-7B-Instruct	SPLADE++	0.5101	0.9787	0.3600	0.4989	0.2953	0.6597	0.6665	0.9593	0.6547	0.1045	0.4001	0.4725	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2642	0.8028	0.5773	0.8929	0.5527	0.9104
methodmugi llmQwen2.5-7B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 1543aaf2b16b0616 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 35d99cd527f8f873 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 9a3a2408466e6ceb · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 1a50c50e259858ab · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 266e1d1425d0563f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 8d15e8f1910cdbc3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id a3c8453a6b8584b7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id e0b6047be3d4daf2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 57db88c60c930ee5 · open full run detail →
	gpt-4.1	BGE-base-en-v1.5	0.6161	0.9900	0.4400	0.5286	0.4294	0.7584	0.7569	0.9767	0.8024	0.1427	0.4898	0.5212	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.4038	0.8415	0.7351	0.8869	0.7203	0.8950
methodmugi llmgpt-4.1 retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 00e843e8c925b1c4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id f0b48f8584d99b00 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 28b0d3f60e3a7872 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 3a295240e0c40c6d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 6d95b7c8744b2c3a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id b96202ef0dceb779 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 9cec0d52b2fd6b87 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d06376bffb72cbc3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id ff1652318730862e · open full run detail →
	gpt-4.1	BM25	0.3758	0.9331	0.4099	0.5309	0.2641	0.6000	0.7345	0.9660	0.7137	0.1739	0.5156	0.6075	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3651	0.8216	0.6952	0.9005	0.6578	0.8996
methodmugi llmgpt-4.1 retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id b74faca3501fd4b0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id a3f95b47cce80d6a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 18843a12545a6179 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 2665699a986b72c9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id bdaf765013c6dc4b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 2baa9c302fcda507 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 6f28d412e9569bdf · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 33511f71fb738c32 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id ee7db32fd7fd3657 · open full run detail →
	gpt-4.1	SPLADE++	0.3703	0.9780	0.3843	0.5137	0.3352	0.6799	0.7059	0.9600	0.6458	0.1118	0.4422	0.5002	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3625	0.8111	0.6859	0.9088	0.6508	0.9199
methodmugi llmgpt-4.1 retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id e3b7f0dd5a400a85 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id a263942c47cbce72 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 2a3dc4da10dec081 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 32603c5fb254483d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 35a2263adb13e22e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id bea518045d1f7b0c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d4800e9883ec51c5 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d072b6af4a8c0537 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id a8e3e639dca6e3a9 · open full run detail →
	gpt-4.1-nano	BGE-base-en-v1.5	0.6184	0.9900	0.4280	0.5284	0.4228	0.7488	0.7457	0.9800	0.7980	0.1425	0.4696	0.5081	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3903	0.8354	0.7169	0.8725	0.7187	0.8911
methodmugi llmgpt-4.1-nano retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 768879cca022b1be · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 3a89cdd2cfac7c41 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 703130ae374e9fb1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id e4cc536f2cf39cef · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id f86076f682a9ae1d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id e41fc6852ba0c7c3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id dc7e8f17e04a7d07 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 6ad6739bad26d317 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id f6edf4a73da7ba10 · open full run detail →
	gpt-4.1-nano	BM25	0.3831	0.9317	0.4085	0.5161	0.2517	0.5802	0.7318	0.9627	0.7062	0.1713	0.4707	0.5873	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3423	0.7924	0.6835	0.8915	0.6473	0.9017
methodmugi llmgpt-4.1-nano retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 358094095831f1c0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 06549906899b1593 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id be4876619f2ae349 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 3c130f631cbf4f91 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 8fe4f49c828ea9aa · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 8e2a04ccecab8874 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 242989f63e84fe11 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id f43e958c28e0e5af · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 89df7427128a5a65 · open full run detail →
	gpt-4.1-nano	SPLADE++	0.3718	0.9787	0.3843	0.5095	0.3171	0.6673	0.6900	0.9527	0.6317	0.1144	0.4072	0.4770	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3254	0.8105	0.6611	0.8904	0.6432	0.9203
methodmugi llmgpt-4.1-nano retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 5c480b8b1c025614 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id f3ef8c28a5c36e35 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 4788c1440c7ebced · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id c4f88c723d5a1b29 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 9ca313c0b69b3eea · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 01a43e6d888af2d7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 3b9d6b6773b91b59 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 9ed888a3da14d919 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 64efba2a45014195 · open full run detail →