Qwen2.5-72B-Instruct

All results produced by QueryGym · fully reproducible!

30 method × retriever configurations using this LLM across BEIR, MS MARCO DL, and DL-HARD.
Click any row or the + button to expand. Tabs switch dataset context. The three steps (reformulate → retrieve → evaluate) update accordingly.

	Method	Retriever	ArguAna		DBPedia		FiQA		SciFact		COVID		News		BRIGHT — AOPS		BRIGHT — Biology		BRIGHT — Earth Science		BRIGHT — Economics		BRIGHT — LeetCode		BRIGHT — Pony		BRIGHT — Psychology		BRIGHT — Robotics		BRIGHT — Stack Overflow		BRIGHT — Sustainable Living		BRIGHT — TheoremQA Questions		BRIGHT — TheoremQA Theorems		DL-HARD		DL 2019		DL 2020
	Method	Retriever	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100																									nDCG@10	R@1k	nDCG@10	R@1k	nDCG@10	R@1k
	csqe	BGE-base-en-v1.5	0.6229	0.9886	0.4024	0.4897	0.3796	0.7461	0.7484	0.9667	0.7793	0.1410	0.4626	0.4812	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3757	0.8531	0.7179	0.8944	0.6687	0.8722
methodcsqe llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id f5fd7777f20b623f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 94875e6bcad50a83 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 96a42cbd361934ea · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 5a337650d6aec8fa · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 5a7dfe39f3fa4606 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id eeb5490fdadd7d2a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 38259981fc0cc7c5 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 207f6f4e8b8b3331 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 90f0e7cdd5df8856 · open full run detail →
	csqe	BM25	0.3864	—	0.3556	0.4639	0.2132	—	0.7141	—	0.6716	0.1491	0.3861	0.4892	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2848	0.6998	0.6391	0.8608	0.5606	0.8603
methodcsqe llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 6e44c9a23591ec9a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 43e3c6b01609e6f0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id be5818c6d67a2fc3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 07f8082534b1b82c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id a04f3be704b59a99 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 7cc031ceec7f22cd · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 374d868e8c9d9b43 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 239707be9e6e9d0b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 4d261dda4cfb7038 · open full run detail →
	csqe	SPLADE++	0.5118	0.9787	0.3686	0.5021	0.3075	0.6521	0.6966	0.9433	0.6118	0.1082	0.3871	0.4548	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2857	0.8246	0.6189	0.9070	0.5736	0.9052
methodcsqe llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 43c9e25451d58e11 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 49c7af4b43926798 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 868b26500a878476 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id dfae7d6c06e355d5 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 92ba76bc29a9d68f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 979e9554baa56930 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id c106bbf40f6cf8e6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 8057160502ca9c91 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method csqe \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id f616925e2ae1ece5 · open full run detail →
	genqr	BGE-base-en-v1.5	0.6248	0.9900	0.3692	0.4808	0.3826	0.7139	0.7339	0.9650	0.7869	0.1416	0.4409	0.5023	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3471	0.8144	0.6741	0.8618	0.6680	0.8652
methodgenqr llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 7f7b4bf0249b08b2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 10131bc146f67ead · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 843f0fa29e39cb85 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id cb931bae333ab05f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id a17046bdd0256307 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 6f7d7fe639cf6c88 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id eb8d8dc5bb77143f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id f9733441573227b3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 3a97f2fc93e12240 · open full run detail →
	genqr	BM25	0.4188	—	0.2649	0.3941	0.1725	—	0.6976	—	0.6129	0.1349	0.4003	0.5838	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2091	0.6822	0.4198	0.7616	0.4238	0.7919
methodgenqr llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id e8335322dca5be04 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 9de683664800281a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id e7d00fc2eb056b9f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 72b99b02937d8708 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id e305bab14adcf270 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 93c6977e1b1e6a0d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 6a6303729251dfe7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id fe245d46cfc2b226 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 9bc21a52ef2dd34c · open full run detail →
	genqr	SPLADE++	0.5201	0.9815	0.3579	0.5275	0.2868	0.6217	0.7468	0.9413	0.6292	0.1055	0.3808	0.4754	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2916	0.7861	0.6154	0.9030	0.5751	0.8971
methodgenqr llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 148aa91670cef258 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 5fff72e0044be6ce · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id a206750947de6c91 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id b884275c4c2c7fd3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 9f27e2872fe1496f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 95e306b89d6cccd8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d52ede3b6a3b2d5e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 6c59db025fdb4940 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 1f8ed462c5bf6734 · open full run detail →
	genqr_ensemble	BGE-base-en-v1.5	0.6254	0.9893	0.3974	0.5309	0.3943	0.7284	0.7496	0.9700	0.7915	0.1407	0.4515	0.5136	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3543	0.8269	0.6819	0.8825	0.6774	0.8585
methodgenqr_ensemble llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 9082596a6d0158ac · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id c27e2ea65c02c76d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id bce8108df0344059 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 6b8be7f20c26644e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id f0552e3090132fb8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id ac2c0b3e4fff403d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 4a3e0c7bd93da38a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 24cf5c1f48edb902 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 6903db1e0c85f881 · open full run detail →
	genqr_ensemble	BM25	0.4080	—	0.3136	0.4161	0.2061	—	0.7089	—	0.6437	0.1451	0.4080	0.5923	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2463	0.6975	0.4739	0.7999	0.4248	0.7820
methodgenqr_ensemble llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 73a94ecc826455b3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id abd50579fccb6cff · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id 2c456eb00d9d2c89 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 59b04f1a88a63ec9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id ddab8adddb1ab66e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 42284a01f5419323 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d08cbd7ce239e045 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 5cd6ab2d47e8224a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 14bb031ca7ba3d45 · open full run detail →
	genqr_ensemble	SPLADE++	0.5193	0.9822	0.4271	0.5565	0.3062	0.6136	0.7135	0.9433	0.6162	0.1099	0.3963	0.5087	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.2849	0.7823	0.5979	0.9053	0.5447	0.8886
methodgenqr_ensemble llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id b5d7e1f37ccc765c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id b0770923fff53c8c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 1e905171b4723501 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 5e2a2a7f7ee8b12a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id e961b4265e038078 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 805d3ad7780ed870 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id df7dd68f00db9700 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 82bd5e7500a6859b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method genqr_ensemble \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 8c88fc4b9ce35ca0 · open full run detail →
	lamer	BGE-base-en-v1.5	0.6210	0.9893	0.4139	0.5001	0.4096	0.7483	0.7524	0.9800	0.7941	0.1401	0.4512	0.4936	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.4055	0.8453	0.7219	0.8859	0.7276	0.9045
methodlamer llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 620c7abfb1ee020f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 6af2a26c4be4997b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id aa8c1a7b3f790d02 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id d5f35cf240628440 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 08be98d55332880c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 78ba0042a329114e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 4f5b1440f5b91af8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id eb7a97a03a5b1ec7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id be768076795538e2 · open full run detail →
	lamer	BM25	0.4111	—	0.4010	0.5217	0.2395	—	0.7251	—	0.7240	0.1667	0.4677	0.6105	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3635	0.7820	0.6651	0.8666	0.6711	0.8920
methodlamer llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 3ba08ce1abc4801d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 36de53b6918e215b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id bea836bea41c6a84 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 985c2f9474647b53 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id e91e6a2bee960c7c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 74a12b2d72abe529 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id fc152cc80664915c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 68be0320c3759961 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 1cb1eb5085c84489 · open full run detail →
	lamer	SPLADE++	0.5161	0.9815	0.3697	0.4883	0.3041	0.6516	0.7046	0.9600	0.6543	0.1057	0.4161	0.4850	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3648	0.8156	0.6651	0.8956	0.6483	0.9195
methodlamer llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 579cc5a3039bc56f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 57940bb11cb19219 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 6c3920399824ba8b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id d4f19d91c62f2f77 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id f72e663204744842 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id ee987027e7b6ca46 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d207cfd370d5d82c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 41df03a4d84da054 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id c45cd567ed787d00 · open full run detail →
	mugi	BGE-base-en-v1.5	0.6194	0.9900	0.4342	0.5318	0.4192	0.7526	0.7453	0.9700	0.7972	0.1425	0.4732	0.5298	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3948	0.8548	0.7512	0.9071	0.7122	0.8894
methodmugi llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id c75e8a83747970d6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id df4b60507d728672 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 20ac8806eccf8139 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 6c9a9f1190dbd1f1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id bd29a2cdc356222a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 8060dace1f95b43e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d5c74c4a59b0f66f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 50f0e78bf714634c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id e78fee0e18c2bd8d · open full run detail →
	mugi	BM25	0.3868	—	0.4103	0.5296	0.2435	—	0.7203	—	0.6927	0.1694	0.5009	0.5921	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3609	0.8122	0.6911	0.9055	0.6268	0.9015
methodmugi llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 6444a5e04bcb9013 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 898543f1681a8169 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id 831cb39c41716188 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id fb35e4c85286196d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 74ad89762275ca4e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 20e68a265b0a0f15 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 74b697f39b901162 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id b8d5bfc95f0c4b50 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 2c11d2c0e20cc0cb · open full run detail →
	mugi	SPLADE++	0.5031	0.9787	0.3735	0.5044	0.3023	0.6787	0.6951	0.9493	0.6639	0.1105	0.4394	0.4972	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3260	0.8098	0.6746	0.9275	0.6419	0.9165
methodmugi llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id c5cd4658373a61c8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id bbed6bd9f822ca4b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 5328e402f08642b8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 9f31850bdb47fef0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id d5d4c66acd86484c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 6afab8749e67e7e2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 00665889c6151a55 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d5b50b564ccc29e8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method mugi \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 2b1068b944df9dec · open full run detail →
	qa_expand	BGE-base-en-v1.5	0.6213	0.9900	0.4013	0.4955	0.3891	0.7274	0.7431	0.9667	0.7775	0.1370	0.4842	0.4983	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3485	0.8498	0.6999	0.8733	0.6916	0.8785
methodqa_expand llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id d75dc3435b61737d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id d84f9458a62e7059 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 72847fa3e0c288e0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id a1c684c668f6fa70 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 63233688412cfd44 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 44dcf54abf2cc8c7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 7364ebdc2ba6093d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id e4ba6fa7944f9595 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 2e243e54153ec180 · open full run detail →
	qa_expand	BM25	0.3995	—	0.3744	0.4709	0.2484	—	0.7015	—	0.6809	0.1600	0.4474	0.5517	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3215	0.7876	0.6109	0.8396	0.6152	0.8727
methodqa_expand llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 596c110a56d70dfe · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 66a97127eca1d555 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id 96c937c633ca0517 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id b21dd96bd1694da4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 107505689e2b41f0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 33c770ec0c64ff7b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 2157cb85743005b6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 7ac8cd9fc5333fcc · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 1e81de4a38d3f0dd · open full run detail →
	qa_expand	SPLADE++	0.5174	0.9794	0.3830	0.5213	0.3333	0.6464	0.6796	0.9393	0.6324	0.1079	0.4168	0.4803	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3347	0.8285	0.6757	0.9005	0.6983	0.9284
methodqa_expand llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 158f4d081331a778 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 97e1f030ff0d3699 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 75929acc64d173b8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 4476b77e7af895c9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 47b49e007d29cc1b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 2af9d9ccf528d1d9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 326588da5cddb65c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id f862bc4b68efd409 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method qa_expand \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 438bf85c0461670d · open full run detail →
	Q2D (FS)	BGE-base-en-v1.5	0.6190	0.9900	0.4113	0.5101	0.4098	0.7431	0.7540	0.9633	0.7891	0.1401	0.4857	0.5135	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3845	0.8568	0.7419	0.9027	0.6792	0.8913
methodQ2D (FS) llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 51d162995ec0a2f3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 6e69a7a71d16e72e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 11285804e631bcdb · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 01d878ea2fc9b053 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 49b18dfea2539cae · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 9518c746ad052cd4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 01d59a5d7b8d4ab4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id eb1caa0c2dd69663 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 9e402b046c4e8388 · open full run detail →
	Q2D (ZS)	BGE-base-en-v1.5	0.6187	0.9900	0.4217	0.5121	0.4060	0.7383	0.7494	0.9667	0.7712	0.1382	0.4681	0.5148	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3954	0.8508	0.7269	0.9092	0.6982	0.8945
methodQ2D (ZS) llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 09a69a5ba27fe9ff · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 9103a6b0e06618d3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 12653e7f3eb402aa · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 495d3baa5bd24e3f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id ddffd81e2c94d80e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 8fb0459d4ecf81f3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 74c56d3f023a6937 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 48b4b59a5b02d010 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 798d2a35e6571e07 · open full run detail →
	Q2D (COT)	BGE-base-en-v1.5	0.6188	0.9900	0.3528	0.4617	0.3941	0.7358	0.7387	0.9600	0.7710	0.1367	0.4070	0.4508	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3498	0.8236	0.7121	0.8712	0.6411	0.8485
methodQ2D (COT) llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 5d4bfa3d7979b79e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 244bcb7d2b89e11e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id ec015f5525195e8f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 842ba7936f288255 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id b0138f4a386d717d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id e87d01511e4e667c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id f25fac91d7babdba · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id b9d7c0a5548195f4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 62d4cc55b339c1e6 · open full run detail →
	Q2D (ZS)	BM25	0.3995	—	0.4034	0.5107	0.2540	—	0.7172	—	0.6973	0.1672	0.4675	0.5557	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3506	0.8002	0.6557	0.8807	0.6207	0.8801
methodQ2D (ZS) llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 8619dbecc084fc26 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 42a0b43076706ad7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id acdc3d0f705b0626 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 6245f4f011f17ead · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 78f54a86913e8247 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id d5ce0328dc65a110 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 7257e4c7b7399f3a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id de9e52ec1241209f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 0a142166226d768d · open full run detail →
	Q2D (COT)	BM25	0.4060	—	0.3787	0.4778	0.2453	—	0.7077	—	0.6785	0.1590	0.4172	0.5578	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3075	0.7526	0.6378	0.8508	0.5651	0.8549
methodQ2D (COT) llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 85e17bdc073973ce · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id ace9ce73f96f3895 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id 3357fe0963c52576 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id fd412b5ac4de8ca3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 49d3711fcae40721 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id d914a0a95d7b5494 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 2f716974eaeb6aab · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 8d6339120c7a33ff · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 445f739af46bde73 · open full run detail →
	Q2D (FS)	BM25	0.3991	—	0.3904	0.4993	0.2509	—	0.7163	—	0.7078	0.1673	0.4807	0.6048	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3467	0.8020	0.6875	0.8959	0.6264	0.8907
methodQ2D (FS) llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id a52cee8a07f43fc7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 26e4b3a7f993e212 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id b8404ff066b9fe20 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id f82daa013c693bfe · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id b57adb12adaa79d2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id c9cdc67d7569d8b2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id ce90fad5adaa8336 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 7846f4e4f7cbdf8e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 8f3dba10175f7a3a · open full run detail →
	Q2D (FS)	SPLADE++	0.5200	0.9801	0.3662	0.5023	0.3261	0.6552	0.7035	0.9567	0.6689	0.1142	0.4238	0.4846	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3333	0.8206	0.7151	0.9124	0.6499	0.9234
methodQ2D (FS) llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id d4c96038b2421b72 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 1431b663f3476c4a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 3b87caa9466242e1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 86a415d70ffeb20b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id b2bfdd905ff25123 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id f938ffe59e55de6a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id bfd75142f402414e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id ec284b56cc4b0cee · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"fs","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 22abf0511cc254f7 · open full run detail →
	Q2D (ZS)	SPLADE++	0.5194	0.9808	0.3707	0.5051	0.3213	0.6469	0.6965	0.9560	0.6272	0.1095	0.4068	0.4700	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3200	0.8248	0.6682	0.9161	0.6144	0.9161
methodQ2D (ZS) llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id e50ed88a6a0d9743 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id b2dbfb16c1c2a469 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 233ebdf8614f845f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 5302e9c1df55f57a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 17e0321bb5b85351 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id f5faf7ec25250d6b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id c6cd57f5e3a52bcc · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 92c02ea9d1094686 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train","mode":"zs"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 7f5e197e86320128 · open full run detail →
	Q2D (COT)	SPLADE++	0.5199	0.9808	0.3897	0.5470	0.3157	0.6411	0.6834	0.9533	0.6425	0.1159	0.4054	0.4627	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3016	0.8393	0.6941	0.9148	0.6099	0.8857
methodQ2D (COT) llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 644866e8f19d2fc8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 51b13cae88a26722 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id b32f12da4795d84a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id f1a394c7e06f8f5b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 973fa6f6c5c68d50 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id d9f19520c34daa46 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id b3cf3dc3d5214a0b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 0190c9dc52a6cfc7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2doc \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"mode":"cot","num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 33c4710d0519694c · open full run detail →
	query2e	BGE-base-en-v1.5	0.6196	0.9900	0.3610	0.4706	0.3793	0.7222	0.7382	0.9567	0.7857	0.1412	0.4509	0.5067	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3744	0.8503	0.7069	0.8760	0.6606	0.8528
methodquery2e llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 81545bd2a66e7b02 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id b3352953a1e7514e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 3f062c4e7a58281f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 54b3cb89c8c63f5c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id c4974093872ab570 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 74ba81f36dac7916 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id dc2bdf9e64c969cd · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id e12ac54481db1c90 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id c5e8c61ac6d0ccb1 · open full run detail →
	query2e	BM25	0.4066	—	0.3578	0.4641	0.2518	—	0.6969	—	0.6942	0.1611	0.4484	0.5647	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3148	0.7605	0.5845	0.8501	0.5546	0.8609
methodquery2e llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 40f110d0a40bc926 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 835fd60092d69d40 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id f74f30707b9b167c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id c9364b60bcba9cc6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id d522590b4b14d2f8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id ffa251db10b65250 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 130dd4da2379e09b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 71fabe1af05dfe6e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 9ce92bd9a02b250b · open full run detail →
	query2e	SPLADE++	0.5188	0.9808	0.3755	0.5195	0.3036	0.6438	0.7049	0.9427	0.6196	0.1201	0.4076	0.4799	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3442	0.8328	0.6686	0.9104	0.6353	0.9286
methodquery2e llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 14378080d839b817 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id fec87705663d16f9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id a78907f0a308a03d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id c6d2a5ca2088376a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 8ad0e2adba9b580e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id a15ecec4d8254e3c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 5daf7ba27c702c81 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id e3c0316a5f22c1b8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method query2e \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id da479f6027f0f77d · open full run detail →