lamer

All results produced by QueryGym · fully reproducible!

12 model × retriever configurations for this method across BEIR, MS MARCO DL, and DL-HARD.
Click any row or the + button to expand. Tabs switch dataset context. The three steps (reformulate → retrieve → evaluate) update accordingly.

	Model	Retriever	ArguAna		DBPedia		FiQA		SciFact		COVID		News		BRIGHT — AOPS		BRIGHT — Biology		BRIGHT — Earth Science		BRIGHT — Economics		BRIGHT — LeetCode		BRIGHT — Pony		BRIGHT — Psychology		BRIGHT — Robotics		BRIGHT — Stack Overflow		BRIGHT — Sustainable Living		BRIGHT — TheoremQA Questions		BRIGHT — TheoremQA Theorems		DL-HARD		DL 2019		DL 2020
	Model	Retriever	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100	nDCG@10	R@100																									nDCG@10	R@1k	nDCG@10	R@1k	nDCG@10	R@1k
	Qwen2.5-72B-Instruct	BGE-base-en-v1.5	0.6210	0.9893	0.4139	0.5001	0.4096	0.7483	0.7524	0.9800	0.7941	0.1401	0.4512	0.4936	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.4055	0.8453	0.7219	0.8859	0.7276	0.9045
methodlamer llmQwen2.5-72B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 620c7abfb1ee020f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 6af2a26c4be4997b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id aa8c1a7b3f790d02 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id d5f35cf240628440 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 08be98d55332880c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 78ba0042a329114e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 4f5b1440f5b91af8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id eb7a97a03a5b1ec7 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id be768076795538e2 · open full run detail →
	Qwen2.5-72B-Instruct	BM25	0.4111	—	0.4010	0.5217	0.2395	—	0.7251	—	0.7240	0.1667	0.4677	0.6105	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3635	0.7820	0.6651	0.8666	0.6711	0.8920
methodlamer llmQwen2.5-72B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-arguana-test run.txt Run id 3ba08ce1abc4801d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 36de53b6918e215b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-fiqa-test run.txt Run id bea836bea41c6a84 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 \ beir-v1.0.0-scifact-test run.txt Run id 985c2f9474647b53 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id e91e6a2bee960c7c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 74a12b2d72abe529 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id fc152cc80664915c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 68be0320c3759961 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 1cb1eb5085c84489 · open full run detail →
	Qwen2.5-72B-Instruct	SPLADE++	0.5161	0.9815	0.3697	0.4883	0.3041	0.6516	0.7046	0.9600	0.6543	0.1057	0.4161	0.4850	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3648	0.8156	0.6651	0.8956	0.6483	0.9195
methodlamer llmQwen2.5-72B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 579cc5a3039bc56f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 57940bb11cb19219 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 6c3920399824ba8b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id d4f19d91c62f2f77 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id f72e663204744842 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id ee987027e7b6ca46 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id d207cfd370d5d82c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 41df03a4d84da054 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-72B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id c45cd567ed787d00 · open full run detail →
	Qwen2.5-7B-Instruct	BGE-base-en-v1.5	0.6195	0.9908	0.3900	0.4838	0.3981	0.7318	0.7466	0.9733	0.7843	0.1360	0.4517	0.4753	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3788	0.8315	0.7113	0.8668	0.6825	0.8940
methodlamer llmQwen2.5-7B-Instruct retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id a50f451a6405c42d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 667de708d3cde181 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 5de7b3c84e18b6cf · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id c3907dd559a87bc4 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 88560f5e959c6818 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 9f17a3b07e7400e0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id f30bba5741441870 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 73a846cf910b84f8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 230e2bbca7a7fd18 · open full run detail →
	Qwen2.5-7B-Instruct	BM25	0.4063	0.9388	0.3896	0.5139	0.2337	0.5558	0.7140	0.9593	0.6955	0.1704	0.4424	0.5960	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3570	0.7633	0.6602	0.8553	0.6322	0.8933
methodlamer llmQwen2.5-7B-Instruct retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 78a09817091818d1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id d86dc93cb8e496dc · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 4ae79ecd932684fd · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id eb8c999ba4df568e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id ffeb0dbe30c15130 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 191f3146ee820605 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 2887a055720c6629 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 67f45dc24ca2c49b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 4d996428a216f673 · open full run detail →
	Qwen2.5-7B-Instruct	SPLADE++	0.5148	0.9794	0.3499	0.4799	0.2944	0.6487	0.6651	0.9560	0.6339	0.1002	0.3967	0.4728	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3280	0.7917	0.6465	0.8654	0.6076	0.9213
methodlamer llmQwen2.5-7B-Instruct retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 0616194d13348b61 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 327fda84b730628b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 6e3f251e13c96034 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 13886968f236369d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id e760de4c23be40a5 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 0fd33ea475108659 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id dbee4e6ccdf98310 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 66711eb3e485ecb1 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model Qwen/Qwen2.5-7B-Instruct \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id ebdfca59bc737639 · open full run detail →
	gpt-4.1	BGE-base-en-v1.5	0.6204	0.9893	0.4018	0.4998	0.4080	0.7410	0.7572	0.9733	0.7796	0.1373	0.4367	0.4591	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.4120	0.8557	0.7032	0.8888	0.7148	0.9026
methodlamer llmgpt-4.1 retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 252da7e5de2cd06e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 04f1373129f2a5be · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 26622b0c17e4a4db · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 77c750df1bd7ea6d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 2d2452fe538db782 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 1a4bbeecc9fca276 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 0e137bc38ea7682b · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id d4be7ddf24e5a797 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 4fecb6d127f71e40 · open full run detail →
	gpt-4.1	BM25	0.4119	0.9452	0.3989	0.5159	0.2616	0.5901	0.7253	0.9487	0.7020	0.1661	0.4799	0.5960	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3555	0.8065	0.6368	0.8566	0.6530	0.9002
methodlamer llmgpt-4.1 retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id bcffbda0dd6fdc12 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id eaec077abfaf854a · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 687ae132065e0b45 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id f51fc1481896b8bc · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 0a644ef98cc95b6f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 6a0e18727969f9d2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id aa3971fcc285abde · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 63c3ce02ce9a2f14 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 8d328e3c1ae63cfa · open full run detail →
	gpt-4.1	SPLADE++	0.3836	0.9829	0.3559	0.4904	0.3292	0.6724	0.7182	0.9577	0.6312	0.1081	0.4520	0.4770	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3673	0.8246	0.6836	0.9065	0.6390	0.9378
methodlamer llmgpt-4.1 retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 8a03d57b02937b1f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 705d1c4e8e7c517d · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id a1dea4fa8242d0c9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id 62108cd5b314e124 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 031a4a3df33cc5ba · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id d0ecf5c9f11b78a0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 7b90f42881e20399 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id b70672ae7d83c378 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1 \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 44018304f3ecf315 · open full run detail →
	gpt-4.1-nano	BGE-base-en-v1.5	0.6254	0.9900	0.3827	0.4804	0.4009	0.7310	0.7507	0.9593	0.8007	0.1340	0.4060	0.4264	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3759	0.8352	0.7265	0.8894	0.7135	0.8846
methodlamer llmgpt-4.1-nano retrieverBGE-base-en-v1.5 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id f73f72e0ee7fbe5c · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id f0718ab9227f9667 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id a23ffa14158a8369 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id f76686096d5949f9 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 224a372dd7d8ee9e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id 4b6265051132c131 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id 94df486d2e5caa53 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 78fdf57d33b3086f · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BGE-base-en-v1.5 (dense) python -m pyserini.search.faiss \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.bge-base-en-v1.5 \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder BAAI/bge-base-en-v1.5 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 531c723827005395 · open full run detail →
	gpt-4.1-nano	BM25	0.4037	0.9388	0.3440	0.4807	0.2360	0.5449	0.7220	0.9393	0.6721	0.1748	0.4328	0.5575	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3398	0.7697	0.6731	0.8548	0.6560	0.8865
methodlamer llmgpt-4.1-nano retrieverBM25 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 80fe25ed5bf87022 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 8aac783cbbebdffe · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 03941802197741ee · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id e6250cabff57cb42 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id 26c47264746a1f10 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.flat \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id c990d296f840b6a0 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id bd50ffd2e8e0f2a6 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id 4813531aa1f83ce8 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · BM25 (lexical) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --bm25 --k1 0.9 --b 0.4 \ --output run.txt \ --hits 1000 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id 5f5130519e341b05 · open full run detail →
	gpt-4.1-nano	SPLADE++	0.3800	0.9780	0.3316	0.4680	0.3014	0.6543	0.7207	0.9443	0.6285	0.1143	0.4012	0.4661	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	0.3459	0.7969	0.6916	0.8975	0.6254	0.9244
methodlamer llmgpt-4.1-nano retrieverSPLADE++ 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-arguana \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-arguana.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-arguana-test run.txt Run id 5f42ef0d9c5eedd2 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-dbpedia-entity \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-dbpedia-entity.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-dbpedia-entity-test run.txt Run id 03cecba09b8621cc · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-fiqa \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-fiqa.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-fiqa-test run.txt Run id 217166f1520b6b38 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-scifact \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-scifact.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-scifact-test run.txt Run id c213988750249688 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-covid \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-covid.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-covid-test run.txt Run id a7f15b7e6404eda3 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset beir-v1.0.0-trec-news \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index beir-v1.0.0-trec-news.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@100 python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.100 \ beir-v1.0.0-trec-news-test run.txt Run id a01865b479e96df5 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.dlhard \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ /mnt/data/son/Thesis/t5/data/dlhard/neutral_queries.tsv run.txt Run id bf4759de8f4f3141 · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2019 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl19-passage run.txt Run id a4920606bfbad73e · open full run detail → 1 reformulate querygym → reformulated_queries.tsv python examples/querygym_pyserini/pipeline.py \ --dataset msmarco-v1-passage.trecdl2020 \ --method lamer \ --model openai/gpt-4.1-nano \ --steps reformulate \ --temperature 1 \ --max-tokens 128 \ --method-params '{"num_examples":4,"train_split":"train"}' \ --output-dir outputs/reproduce 2 retrieve pyserini · SPLADE++ (learned_sparse) python -m pyserini.search.lucene \ --threads 16 --batch-size 128 \ --index msmarco-v1-passage.splade-pp-ed \ --topics outputs/reproduce/queries/reformulated_queries.tsv \ --encoder naver/splade-cocondenser-ensembledistil \ --output run.txt \ --hits 1000 --impact 3 evaluate trec_eval · nDCG@10 + R@1k python -m pyserini.eval.trec_eval -c -m ndcg.cut.10 -m recall.1000 \ dl20-passage run.txt Run id ece61fc263f8c089 · open full run detail →