PromptRank Approach to Improve Few-shot Efficiency for Complex Question Answering in LLMs

Multi-hop Question Answering (MQA), which requires passing through multiple documents to answer complex questions, hinges on finding the document path relevant to the question. However, building a large-scale dataset of question-path pairs incurs enormous cost, and simple heuristics carry a high risk of including inaccurate documents.

PromptRank proposes a two-stage process to address this problem. First, a simple unsupervised retrieval method such as TF-IDF is used to select an initial set of document paths that are likely relevant to the question. Then, a more sophisticated Reranker model re-evaluates the relevance of each path.

The core of the reranker lies in its Scoring Function. PromptRank computes $P_{LM}(q|T_c)$, the conditional probability that the question($q$) is generated given the path($T_c$). This has the following advantages.

• Alignment with pre-training: It matches the LM pre-training pattern of FAQ or survey data, where a question follows a document.
• Resolving Surface Form sensitivity: It prevents the problem of probability varying with the length of the path, and enables a fairer evaluation by comparing the probability of the question, which is the same sequence.

Additionally, when constructing the Path Prompt, an Instruction such as 'read the preceding document and ask a question' is used to guide the model to assign higher scores to more relevant paths.