This image was created using an AI image creation program
介绍作为一个每天与向量数据库打交道的人，我已经习惯了检索增强生成 （RAG） 在从密集的用户手册中提取信息、浏览复杂的代码库或进行深入的法律研究等场景中的传统应用。这些“与你的文档对话”的用例虽然令人印象深刻，但往往围绕着不同数据集的类似挑战，这可能会变得有些单调。
因此，当我看到加州大学伯克利分校的研究人员Danny Halawi、Fred Zhang、Chen Yueh-Han和Jacob Steinhardt的论文《用语言模型进行人类水平预测》时，我感到特别耳目一新。他们提出了RAG的一种新颖的（至少对我而言）用途：预测事件！
在这篇博文中，我们将从头到尾详细演练示例预测，以了解系统如何采用提示工程来驱动其预测，并以结果的简要概述结束。
问题陈述想象一下，现在是 2023 年 6 月 15 日，你是一个狂热的 Reddit 用户。当然，您对以下问题非常感兴趣：
Reddit 是否会在 2023 年 7 月 1 日之前宣布对其拟议的 API 费用定价进行更改或延迟？
为了给您提供更多的背景信息，以下是对情况的简要概述：
2023 年 4 月，Reddit 宣布了其应用程序编程接口 （API） 的新条款，该工具是第三方应用程序开发人员访问 Reddit 数据的工具。这一变化包括引入 API 使用定价，这与之前免费提供 API 的做法大相径庭。这引起了第三方 Reddit 应用程序的开发人员和用户的强烈抗议，因为这些变化可能会给开发人员带来大量成本，并可能关闭许多流行的第三方应用程序。
流行的第三方应用程序Apollo for Reddit的开发者克里斯蒂安·塞利格（Christian Selig）表示，根据新条款，他将承担每年2000万美元的费用。这一启示引发了Reddit用户的广泛不满，导致了一场涉及8000多个Reddit社区的协调抗议活动，其中包括一些最大和最活跃的社区。一些社区承诺无限期地保持私密，直到Reddit做出改变。Reddit的首席执行官史蒂夫·霍夫曼（Steve Huffman）对API的变化仍然坚定不移，他在Ask Me Anything（AMA）会议和内部备忘录中强调，公司必须继续进行这些变化。
这一立场被认为是Reddit提高盈利能力的更广泛努力的一部分，特别是考虑到该公司最近的裁员和明年实现收支平衡的计划。然而，这种方法加剧了用户的不满，一些人猜测这些变化是为了限制第三方应用程序的可行性，并鼓励用户迁移到官方Reddit应用程序。与 ChatGPT 共同编写的背景。
这个问题实际上来自研究人员从几个预测市场平台中整理的数据集。
让我们探讨一下本文中提出的 RAG 系统将如何解决这个问题。剧透警报：该系统通过根据不同的数据检索和提示策略生成不同的预测来模拟专家小组。然后，它会智能地汇总这些响应，以制定最终的、明智的预测！
机制步骤 1：生成全面的搜索查询该系统使用互联网作为知识库来形成其判断的基础。在这篇文章中，我们将仅使用Google作为检索相关信息的来源。
现在，我们可以将问题用作搜索查询，但这不是专家处理问题的方式。专家会将问题分解为更小的部分，并搜索有助于他们回答子问题的信息。我们可以使用LLM来模拟专家的思维过程，并遵循以下思维链提示：
I will provide you with a forecasting question and the background information for the question. I will then ask you to generate short search queries (up to 8 words each) that I'll use to find articles on Google News to help answer the question.Question:Will Reddit announce changes or a delay to its proposed API fee pricing before July 1, 2023?Question Background:In April 2023, Reddit announced new terms for its Application Programming Interface (API), the tool through which developers of third-party apps access Reddit's data. This change included the introduction of pricing for API usage, a significant departure from the previous practice of providing the API for free. This led to an outcry from both developers and users of third-party Reddit apps, as these changes could result in substantial costs for developers and the potential shutdown of many popular third-party apps.The developer of the popular third-party app, Apollo for Reddit, Christian Selig, stated he would be liable for an annual cost of $20 million under the new terms. This revelation sparked widespread discontent among Reddit users, leading to a coordinated protest involving more than 8,000 Reddit communities, including some of the largest and most active, going dark. Some communities have pledged to remain private indefinitely until Reddit makes changes.Reddit's CEO, Steve Huffman, has remained resolute regarding the API changes, emphasizing in an Ask Me Anything (AMA) session and internal memos that the company must proceed with the changes. This stance is perceived as part of Reddit's broader effort to increase profitability, especially given the company's recent layoffs and plans to break even next year. However, the approach has escalated user dissatisfaction, with some speculating that the changes are an attempt to limit the viability of third-party apps and encourage users to migrate to the official Reddit app.Background co-written with ChatGPT.Today's date: 2023-06-15Question close date: 2023-06-16You must generate this exact amount of queries: 3Start off by writing down sub-questions. Then use your sub-questions to help steer the search queries you produce.Your response should take the following structure:Thoughts:{ Insert your thinking here. }Search Queries:{ Insert the queries here. Use semicolons to separate the queries. }m这使得 LLM（使用 gpt-4–1106-preview）生成如下响应：
Thoughts:To answer the main question, we need to look for any recent developments or announcements from Reddit regarding the API fee pricing. The sub-questions that arise from this are:1. Has Reddit made any recent announcements about the API fee pricing?2. Is there any news on Reddit delaying the API fee implementation?3. Are there any updates from Reddit's CEO or official channels about changes to the API pricing policy in response to the backlash?These sub-questions will guide the search queries to find the most relevant and recent news articles that could indicate whether Reddit will announce changes or a delay to its proposed API fee pricing before the given date.Search Queries:Reddit API fee pricing announcement; Reddit API fee delay news; Reddit CEO statement on API pricing changes;因此，根据 LLM 的说法，要回答 Reddit 是否会在 2023 年 7 月 1 日之前宣布更改或延迟其拟议的 API 费用定价的问题，我们需要搜索：
Reddit API 费用定价公告Reddit API 费用延迟Reddit CEO 关于 API 定价变化的声明步骤 2：删除重复的搜索查询将每个提示视为对问题具有不同观点的独特专家。我们可以生成一个类似的提示来模拟另一个专家的思维过程，并生成一组不同的搜索查询。这种观点的多样性有助于系统生成更全面的搜索结果集。
例如，另一个提示可能是：
I will provide you with a forecasting question and the background information for the question.Question:Will Reddit announce changes or a delay to its proposed API fee pricing before July 1, 2023?Question Background:In April 2023, Reddit announced new terms for its Application Programming Interface (API), the tool through which developers of third-party apps access Reddit's data. This change included the introduction of pricing for API usage, a significant departure from the previous practice of providing the API for free. This led to an outcry from both developers and users of third-party Reddit apps, as these changes could result in substantial costs for developers and the potential shutdown of many popular third-party apps.The developer of the popular third-party app, Apollo for Reddit, Christian Selig, stated he would be liable for an annual cost of $20 million under the new terms. This revelation sparked widespread discontent among Reddit users, leading to a coordinated protest involving more than 8,000 Reddit communities, including some of the largest and most active, going dark. Some communities have pledged to remain private indefinitely until Reddit makes changes.Reddit's CEO, Steve Huffman, has remained resolute regarding the API changes, emphasizing in an Ask Me Anything (AMA) session and internal memos that the company must proceed with the changes. This stance is perceived as part of Reddit's broader effort to increase profitability, especially given the company's recent layoffs and plans to break even next year. However, the approach has escalated user dissatisfaction, with some speculating that the changes are an attempt to limit the viability of third-party apps and encourage users to migrate to the official Reddit app.Background co-written with ChatGPT.Today's date: 2023-06-15Question close date: 2023-06-16Task:- Generate brief search queries (up to 8 words each) to gather information on Google that could influence the forecast.You must generate this exact amount of queries: 3Your response should take the following structure:Thoughts:{ Insert your thinking here. }Search Queries:{ Insert the queries here. Use semicolons to separate the queries. }这会导致以下响应：
Thoughts:To forecast whether Reddit will announce changes or a delay to its proposed API fee pricing, we need to look for the most recent statements from Reddit or its CEO, any news on negotiations with developers, and the current sentiment or actions of the Reddit community. The search queries should be designed to capture any new developments, official announcements, or significant community actions that might pressure Reddit into changing its stance.Search Queries:Reddit API fee pricing update 2023; Steve Huffman Reddit API pricing statement; Reddit community response to API fees;因此，将两组搜索查询与问题相结合并删除重复项，我们得到以下一组搜索查询：
Reddit 是否会在 2023 年 7 月 1 日之前宣布对其拟议的 API 费用定价进行更改或延迟？Reddit API 费用延迟史蒂夫·霍夫曼 Reddit API 定价声明Reddit API 费用定价公告Reddit API 费用定价更新Reddit 社区对 API 费用的回应Reddit CEO 关于 API 定价变化的声明第 3 步：检索文章这部分很简单。对于每个搜索查询，我们可以从 Google 中检索排名前 n 的文章。
第 4 步：选择最相关的文章排名靠前的搜索结果实际上可能并不包含回答我们问题的相关信息。因此，这些需要从进一步考虑中删除。
我们可以请LLM帮助我们完成这项任务。这里有一个提示，用于确定一篇文章是否有助于回答问题：
Please consider the following forecasting question and its background information.After that, I will give you a news article and ask you to rate its relevance with respect to the forecasting question.Question:Will Reddit announce changes or a delay to its proposed API fee pricing before July 1, 2023?Question Background:In April 2023, Reddit announced new terms for its Application Programming Interface (API), the tool through which developers of third-party apps access Reddit's data. This change included the introduction of pricing for API usage, a significant departure from the previous practice of providing the API for free. This led to an outcry from both developers and users of third-party Reddit apps, as these changes could result in substantial costs for developers and the potential shutdown of many popular third-party apps.The developer of the popular third-party app, Apollo for Reddit, Christian Selig, stated he would be liable for an annual cost of $20 million under the new terms. This revelation sparked widespread discontent among Reddit users, leading to a coordinated protest involving more than 8,000 Reddit communities, including some of the largest and most active, going dark. Some communities have pledged to remain private indefinitely until Reddit makes changes.Reddit's CEO, Steve Huffman, has remained resolute regarding the API changes, emphasizing in an Ask Me Anything (AMA) session and internal memos that the company must proceed with the changes. This stance is perceived as part of Reddit's broader effort to increase profitability, especially given the company's recent layoffs and plans to break even next year. However, the approach has escalated user dissatisfaction, with some speculating that the changes are an attempt to limit the viability of third-party apps and encourage users to migrate to the official Reddit app.Background co-written with ChatGPT.Question Resolution Criteria:This question will resolve as Yes if, before July 1, 2023, Reddit announces changes or a delay of implementation to the pricing of its newly announced data API fees for apps that require higher usage. The pricing will be considered to have changed or delayed if Reddit announces any of the following:That they will charge anything other than "$0.24 per 1K API calls" or an equivalent amount.Any change to their threshold for free API usage of "100 queries per minute per OAuth client id if you are using OAuth authentication and 10 queries per minute if you are not using OAuth authentication".That the new pricing will take effect on a date later than July 1, 2023.Other rates or implementation dates reported to be negotiated with individual organizations do not qualify, this question is asking about the broader official fee structure policy. Metaculus will make the final determination as to whether an announcement qualifies and may annul the question in the event it is unclear whether an announcement meets these criteria.Article:---(Below I provide the first 250 tokens of the article.)Title: Despite widespread user protest, Reddit CEO says company is ‘not negotiating’ on 3rd-party app chargesWASHINGTON (AP) — If you hopped on Reddit to scroll through your favorite forums this week, you may have encountered “private” or “restricted” messages. That’s because thousands of subreddits chose to go dark in an ongoing protest of some controversial changes announced by the online discussion network.The blackout, which began Monday, emerged out of outrage over Reddit’s upcoming changes to its application programming interface (API) access — notably the company’s plan to start charging high usage third-party apps, who have long-used the Reddit data at no cost to build resources for users not available through the official site or app.Organizers of the protest say that Reddit’s new policy threatens to end key ways of historically customizing the platform, which relies heavily on the volunteer labor of subreddit moderators. Many users currently rely on third-party apps to access features that are unavailable in the official Reddit app, particularly for content moderation and access---Please rate the relevance of the article to the question, at the scale of 1-61 -- irrelevant2 -- slightly relevant3 -- somewhat relevant4 -- relevant5 -- highly relevant6 -- most relevantGuidelines:- You don't need to access any external sources. Just consider the information provided.- Focus on the content of the article, not the title.- If the text content is an error message about JavaScript, paywall, cookies or other technical issues, output a score of 1.Your response should look like the following:Thoughts: { insert your thinking }Rating: { insert your rating }对于那篇文章，LLM说：
Thoughts: The article discusses the ongoing protest and outrage over Reddit's upcoming changes to its API access, which directly relates to the background information about Reddit's announcement of new terms for its API. It also mentions the potential impact on third-party apps and the dissatisfaction among users and moderators. However, it does not provide any specific information about whether Reddit will announce changes or a delay to its proposed API fee pricing before July 1, 2023.Rating: 4请注意，我们还将问题视为已解决的条件（请参阅问题解决标准）作为提示上下文的一部分，以帮助 LLM 确定文章的相关性。
我们可以使用这种方法将任何评分为 4 或更高的文章视为相关文章，并丢弃其余文章。
第 5 步：总结相关文章文章可能很长，LLM 的上下文窗口有限。因此，我们可以请 LLM 为我们总结相关文章，以便我们可以尽可能多地填充有用和多样化的信息。例如，我们可以将自己限制在前 20 篇最相关的文章中，并要求 LLM 对它们进行总结。
以下是总结文章的提示：
I want to make the following article shorter (condense it to no more than 100 words).Article:---{article}---When doing this task for me, please do not remove any details that would be helpful for making considerations about the following forecasting question.Forecasting Question: Will Reddit announce changes or a delay to its proposed API fee pricing before July 1, 2023?Question Background: In April 2023, Reddit announced new terms for its Application Programming Interface (API), the tool through which developers of third-party apps access Reddit's data. This change included the introduction of pricing for API usage, a significant departure from the previous practice of providing the API for free. This led to an outcry from both developers and users of third-party Reddit apps, as these changes could result in substantial costs for developers and the potential shutdown of many popular third-party apps.The developer of the popular third-party app, Apollo for Reddit, Christian Selig, stated he would be liable for an annual cost of $20 million under the new terms. This revelation sparked widespread discontent among Reddit users, leading to a coordinated protest involving more than 8,000 Reddit communities, including some of the largest and most active, going dark. Some communities have pledged to remain private indefinitely until Reddit makes changes.Reddit's CEO, Steve Huffman, has remained resolute regarding the API changes, emphasizing in an Ask Me Anything (AMA) session and internal memos that the company must proceed with the changes. This stance is perceived as part of Reddit's broader effort to increase profitability, especially given the company's recent layoffs and plans to break even next year. However, the approach has escalated user dissatisfaction, with some speculating that the changes are an attempt to limit the viability of third-party apps and encourage users to migrate to the official Reddit app.Background co-written with ChatGPT.请注意，我们强调不要删除任何有助于考虑预测问题的细节。这是为了确保摘要保留与我们想要回答的问题最相关的信息。
步骤 6：生成预测现在我们有了一组总结的文章，我们可以去找专家，询问他们的预测。我们可以通过 LLM 来模拟这一点，方法是要求它使用以下提示模板根据摘要文章生成预测：
Question:{question}Question Background:{background}Resolution Criteria:{resolution_criteria}Today's date: {date_begin}Question close date: {date_end}We have retrieved the following information for this question:{retrieved_info}Instructions:1. Write down any additional relevant information that is not included above. This should be specific facts that you already know the answer to, rather than information that needs to be looked up.{{ Insert additional information }}2. Provide at least 3 reasons why the answer might be no.{{ Insert your thoughts }}3. Provide at least 3 reasons why the answer might be yes.{{ Insert your thoughts }}4. Rate the strength of each of the reasons given in the last two responses. Think like a superforecaster (e.g. Nate Silver).{{ Insert your rating of the strength of each reason }}5. Aggregate your considerations.{{ Insert your aggregated considerations }}6. Output your answer (a number between 0 and 1) with an asterisk at the beginning and end of the decimal.{{ Insert your answer }}或者，这也行得通：
Question:{question}Question Background:{background}Resolution Criteria:{resolution_criteria}Today's date: {date_begin}Question close date: {date_end}We have retrieved the following information for this question:{retrieved_info}Think step by step: {{ Insert your step by step consideration }}Aggregating considerations: {{ Aggregate your considerations }}Answer: {{ Output your answer (a number between 0 and 1) with an asterisk at the beginning and end of the decimal }}这样，我们可以根据从摘要文章中检索到的信息生成一组不同的预测。此外，如果我们可以访问多个 LLM，我们可以通过将不同的 LLM 与不同的提示配对来生成更多样化的预测。因此，如果我们有 2 个 LLM，则通过将每个 LLM 与上面的 2 个提示配对，我们可以得到 4 个不同的预测。
为了节省空间，这里只是 LLM 使用第二个提示模板的响应：
Think step by step:1. The CEO of Reddit, Steve Huffman, has faced significant backlash from developers and users alike regarding the proposed API pricing changes set to take place on July 1, 2023.2. Despite protests involving more than 8,000 communities going dark, Huffman has repeatedly emphasized the company’s stand on not negotiating the decision, which suggests a strong likelihood of the changes being implemented as initially stated.3. The background information and articles consistently highlight Reddit's firm stance and the CEO's repeated confirmation that they are moving forward with the pricing changes.4. There has been mention of widespread discontent and the potential for continued or even indefinite protests, but there does not appear to be a significant movement or pressure that could cause the CEO to reverse or delay the decision.5. Based on the articles, there is no indication that Reddit is considering changing or delaying the implementation. Instead, the CEO has downplayed the protests and emphasized the need for the company to proceed with the changes.Aggregating considerations:Every related article points toward Reddit’s CEO Steve Huffman's resolution to implement the API changes without delay or alterations. While discontent and protests are significant, there has been no statement or indication from official Reddit communications suggesting an openness to altering or postponing the fee implementation.Answer: Given the current information and the firm stance of Reddit's CEO, there is insufficient evidence to suggest that Reddit will change or delay the proposed API fee pricing before July 1, 2023.0.1第 7 步：整合预测现在我们有一组来自不同“专家”的 4 个预测，我们可以将它们组合在一起以获得最终预测。我们有很多方法可以做到这一点，例如多数票、加权多数票等。但是，既然我们一直依靠 LLM 来完成大部分工作，为什么不使用 LLM 来集成预测呢？下面是一个提示模板，可以做到这一点：
I need your assistance with making a forecast. Here is the question and its metadata.Question: {question}Background: {background}Resolution criteria: {resolution_criteria}Today's date: {date_begin}Question close date: {date_end}I have retrieved the following information about this question.Retrieved Info:{retrieved_info}In addition, I have generated a collection of other responses and reasonings from other forecasters:{base_reasonings}Your goal is to aggregate the information and make a final prediction.Instructions:1. Provide reasons why the answer might be no.{{ Insert your thoughts here }}2. Provide reasons why the answer might be yes.{{ Insert your thoughts here }}3. Aggregate your considerations.{{ Insert your aggregated considerations here }}4. Output your prediction (a number between 0 and 1) with an asterisk at the beginning and end of the decimal.{{ Insert the probability here }}填充的模板如下所示：
I need your assistance with making a forecast. Here is the question and its metadata.Question: Will Reddit announce changes or a delay to its proposed API fee pricing before July 1, 2023?Background: In April 2023, Reddit announced new terms for its Application Programming Interface (API), the tool through which developers of third-party apps access Reddit's data. This change included the introduction of pricing for API usage, a significant departure from the previous practice of providing the API for free. This led to an outcry from both developers and users of third-party Reddit apps, as these changes could result in substantial costs for developers and the potential shutdown of many popular third-party apps.The developer of the popular third-party app, Apollo for Reddit, Christian Selig, stated he would be liable for an annual cost of $20 million under the new terms. This revelation sparked widespread discontent among Reddit users, leading to a coordinated protest involving more than 8,000 Reddit communities, including some of the largest and most active, going dark. Some communities have pledged to remain private indefinitely until Reddit makes changes.Reddit's CEO, Steve Huffman, has remained resolute regarding the API changes, emphasizing in an Ask Me Anything (AMA) session and internal memos that the company must proceed with the changes. This stance is perceived as part of Reddit's broader effort to increase profitability, especially given the company's recent layoffs and plans to break even next year. However, the approach has escalated user dissatisfaction, with some speculating that the changes are an attempt to limit the viability of third-party apps and encourage users to migrate to the official Reddit app.Background co-written with ChatGPT.Resolution criteria: This question will resolve as Yes if, before July 1, 2023, Reddit announces changes or a delay of implementation to the pricing of its newly announced data API fees for apps that require higher usage. The pricing will be considered to have changed or delayed if Reddit announces any of the following:That they will charge anything other than "$0.24 per 1K API calls" or an equivalent amount.Any change to their threshold for free API usage of "100 queries per minute per OAuth client id if you are using OAuth authentication and 10 queries per minute if you are not using OAuth authentication".That the new pricing will take effect on a date later than July 1, 2023.Other rates or implementation dates reported to be negotiated with individual organizations do not qualify, this question is asking about the broader official fee structure policy. Metaculus will make the final determination as to whether an announcement qualifies and may annul the question in the event it is unclear whether an announcement meets these criteria.Today's date: 2023-06-16Question close date: 2023-06-30I have retrieved the following information about this question.Retrieved Info:---ARTICLES[1] <omitted for brevity>[2] <omitted for brevity>[3] <omitted for brevity>[4] <omitted for brevity>[5] <omitted for brevity>[6] <omitted for brevity>[7] <omitted for brevity>[8] <omitted for brevity>[9] <omitted for brevity>[10] <omitted for brevity>[11] <omitted for brevity>----In addition, I have generated a collection of other responses and reasonings from other forecasters:---Response from forecaster 1:1. There is no additional information to provide.2. Reasons why the answer might be no: - CEO Steve Huffman has repeatedly emphasized the necessity of the API fee changes for Reddit's profitability and sustainability and has remained resolute in the face of backlash, suggesting a firm commitment to the new pricing structure. - The company's statements so far have indicated a clear stance against negotiation or alteration of the planned pricing, which suggests they are prepared to withstand a significant amount of community backlash without changing course. - From a business perspective, changing the pricing plan now could set a precedent where Reddit might be seen as bending under pressure, potentially weakening its position in future negotiations or policy rollouts.3. Reasons why the answer might be yes: - The ongoing, widespread protest involving thousands of subreddits and key developers expressing outrage could be a significant pressure point, especially if the protest escalates or continues for a prolonged period, potentially harming Reddit's brand image and user satisfaction. - If advertisers or other revenue-generating partners express concerns or withdraw their business due to the negative press and user reactions, Reddit may be financially incentivized to revise or delay the pricing changes. - The potential shutdown of popular third-party apps and services could lead to a significant loss of user engagement, which Reddit may seek to mitigate by altering the course of their decision before it causes irreparable damage to their user base.4. Rating the strength of each reason: - The CEO's repeated emphasis on the necessity of changes: Strong. Public statements by company leaders often give insight into the company's strategic direction, and Huffman's consistent messaging indicates a strong resolve. - Stance against negotiation: Moderate to Strong. While the company has been clear about its plans, there's always room for strategic shifts if new data or circumstances arise. - Concern about setting a precedent: Moderate. While businesses aim to maintain a strong negotiating position, user and community backlash of this scale can prompt a reassessment of policies. - Ongoing widespread protests: Moderate to Strong. Although Reddit seems determined, it is ultimately a community-driven platform, and sustained backlash can be very influential. - Reaction from advertisers and partners: Moderate. Financial concerns carry significant weight, but it is unclear how partners are responding. - Loss of user engagement: Moderate. The potential long-term impact on user engagement poses a risk to Reddit's health as a platform, though the company may be betting on absorbing or outlasting these impacts.5. Aggregate your considerations:Considering the CEO's firm stance, the company's clear positioning against changes, and the desire to avoid setting a precedent, it seems probable that Reddit will follow through with the API fee introduction as planned. Nonetheless, if the protests continue to escalate and if there is significant pushback from advertisers and a clear decline in user engagement, there could be a strategic reconsideration. However, such developments would have to outweigh the company's current commitment to the pricing changes.6. Output your answer:Considering all the information and arguments listed above, my confidence level in Reddit not announcing changes or a delay leans towards 'no'. It seems that while there may be pressure, the company is quite steadfast in its position.0.75-Response from forecaster 2:<omitted for brevity>Answer: 0.1-Response from forecaster 3:<omitted for brevity>7. Final answer:0.35-Response from forecaster 4:<omitted for brevity>7. Output final answer:0.25---Your goal is to aggregate the information and make a final prediction.Instructions:1. Provide reasons why the answer might be no.{ Insert your thoughts here }2. Provide reasons why the answer might be yes.{ Insert your thoughts here }3. Aggregate your considerations.{ Insert your aggregated considerations here }4. Output your prediction (a number between 0 and 1) with an asterisk at the beginning and end of the decimal.{ Insert the probability here }答案是：
1. Provide reasons why the answer might be no. - Reddit's CEO, Steve Huffman, has consistently emphasized the necessity of the API fee changes for the company's profitability and sustainability. His firm stance suggests a strong commitment to the new pricing structure. - The company has indicated a clear stance against negotiation or alteration of the planned pricing, suggesting they are prepared to withstand a significant amount of community backlash without changing course. - From a business perspective, changing the pricing plan now could set a precedent where Reddit might be seen as bending under pressure, potentially weakening its position in future negotiations or policy rollouts. - Huffman has downplayed the impact of the protests, suggesting that he believes they do not represent a significant enough portion of the user base to warrant a change in policy.2. Provide reasons why the answer might be yes. - The ongoing, widespread protest involving thousands of subreddits and key developers expressing outrage could be a significant pressure point, especially if the protest escalates or continues for a prolonged period, potentially harming Reddit's brand image and user satisfaction. - If advertisers or other revenue-generating partners express concerns or withdraw their business due to the negative press and user reactions, Reddit may be financially incentivized to revise or delay the pricing changes. - The potential shutdown of popular third-party apps and services could lead to a significant loss of user engagement, which Reddit may seek to mitigate by altering the course of their decision before it causes irreparable damage to their user base.3. Aggregate your considerations.Considering the CEO's firm stance, the company's clear positioning against changes, and the desire to avoid setting a precedent, it seems probable that Reddit will follow through with the API fee introduction as planned. Nonetheless, if the protests continue to escalate and if there is significant pushback from advertisers and a clear decline in user engagement, there could be a strategic reconsideration. However, such developments would have to outweigh the company's current commitment to the pricing changes.4. Output your prediction (a number between 0 and 1) with an asterisk at the beginning and end of the decimal.0.35因此，根据我们的“专家”群体，Reddit 在 2023 年 7 月 1 日之前宣布更改或延迟其拟议的 API 费用定价的概率为 0.35。
结果用于衡量 RAG 系统性能的基准是人群。更具体地说，研究人员将RAG系统的预测与预测市场平台在参与者提交预测时计算的聚合预测进行了比较。
为了评估这些预测与实际结果的接近程度，采用 Brier 分数作为评估指标。它计算事件发生的预测概率（范围从 0 到 1）与实际结果（即 0 或 1）之间的差值，将此差值平方，并在所有预测中取平均值。Brier 分数越低表示预测越准确。例如，完全随机的预测变量得分为 0.25，而完全准确的预测变量得分为 0。
为问题分配一个类别。下表比较了 RAG 系统的 Brier 分数与人群对不同类别的预测：
我们看到，对于每个类别和所有类别，RAG 系统（“我们的”）与 Crowd 相比表现不佳。
然而，研究人员发现RAG系统在某些情况下表现优于人群。特别是：
首先，当人群的预测表现出高度的不确定性时，我们的系统明显优于人群。具体来说，当人群的预测在 .3 到 .7 之间时，我们的 Brier 分数是 .199，而人群的预测是 .246。然而，我们的系统在人群高度确定的问题上表现不佳，可能是因为它很少输出低概率（图 4b）。我们假设这源于我们的模型由于其安全训练而倾向于对冲预测（参见图 17 的定性示例）。为了支持这一点，我们的系统在人群预测在0或1的0.05以内的问题上实现了7%的准确率，但Brier分数差了0.04。
此外，研究人员发现他们的RAG系统可以补充人群的预测：
将系统的预测与人群相结合，使用加权平均 4 倍的人群权重，我们发现这在验证集上是最佳的，将整个测试集的总体 Brier 分数从 0.149 提高到 0.146（表 3，顶行）。
结论很明显，虽然本文中描述的RAG系统可能不会优于人群预测，但它们非常接近。这种性能上的接近凸显了RAG作为在复杂场景中增强决策的强大工具的潜力。RAG系统处理和分析大量数据集的能力，通过进一步的特定领域微调，可以提高预测准确性。
此外，这些系统可以通过为预测提供强大的、数据驱动的基础，从根本上改变各个部门的决策方式。想象一下，在未来，由先进的 LLM 和检索技术提供支持并连接到广泛的公共和私人知识库的自主代理将参与预测市场。这可能会带来让人想起金融市场中高频交易机器人的效率。这些进步可以使数据驱动的决策民主化，使高质量的预测更容易获得，并减少人类分析中固有的偏见。
