Haystack
获取企业支持
开始使用
📘 **TELUS Agriculture & Consumer Goods** 如何通过 **Haystack Agents** 转变促销交易

使用 Gemini 进行函数调用和多模态 QA


在 Colab 中打开 下载
最后更新：2025 年 8 月 8 日

作者：Tuana Celik：Twitter，LinkedIn，Tilde Thurium：Twitter，LinkedIn 和 Silvano Cerza：LinkedIn

这是一个 notebook，展示了如何将 Gemini + Vertex AI 与 Haystack 结合使用。

要在 Gemini 开发者 API 上将 Gemini 模型与 Haystack 一起使用，请查看我们的文档。

Gemini 是 Google 的最新模型。您可以在此处阅读更多关于其功能的信息。

安装依赖项

作为先决条件，您需要设置一个 Google Cloud 项目，该项目具有 Vertex AI 和 Gemini 的访问权限。

有用资源

完成以上步骤后，您只需要在此 Colab 中进行身份验证。

首先，我们需要安装我们的依赖项，包括Google Gen AI集成

! pip install haystack-ai google-genai-haystack trafilatura

让我们使用应用程序默认凭据 (ADC) 登录。有关更多信息，请参阅官方文档。

from google.colab import auth

auth.authenticate_user()

请记住将 project_id 变量设置为一个有效的项目 ID，您拥有足够的授权来使用 Gemini。我们将在整个示例中使用此项目 ID！

要查找您的项目 ID，您可以在GCP 资源管理器中找到它，或者通过在终端中运行 gcloud projects list 来本地查找。有关 gcloud CLI 的更多信息，请参阅官方文档。

project_id = input("Enter your project ID:")

使用 gemini-2.5-flash

回答问题

现在我们已经设置好了一切，我们可以创建我们的GoogleGenAIChatGenerator实例。此组件同时支持 Gemini 和 Vertex AI。在此演示中，我们将设置 api="vertex"，并将我们的 project_id 作为 vertex_ai_project 传递。

from haystack_integrations.components.generators.google_genai import GoogleGenAIChatGenerator

gemini = GoogleGenAIChatGenerator(model="gemini-2.5-flash", api="vertex", vertex_ai_project=project_id, vertex_ai_location="europe-west1")

让我们从问一些简单的问题开始。

此组件期望 ChatMessage 对象的列表作为输入到 run() 方法。您可以通过消息传递文本或函数调用。

from haystack.dataclasses import ChatMessage

messages = [ChatMessage.from_user("What is the most interesting thing you know?")]
result = gemini.run(messages = messages)
for answer in result["replies"]:
    print(answer.text)

The most interesting thing I know, and one of the most profound mysteries in all of science, is that **about 95% of the universe is made of something we cannot see or directly detect: dark energy and dark matter.**

Imagine if 95% of the world around you was completely invisible and unknown, yet it fundamentally shaped everything you *could* see. That's our current situation with the cosmos.

*   **Dark Matter** makes up about 27% of the universe. We know it exists because of its gravitational effects – it holds galaxies together, prevents clusters from flying apart, and influenced the large-scale structure of the early universe. But it doesn't absorb, reflect, or emit light, making it "dark." We don't know what particles it's made of.
*   **Dark Energy** makes up about 68% of the universe. It's an even bigger enigma. We infer its existence because it's responsible for the accelerated expansion of the universe. It's essentially pushing the cosmos apart, overcoming the attractive force of gravity. Its nature is one of the biggest unsolved problems in physics.

This means that all the stars, planets, galaxies, gas, and dust – everything we can observe with telescopes – makes up only about 5% of the universe's total mass-energy content. The vast majority of reality is utterly mysterious, and understanding it is one of the greatest scientific quests of our time. It dictates the fate of the cosmos itself.

回答关于图片的问题

让我们尝试一些不同的！gemini-2.5-flash 也可以处理图片，让我们看看它能否回答一些关于机器人图片的问题 👇

我们将为此示例下载一些图片。🤖

from haystack.dataclasses import ImageContent

urls = [
    "https://upload.wikimedia.org/wikipedia/en/5/5c/C-3PO_droid.png",
    "https://platform.theverge.com/wp-content/uploads/sites/2/chorus/assets/4658579/terminator_endoskeleton_1020.jpg",
    "https://upload.wikimedia.org/wikipedia/en/3/39/R2-D2_Droid.png",
]

images = [ImageContent.from_url(url) for url in urls]

messages = [ChatMessage.from_user(content_parts=["What can you tell me about these robots? Be short and graceful.", *images])]
result = gemini.run(messages = messages)
for answer in result["replies"]:
    print(answer.text)

These are iconic robots from popular culture:

1.  **C-3PO:** A refined protocol droid, fluent in countless languages, known for his golden appearance and nervous demeanor.
2.  **T-800 Endoskeleton:** A formidable, relentless combat machine, skeletal and chilling, from a dystopian future.
3.  **R2-D2:** A courageous and resourceful astromech, full of personality, who communicates in beeps and whistles.

使用 gemini-2.5-flash 进行函数调用

使用 gemini-2.5-flash，我们还可以使用函数调用！所以让我们看看如何做到这一点 👇

让我们看看是否可以构建一个系统，该系统可以根据自然语言提出的问题来运行 get_current_weather 函数。

首先，我们创建函数定义和工具（在文档中了解更多关于工具的信息）。

出于演示目的，我们只是创建一个 get_current_weather 函数，该函数返回一个对象，该对象将始终告诉我们“晴朗，气温 21.8 度”… 如果是摄氏度，那真是个好天气！☀️

from haystack.components.tools import ToolInvoker
from haystack.tools import tool
from typing import Annotated

@tool
def get_current_weather(
    location: Annotated[str, "The city for which to get the weather, e.g. 'San Francisco'"] = "Munich",
    unit: Annotated[str, "The unit for the temperature, e.g. 'celsius'"] = "celsius",
):
  return {"weather": "sunny", "temperature": 21.8, "unit": unit}

user_message = [ChatMessage.from_user("What is the temperature in celsius in Berlin?")]
replies = gemini.run(messages=user_message, tools=[get_current_weather])["replies"]
print(replies)

[ChatMessage(_role=<ChatRole.ASSISTANT: 'assistant'>, _content=[TextContent(text=''), ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None)], _name=None, _meta={'model': 'gemini-2.5-flash', 'finish_reason': 'stop', 'usage': {'prompt_tokens': 53, 'completion_tokens': 10, 'total_tokens': 126}})]

看！现在我们收到了一条包含一些有趣信息的消息。我们可以使用这些信息调用本地的真实函数。

让我们正是这样做，并将结果传回 Gemini。

tool_invoker = ToolInvoker(tools=[get_current_weather])
tool_messages = tool_invoker.run(messages=replies)["tool_messages"]
print(tool_messages)

messages = user_message + replies + tool_messages

res = gemini.run(messages = messages)
print(res["replies"][0].text)

[ChatMessage(_role=<ChatRole.TOOL: 'tool'>, _content=[ToolCallResult(result="{'weather': 'sunny', 'temperature': 21.8, 'unit': 'celsius'}", origin=ToolCall(tool_name='get_current_weather', arguments={'unit': 'celsius', 'location': 'Berlin'}, id=None), error=False)], _name=None, _meta={})]
The temperature in Berlin is 21.8°C and it's sunny.

天气似乎很好，阳光明媚，记得戴上太阳镜。😎

使用 gemini-2.5-flash 构建完整的检索增强生成（RAG）流水线

作为最后的练习，让我们将 GoogleGenAIChatGenerator 添加到一个完整的 RAG 流水线中。在下面的示例中，我们正在构建一个 RAG 流水线，该流水线使用 gemini-2.5-flash 对网络进行问题解答。

from haystack.components.fetchers.link_content import LinkContentFetcher
from haystack.components.converters import HTMLToDocument
from haystack.components.preprocessors import DocumentSplitter
from haystack.components.rankers import SentenceTransformersSimilarityRanker
from haystack.components.builders.chat_prompt_builder import ChatPromptBuilder
from haystack import Pipeline

fetcher = LinkContentFetcher()
converter = HTMLToDocument()
document_splitter = DocumentSplitter(split_by="word", split_length=50)
similarity_ranker = SentenceTransformersSimilarityRanker(top_k=3)
gemini = GoogleGenAIChatGenerator(model="gemini-2.5-flash", api="vertex", vertex_ai_project=project_id, vertex_ai_location="europe-west1")

prompt_template = [ChatMessage.from_user("""
According to these documents:

{% for doc in documents %}
  {{ doc.content }}
{% endfor %}

Answer the given question: {{question}}
Answer:
""")]
prompt_builder = ChatPromptBuilder(template=prompt_template)

pipeline = Pipeline()
pipeline.add_component("fetcher", fetcher)
pipeline.add_component("converter", converter)
pipeline.add_component("splitter", document_splitter)
pipeline.add_component("ranker", similarity_ranker)
pipeline.add_component("prompt_builder", prompt_builder)
pipeline.add_component("gemini", gemini)

pipeline.connect("fetcher.streams", "converter.sources")
pipeline.connect("converter.documents", "splitter.documents")
pipeline.connect("splitter.documents", "ranker.documents")
pipeline.connect("ranker.documents", "prompt_builder.documents")
pipeline.connect("prompt_builder.prompt", "gemini")

让我们尝试让 Gemini 告诉我们 Haystack 是什么以及如何使用它。

question = "What do graphs have to do with Haystack?"
result = pipeline.run({"prompt_builder": {"question": question},
                   "ranker": {"query": question},
                   "fetcher": {"urls": ["https://haystack.com.cn/blog/introducing-haystack-2-beta-and-advent"]}})

for message in result["gemini"]["replies"]:
  print(message.text)

In Haystack, pipelines are structured as graphs. Specifically, Haystack 1.x pipelines were based on Directed Acyclic Graphs (DAGs). In Haystack 2.0, the "A" (acyclic) is being removed from DAG, meaning pipelines can now branch out, join, and cycle back to other components, allowing for more complex graph structures that can retry or loop.

现在您已经了解了 Gemini 的一些功能以及如何将其集成到 Haystack 中。如果您想了解更多信息，请查看 Haystack 的文档或教程。