Gemini JSON Mode Not Working

[dcb6]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

const model = new ChatGoogleGenerativeAI({
      model: "gemini-1.5-flash-001",
      apiKey: env.GOOGLE_GEMINI_API_KEY,
    });

const format = z.object({
  joke: z.string(),
});

const modelWithParser = this.model.withStructuredOutput(
  format,
  {
    method: "jsonMode",
  },
);

const result = await modelWithParser.invoke("Tell me a joke");

Error Message and Stack Trace (if applicable)

Error: ChatGoogleGenerativeAI only supports "functionCalling" as a method.
at ChatGoogleGenerativeAI.withStructuredOutput

Description

Despite this closed ticket that seems to mention support for Gemini structured outputs via JSON mode, I still get that error. It seems maybe PR that closed it just implemented support for a general JSON response type?

System Info

@langchain/openai: >=0.1.0 <0.4.0 openapi-types: ^12.1.3
@langchain/textsplitters: >=0.0.0 <0.2.0 p-retry: 4
js-tiktoken: ^1.0.12 uuid: ^10.0.0
js-yaml: ^4.1.0 yaml: ^2.2.1
jsonpointer: ^5.0.1 zod-to-json-schema: ^3.22.3
langsmith: ^0.2.0 zod: ^3.22.4

[dosubot]

Hey @dcb6! I'm here to help you with any bugs, questions, or contributions. If you need any assistance, feel free to ask. Dosu

The issue you're encountering with the Gemini JSON mode in LangChain.js is due to the fact that JSON mode is not supported by the ChatGoogleGenerativeAI model. The error message you received, "ChatGoogleGenerativeAI only supports 'functionCalling' as a method," confirms this limitation. The model currently supports function calling but not JSON mode ^[1].

There is ongoing development to support JSON mode, as indicated by test cases demonstrating its use. However, according to the latest documentation, JSON mode is still not supported ^[2][3].

For structured output, you can use the withStructuredOutput method with function calling, which allows you to define a schema using zod to specify the structure of the output. This method is currently the alternative to achieve structured outputs with the ChatGoogleGenerativeAI model ^[1][4].

To continue talking to Dosu, mention @dosu.

[dcb6]

@dosu If I stop trying to use JSON mode, I just get the error mentioned here:

#6901

So right now, I have no way to get structured output with gemini. Would love to sees JSON mode supported!

[dosubot]

To achieve structured output with the Gemini model in LangChain.js without using JSON mode, you can use the withStructuredOutput() method with function calling. This method allows you to define a schema using zod and ensures the model's output conforms to this schema. Here's an example of how you can implement it:

import { z } from "zod";
import { ChatOpenAI } from "@langchain/openai";

// Define the schema using Zod
const ResponseFormatter = z.object({
  joke: z.string().describe("A joke"),
});

// Initialize the model
const model = new ChatOpenAI({
  modelName: "gpt-4",
  temperature: 0,
});

// Bind the schema to the model
const modelWithStructure = model.withStructuredOutput(ResponseFormatter);

// Invoke the model to produce structured output
const structuredOutput = await modelWithStructure.invoke("Tell me a joke");

// Get back the object
console.log(structuredOutput);
// { joke: "Why don't scientists trust atoms? Because they make up everything!" }

This approach allows you to enforce structured output without relying on JSON mode, which is currently not supported by the Gemini model. If you encounter errors related to function calling, ensure that your model and schema setup are correct and compatible with the function calling method ^[1].

To continue talking to Dosu, mention @dosu.

[tonyabracadabra]

this is so broken and can't believe no one is fixing it yet, switched to generateObject and never regret it!

[jacoblee93]

CC @bracesproul @afirstenberg can you have a look?