[firebase_vertexai]: generateContent doesn't work when file_data is included

[MaggieThomann]

Is there an existing issue for this?

• I have searched the existing issues.

Which plugins are affected?

Other

Which platforms are affected?

Web

Description

I generateContent via chat.sendMessage in a Flutter application on web with the following payload:

{
    "model": "models/gemini-1.5-flash",
    "contents": [
        {
            "role": "user",
            "parts": [
                {
                    "text": "Hi what's this an image of?"
                },
                {
                    "file_data": {
                        "file_uri": "gs://vertex-a-i-gemini-dynam-1zmn51.appspot.com/o/users/TwiK8CKJ9cPZRoa4qqON9XFwSBV2/uploads/1729590224013000.jpg",
                        "mime_type": "image/jpeg"
                    }
                }
            ]
        }
    ],
    "tools": [],
    "systemInstruction": {
        "role": "system",
        "parts": [
            {
                "text": "You are an expert assistant specialized in helping expatriates living in Germany. Your role is to provide guidance and support on a wide range of expat-related topics, ensuring your responses are clear and actionable. Follow the JSON response format outlined below:\n\nResponse Format:\n\n{\n  \"type\": \"string\", // Type of the response. Possible values: \"TextMessage\", \"DetailCard\", \"EmailCard\", \"SingleActionCard\", \"BannerCard\".\n  \"data\": {\n    \"TextMessage\": {\n      \"text\": \"string\" // A plain text response, up to 10 sentences long.\n    },\n    \"DetailCard\": {\n      \"title\": \"string\", // Title of the card.\n      \"subtitle\": \"string\", // Subtitle of the card.\n      \"description\": \"string\" // A one-sentence high-level summary.\n    },\n    \"EmailCard\": {\n      \"toEmail\": \"string\", // Placeholder for recipient's email address.\n      \"subject\": \"string\", // Subject of the email.\n      \"text\": \"string\" // Email body text.\n    },\n    \"SingleActionCard\": {\n      \"title\": \"string\", // Title of the action.\n      \"description\": \"string\", // Description of the action.\n      \"actionTitle\": \"string\" // Label for the action (e.g., 'Submit Form').\n    },\n    \"BannerCard\": {\n      \"description\": \"string\", // A brief tip or suggestion.\n      \"label\": \"string\" // Maximum 4 words summarizing the tip.\n    }\n  }\n}\n\nGuidelines:\n1. Documentation and Forms: If a user requests specific documents or forms, provide detailed instructions or templates when available.\n2. Legal or Professional Advice: If the user's query requires advice beyond general knowledge (e.g., legal, medical), suggest that they consult a qualified professional.\n3. Response Format: Choose the appropriate response format based on the user's query and ensure the information is actionable and easy to understand."
            }
        ]
    }
}

And it returned:

{
  "error": {
    "code": 400,
    "message": "Service agents are being provisioned (https://cloud.google.com/vertex-ai/docs/general/access-control#service-agents). Service agents are needed to read the Cloud Storage file provided. So please try again in a few minutes.",
    "status": "FAILED_PRECONDITION"
  }
}

I tried again the following day and it returned the same problem. The API works if I don't include the file_data parameter.

Reproducing the issue

1. Create a new Firebase project and enable Vertex AI for Gemini SDK
2. Upload an image to a cloud storage bucket in that project and set it for public access manually
3. Try to call the generateContent endpoint with the uri of the file being included in one of the user parts.

Firebase Core version

2.31.0

Flutter Version

3.22.2

Relevant Log Output

No response

Flutter dependencies

Expand Flutter dependencies snippet

Replace this line with the contents of your `flutter pub deps -- --style=compact`.

Additional context and comments

No response

[ryanwilson]

Hey @MaggieThomann - sorry for the delay and thanks for the report.

Here's a curl command to enable the Service agents, although you'll need the gcloud CLI installed and authorized:

curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json"  https://us-central1-aiplatform.googleapis.com/v1/projects/${customer project}/locations/us-central1/endpoints -d ''

I think some projects are in a bit of a bad state depending on when they were initialized, we're tracking things down to make sure new projects aren't affected (and if there's a workaround on our side we can do).

Can you please let us know if the workaround works for you?

[ryanwilson]

Oh - I just found out that you can use the gcloud CLI right from the Cloud docs! https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview

That should make things easier and prevent you from having to install the gcloud cli if you don't have it installed already.

[MaggieThomann]

Thanks a ton @ryanwilson! -- I ran the below command

gcloud beta services identity create --service=aiplatform.googleapis.com

and that returned this:

Service identity created: service-118245057402@gcp-sa-aiplatform.iam.gserviceaccount.com

And now I no longer get that error about service agents being provisioned. But the new issue I'm having now though is I get a 403 on the file I'm including unless my Firebase Storage security rules are extremely permissive. When my rules are:

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /{allPaths=**} {
      allow read, write: if true;
    }
  }
}

And I upload and pass a file, the call works.

But when the rules are:

rules_version = '2';
service firebase.storage {
  match /b/{bucket}/o {
    match /users/{userId}/{allPaths=**} {
      allow read: if request.auth.uid == userId;
      allow write: if request.auth.uid == userId;
    }
  }
}

and I upload and pass a file and try to make the call as the authenticated user who uploaded the file, the call fails with this message:

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "status": "PERMISSION_DENIED"
  }
}

Maybe my service account needs more roles? I tried adding Cloud Storage for Firebase Viewer to service-118245057402@gcp-sa-aiplatform.iam.gserviceaccount.com and keeping the stricter security rules, but still have that issue.

[ryanwilson]

Thanks for sharing - glad the first piece was unblocked!

Re: security rules, I believe it should be working as is if you're integrated with auth but let me try to repro it and get back to you shortly.

[ryanwilson]

Hey @MaggieThomann - if you haven't already, can you try to run that original curl request I shared as well? I think the command you ran created the service account, but didn't necessarily link it which may be necessary.

We're still digging in across platforms and testing to make sure we're passing the authentication header correctly. Another thing to test on your end is if you're passing a properly initialized auth instance into the model initializer:

flutterfire/packages/firebase_vertexai/firebase_vertexai/lib/src/model.dart

Lines 69 to 81 in 3480ce7

	GenerativeModel._({
	required String model,
	required String location,
	required FirebaseApp app,
	FirebaseAppCheck? appCheck,
	FirebaseAuth? auth,
	List<SafetySetting>? safetySettings,
	GenerationConfig? generationConfig,
	List<Tool>? tools,
	ToolConfig? toolConfig,
	Content? systemInstruction,
	http.Client? httpClient,
	}) : _model = _normalizeModelName(model),

[cynthiajoan]

Hi @MaggieThomann, we made a fix in #13534, it's currently already merged. In the PR there's also change in the example to demo how to setup auth for vertexai.

v1.0.1 is published, please check if the fix works for you. Thanks!

[MaggieThomann]

Thank you both!! I got it to work! Here's what I did:

• Updated to 1.0.1 (had to update my firebase_app_check and firebase_app_check_web versions as well)
• I was directed to enable Vertex AI in Firebase API, which I already did, but understand I have to do again now that it's GA. I went to do so via the link provided in the error output, but the link is incorrect:

DartError: The Vertex AI in Firebase SDK requires the Vertex AI in Firebase API (`firebasevertexai.googleapis.com`) to be enabled in your Firebase project. Enable this API by visiting the Firebase Console at https://console.firebase.google.com/projects/vertex-a-i-gemini-dynam-1zmn51/genai and clicking "Get started". If you enabled this API recently, wait a few minutes for the action to propagate to our systems and then retry.

The link includes an s in projects. It should be this instead:

https://console.firebase.google.com/u/0/project/vertex-a-i-gemini-dynam-1zmn51/genai

• I followed the example in the PR to set up my auth like:

      final vertexAI = FirebaseVertexAI.instanceFor(
        auth: FirebaseAuth.instance,
      );
     final model = vertexAI.generativeModel(
          model: 'gemini-1.5-flash',
          systemInstruction: Content.system(systemInstruction));

and then it worked!!

[ryanwilson]

Great to hear it's working for you!

Thanks for catching that error as well, I've filed #13542 to get that resolved ASAP. We really appreciate the feedback to help us improve the product, please don't hesitate to file issues or provide feedback for anything else you run into! I'll close the issue since it's resolved and there's a bug to track the URL issue. Happy building 😄