Generative AI – Doodle Coach App

In this tutorial, we will create an app that uses the Google Gemini API to describe a hand-drawn picture and suggest improvements.

You can download the source code (aia file) from the download section at the bottom of the page.

For calling the Google Gemini API, we’ll use Google Apps Script (GAS). The GAS script is written with Google Gemini 2.5 Pro.

The Gemini API created in Google AI Studio can be used for free with Gemini 2.5 Pro within the following limits:

  • Requests per minute (RPM): 5
  • Tokens per minute (input) (TPM): 250,000
  • Requests per day (RPD): 100

What You Will Learn in This Tutorial

  • How to create an API with GAS
  • How to write a GAS script using Gemini
  • How to integrate with the Gemini API

Components Used

Button, Label, TextBox, Canvas, Image, Web, ImageBase641

Blocks Used

Global Variable, List, if then else, Dictionaries

Setting up Google Apps Script

Set up the server-side program in Google Apps Script to make API calls.

Generate a Google Apps Script Using Gemini 2.5 Pro

You can use Gemini 2.5 Pro from here even with a free account.

Prompt

I want to add descriptions to the drawings made on the App Inventor canvas and have it suggest improvements. I’ll use Google Apps Script and Google AI Studio. The drawing on the canvas will be base64-encoded with an extension and uploaded in JSON format. The result will be returned in JSON. The API key will be read from the script properties. I’ll handle the App Inventor part myself, so please explain the rest.

Response

Of course. You can set up a Google Apps Script web app to act as an intermediary. It will receive the image data from App Inventor, forward it to the Google AI API for analysis, and then return the structured results.
Here’s a step-by-step guide to setting up the Google Apps Script and handling the AI interaction.

Overview of the Process
1. Get Google AI API Key: You'll get a free API key from Google AI Studio.
2. Create a Google Apps Script: This script will contain the code to handle incoming requests.
3. Store API Key Securely: You'll save your API key in the script's properties so it's not exposed in the code.
4. Write the doPost Function: This function will receive the image from App Inventor, call the Gemini API, parse the result, and send it back.
5. Deploy as a Web App: You'll publish the script so App Inventor can send it requests.
## Step 1: Get Your Google AI API Key 🔑
First, you need an API key to use Google's generative models.
1. Go to Google AI Studio.
2. Sign in with your Google account.
3. Click on "Get API key" and then "Create API key in new project".
4. Copy the generated key and save it somewhere safe for the next step.
## Step 2: Set Up the Google Apps Script Project
This script will be your backend service.
1. Go to script.google.com and create a New project.
2. Give your project a name, like "App Inventor AI Helper".
3. Store your API key securely:
  Click on the Project Settings ⚙️ icon on the left sidebar.
  Scroll down to the Script Properties section and click "Add script property".
  For the Property name, enter GEMINI_API_KEY.
  For the Value, paste the API key you got from Google AI Studio.
  Click "Save script properties". This keeps your key out of the code itself.
## Step 3: Write the Apps Script Code ✍️
Now, replace the default code in the Code.gs file with the following script. This code handles incoming POST requests, which is how App Inventor's Web component sends data.
function doPost(e) {
  try {
    // Parse the JSON data sent from App Inventor
    const postData = JSON.parse(e.postData.contents);
    const base64Image = postData.image; // "image" key expected from App Inventor

    // Load API key from script properties
    const API_KEY = PropertiesService.getScriptProperties().getProperty('API_KEY');
    if (!API_KEY) {
      return createErrorResponse('API key is not set.');
    }

    // Gemini API endpoint
    const API_URL = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:generateContent?key=${API_KEY}`;
  
    // Request body to send to Gemini API
    const payload = {
      "contents": [{
        "parts": [{
          "text": "Please describe this drawing and suggest improvements. Separate description and suggestions clearly. Limit to 100 characters."
        }, {
          "inline_data": {
            "mime_type": "image/png", // App Inventor Canvas produces PNG format
            "data": base64Image
          }
        }]
      }]
    };

    // Request settings for Gemini API
    const options = {
      'method': 'post',
      'contentType': 'application/json',
      'payload': JSON.stringify(payload),
      'muteHttpExceptions': true // Capture error responses instead of throwing exceptions
    };

    // Send request to Gemini API
    const response = UrlFetchApp.fetch(API_URL, options);
    const responseCode = response.getResponseCode();
    const responseBody = response.getContentText();

    if (responseCode !== 200) {
       return createErrorResponse(`API request error: ${responseCode} ${responseBody}`);
    }
    
    // Parse response from Gemini API
    const resultJson = JSON.parse(responseBody);
    
    // Extract generated text from response structure
    let generatedText = "Failed to retrieve analysis result.";
    if (resultJson.candidates && resultJson.candidates.length > 0 &&
        resultJson.candidates[0].content && resultJson.candidates[0].content.parts &&
        resultJson.candidates[0].content.parts.length > 0) {
      generatedText = resultJson.candidates[0].content.parts[0].text;
    } else {
       // Unexpected response format
       return createErrorResponse(`Unexpected API response format: ${responseBody}`);
    }

    // Build JSON response for App Inventor
    const output = JSON.stringify({
      "status": "success",
      "result": generatedText
    });

    // Return result in JSON format
    return ContentService.createTextOutput(output).setMimeType(ContentService.MimeType.JSON);

  } catch (error) {
    // Catch unexpected script errors
    return createErrorResponse(`Script error: ${error.toString()}`);
  }
}

/**
 * Helper function to create an error response
 * @param {string} message - Error message
 * @return {ContentService.TextOutput} - JSON object containing error information
 */
function createErrorResponse(message) {
  const errorOutput = JSON.stringify({
    "status": "error",
    "message": message
  });
  return ContentService.createTextOutput(errorOutput).setMimeType(ContentService.MimeType.JSON);
}
## Step 4: Deploy the Script as a Web App 🚀
For App Inventor to be able to call your script, you must deploy it.
1. Click the Deploy button at the top right.
2. Select New deployment.
3. Click the gear icon next to "Select type" and choose Web app.
4. In the configuration settings:
  Description: (Optional) "AI image analyzer for App Inventor".
  Execute as: Me (Your Google Account).
  Who has access: Anyone. This is critical. It makes your API public, but it's necessary for App Inventor to access it without a login.
5. Click Deploy.
6. You'll be asked to Authorize access. Follow the prompts, choosing your Google account, and click "Allow" on the warning screen (it's safe, as you wrote the code).
7. After authorizing, a Deployment ID and Web app URL will appear. Copy the Web app URL. This is the URL you will use in your App Inventor project's Web component.

App Inventor App

From the [Projects] menu, select [Start new project] and name it “DoodleCoach.

Designer

Set the title of Screen1 to “DoodleCoach“.

Download the uk.co.metricrat.imagebase64v2.aix extension from https://metricrat-hosting.web.app/files/uk.co.metricrat.imagebase64v2.aix , upload it to the Extensions palette, and then add it to the Viewer.

From the User Interface palette, add a Label, a TextBox, another Label, and an Image to the Viewer in this order. Their names will be Label1, TextBox1, Label2, and Image1, respectively.

Add a Canvas from the Drawing and Animation palette below Label2. Its name will be Canvas1.

From the Layout palette, add a HorizontalArrangement component below Image1. Inside it, add three Buttons from the User Interface palette. Name them HorizontalArrangement1, Button1, Button2, and Button3.

Finally, add a Web component from the Connectivity palette. Its name will be Web1.

Set the properties as follows.

  • Label1: FontBold = Checked, FontSize = 20, Text = Analysis
  • TextBox1:  BackgroundColor = Gray, Height = 20%, Width = Fill Parent, TextColor = White, MultiLine = Checked
  • Label2: FontBold = Checked, FontSize = 20, Text = Drawing
  • Canvas1: Height = Fill Parent, Width = Fill Parent
  • Image1: Visible = Unchecked
  • HorizontalArrangement1: AlignHorizontal = Center, Width = Fill Parent
  • Button1: Text = Analyze
  • Button2: Text = Erase
  • Button3: Text = Clear
  • Web1: No Change

Blocks Editor

Global Variables

From the Variables drawer in the Built-in Blocks, drag and drop “initialize global variable name to" blocks. Rename the variable names as shown below and set their initial values.

  • postUrl: The URL of the web app. Enter the Web App URL copied from the Google Apps Script settings.
  • base64Encoded: A variable to store the base64-encoded result
  • responseDictionary: A variable to store the dictionary obtained by decoding the received JSON text
  • status: The status string in the received JSON text, either success or error.
  • analyzing: The string displayed during analysis
  • erase: A variable for eraser mode, true or false
  • penColor: The color of the pen used for drawing

Screen1.Initialize

To initialize the app when it starts, click the Screen1 panel on the left side of the Blocks editor to open it, then drag and drop the “when Screen1.Initialize do” block. Inside it, set the URL of Web1 to the postUrl variable, enable the Analyze button, and set the PaintColor to the penColor.

Drawing

Drag and drop the “Canvas1.DrawLine” block from the Canvas1 panel into the “when Canvas1.Dragged” block. Since the Canvas.Dragged event occurs repeatedly while the user is moving their finger across the screen, draw a line from the previous X, previous Y to the current X, current Y.

Now you can draw on the screen.

When “Analyze" button is clicked

After clicking the Analyze button, save the image drawn on Canvas1 and display it in Image1 (note that Image1 is not visible because its Visible property is unchecked), then encode it in Base64. Next, display “Analyzing” in TextBox1 and disable the Analyze button.

AfterImageBase64

When the Base64 encoding is complete, the ImageBase641.AfterImageBase64 event is triggered. Since the Gemini API throws an error if the Base64-encoded string contains line breaks, remove the newline characters (\n) from the Base64-encoded string. Then, create a dictionary with image as the key and the Base64-encoded string as the value, call Web1’s JSON object encoding, and post the JSON text to Web1 to invoke GAS.

After receiving result from GAS

When the result returns from GAS, the “when Web1.GotText” event is triggered. In this event, decode the received JSON text and display the recognition result. Since recognition may not always succeed, error handling is also implemented.

When “Erase" button is clicked

If the eraser variable is false, make the Eraser button text red to indicate it’s selected, set the eraser variable to true, and change the canvas drawing color to white. Otherwise, set the Eraser button text back to black, set the eraser variable to false, and change the canvas drawing color to the color of the currently selected color button.

When “Clear" button is clicked

When the Clear button is clicked, clear Canvas1 and TextBox1.

Download

Source Code: Click here to download the source code

AI,API,Cloud,Gemini,Generative AI,Google Apps Script,Intermediate