Index pages with the Java Google Indexing API

Requirements
Library
HttpRequestInitializer
Notify to update a URL
Notify to remove a URL
Get information of a URL
Request URL batch indexing
Error handling

The Google Indexing API allows any site owner to directly notify Google when pages are added or removed. This allows Google to schedule pages for a fresh crawl, which can lead to higher quality user traffic. For websites with many short-lived pages like job postings or livestream videos, the Indexing API keeps content fresh in search results because it allows updates to be pushed individually.

Currently, the Indexing API can only be used to crawl pages with either "JobPosting" or "BroadcastEvent embedded in a VideoObject".
https://developers.google.com/search/docs/appearance/structured-data/job-posting
https://developers.google.com/search/docs/appearance/structured-data/video#broadcast-event

Here are some of the things you can do with the Google Indexing API:

Supported	Description
Update a URL	Notify Google of a new URL to crawl or that content at a previously-submitted URL has been updated.
Remove a URL	After you delete a page from your servers, notify Google so that we can remove the page from our index and so that we don't attempt to crawl the URL again.
Get the status of a request	Check the last time Google received each kind of notification for a given URL.
Send batch indexing requests	Instead of submitting index requests for individual URL, you can combine up to 100 URLs to notify Google in one request.

1. Requirements

First you need to register and create a project on Google Cloud Console.

https://console.cloud.google.com/

Enable the "Google Indexing" service for your account.

https://console.cloud.google.com/apis/library/indexing.googleapis.com

Create a credential of type "Service Account" and download it as a JSON file.

Create a Google Service Account

After the above step, you are provided with an email in the format "xxx@yyy.iam.gserviceaccount.com".

On Google Search Console, add the email you got in the step above as owner for your website.

2. Library

<!-- SEE ALL VERSIONS -->
<!-- https://libraries.io/maven/com.google.apis:google-api-services-indexing/versions -->
<dependency>
  <groupId>com.google.apis</groupId>
  <artifactId>google-api-services-indexing</artifactId> 
  <version>v3-rev20230927-2.0.0</version> 
</dependency>  

<!-- https://mvnrepository.com/artifact/com.google.auth/google-auth-library-oauth2-http -->
<dependency>
  <groupId>com.google.auth</groupId>
  <artifactId>google-auth-library-oauth2-http</artifactId>
  <version>1.20.0</version>
</dependency> 
    
<!-- https://mvnrepository.com/artifact/com.google.api-client/google-api-client-gson --> 
<dependency>
  <groupId>com.google.api-client</groupId>
  <artifactId>google-api-client-gson</artifactId>
  <version>2.2.0</version>
</dependency>

<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
<dependency>
    <groupId>commons-io</groupId>
    <artifactId>commons-io</artifactId>
    <version>2.15.1</version>
</dependency>

3. HttpRequestInitializer

To send requests to Google services via the Google Java API, you need to create an HttpRequestInitializer object.

HttpRequestInitializer will initialize the necessary information included in requests each time they are sent. The most important of which is credentials, to ensure that you have access to relevant services and resources.

In the previous step you created a credential and downloaded it as a JSON file. Now we will write a utility class to create an HttpRequestInitializer object.

MyUtils.java

package org.o7planning.googleapis.utils;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;

import org.apache.commons.io.FileUtils;

import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.json.JsonObjectParser;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.util.ObjectParser;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ServiceAccountCredentials;

public class MyUtils {

	public static final String SERVICE_ACCOUNT_FILE_PATH = "/Volumes/New/_gsc/test-google-search-console-key.json";

	private static byte[] serviceAccountBytes;

	public static HttpRequestInitializer createHttpRequestInitializer(String... scopes) throws IOException {
		InputStream serviceAccountInputStream = getServiceAccountInputStream();

		GoogleCredentials credentials = ServiceAccountCredentials //
				.fromStream(serviceAccountInputStream) //
				.createScoped(scopes);

		HttpRequestInitializer requestInitializer = new HttpRequestInitializer() {

			@Override
			public void initialize(HttpRequest request) throws IOException {
				HttpCredentialsAdapter adapter = new HttpCredentialsAdapter(credentials);
				adapter.initialize(request);
				//
				if (request.getParser() == null) {
					ObjectParser parser = new JsonObjectParser(GsonFactory.getDefaultInstance());
					request.setParser(parser);
				}
				//
				request.setConnectTimeout(60000); // 1 minute connect timeout
				request.setReadTimeout(60000); // 1 minute read timeout
			}

		};
		return requestInitializer;
	}

	public static synchronized InputStream getServiceAccountInputStream() throws IOException {
		if (serviceAccountBytes == null) {
			serviceAccountBytes = FileUtils.readFileToByteArray(new File(SERVICE_ACCOUNT_FILE_PATH));
		}
		return new ByteArrayInputStream(serviceAccountBytes);
	}
}

You can also create GoogleCredentials with "API Key" or "OAuth Client ID",...
Java Google APIs commons Credentials

4. Notify to update a URL

The Google Indexing API allows you to submit indexing requests for certain URL on your website. Note that this request will be put into a queue, it will not be executed immediately no matter how many such requests you send. The time you last requested to update a URL will be saved by Google.

Each website will have a daily quota to send requests to Google. If the quota is exceeded, the request will be denied. Quotas may vary for different websites, depending on the quality of your website, specifically user traffic.

https://developers.google.com/search/apis/indexing-api/v3/quota-pricing

A utility class for sending an indexing request to Google.

NotifyToUpdateUrlUtils.java

package org.o7planning.java_14285_google_apis;

import org.o7planning.googleapis.utils.MyUtils;

import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.Indexing.UrlNotifications;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;

public class NotifyToUpdateUrlUtils {

	private static final String MY_APPLICATION = "My-Application";

	/**
	 * <pre>
	 * {
	 *  url: "http://foo.com/page",
	 *  latestUpdate: {
	 *    url: "http://foo.com/page",
	 *    type: "URL_UPDATED",
	 *    notifyTime: "2017-09-31T19:30:54.524457662Z"
	 *  },
	 *  latestRemove: {
	 *    url: "http://foo.com/page",
	 *    type: "URL_DELETED",
	 *    notifyTime: "2017-08-31T19:30:54.524457662Z"
	 *  }
	 * }
	 * </pre>
	 */
	public static UrlNotificationMetadata requestToUpdatePageUrl(String applicationName, String pageUrl)
			throws Exception {
		HttpRequestInitializer httpRequestInitializer //
				= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
		//
		HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
		//

		Indexing indexing = new Indexing.Builder(httpTransport, //
				GsonFactory.getDefaultInstance(), //  
				httpRequestInitializer) //
				.setApplicationName(applicationName) //
				.build();

		UrlNotification content = new UrlNotification();
		content.setType("URL_UPDATED");
		content.setUrl(pageUrl);
		//
		UrlNotifications.Publish publish = indexing.urlNotifications().publish(content);
		PublishUrlNotificationResponse respone = publish.execute();

		UrlNotificationMetadata urlNotification = respone.getUrlNotificationMetadata(); 
		return urlNotification;
	}

	public static UrlNotificationMetadata updatePageUrlIndexing(String pageUrl) throws Exception {
		return requestToUpdatePageUrl(MY_APPLICATION, pageUrl);
	}

	public static void main(String[] args) throws Exception {
		// Test:
		UrlNotificationMetadata metadata = updatePageUrlIndexing(MyTestConstants.URL_TO_TEST);
		System.out.println("Return: " + metadata.toPrettyString());
	}
}

Output:

{
  url: "http://foo.com/page",
  latestUpdate: {
    url: "http://foo.com/page",
    type: "URL_UPDATED",
    notifyTime: "2017-09-31T19:30:54.524457662Z"
  },
  latestRemove: {
    url: "http://foo.com/page",
    type: "URL_DELETED",
    notifyTime: "2017-08-31T19:30:54.524457662Z"
  }
 }

5. Notify to remove a URL

Sometimes there are some URLs on your website that are no longer needed and you want to remove them from Google Search's index. You can send a deletion request to Google, and this URL must return a 404 error code.

NotifyToDeleteUrlUtils.java

package org.o7planning.java_14285_google_apis;

import org.o7planning.googleapis.utils.MyUtils;

import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.Indexing.UrlNotifications;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;

public class NotifyToDeleteUrlUtils {

	private static final String MY_APPLICATION = "My-Application";

	/**
	 * <pre>
	 * {
	 *  "urlNotificationMetadata": {
	 *   "url": "http://foo.com/path",
	 *  "latestUpdate": {
	 *    "url": "http://foo.com/path",
	 *    "type": "URL_UPDATED",
	 *     "notifyTime": "2023-11-28T12:49:16.868776828Z"
	 *   },
	 *   "latestRemove": {
	 *     "url": "http://foo.com/path",
	 *     "type": "URL_DELETED",
	 *     "notifyTime": "2023-11-28T13:50:29.662337626Z"
	 *     }
	 *   }
	 * }
	 * </pre>
	 */
	public static PublishUrlNotificationResponse requestToDeletePageUrl(String applicationName, String pageUrl)
			throws Exception {

		HttpRequestInitializer httpRequestInitializer //
				= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);

		//
		HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
		//

		Indexing indexing = new Indexing.Builder(httpTransport, //
				GsonFactory.getDefaultInstance(), //  
				httpRequestInitializer) //
				.setApplicationName(applicationName) //
				.build();

		UrlNotification content = new UrlNotification();
		content.setType("URL_DELETED");
		content.setUrl(pageUrl);
		//
		UrlNotifications.Publish publish = indexing.urlNotifications().publish(content);
		PublishUrlNotificationResponse respone = publish.execute();
		return respone;
	}

	public static PublishUrlNotificationResponse deletePageUrlIndexing(String pageUrl) throws Exception {
		return requestToDeletePageUrl(MY_APPLICATION, pageUrl);
	}

	public static void main(String[] args) throws Exception {
		// Test:
		PublishUrlNotificationResponse response = deletePageUrlIndexing(MyTestConstants.URL_TO_TEST);
		System.out.println("Delete Response: " + response.toPrettyString());
		UrlNotificationMetadata meta = response.getUrlNotificationMetadata();
		System.out.println("\nMetadata: " + meta.toPrettyString());
	}
}

Output:

{
  "urlNotificationMetadata": {
   "url": "http://foo.com/path",
  "latestUpdate": {
    "url": "http://foo.com/path",
    "type": "URL_UPDATED",
     "notifyTime": "2023-11-28T12:49:16.868776828Z"
   },
   "latestRemove": {
     "url": "http://foo.com/path",
     "type": "URL_DELETED",
     "notifyTime": "2023-11-28T13:50:29.662337626Z"
     }
   }
 }

6. Get information of a URL

The Google Indexing API allows you to see the last time you submitted a request to index or remove a specific URL. This information does not include the actual status of the URL (updated or removed).

If you want to know the real status of a URL, continue with the article below:
Get Started with the Google Search Console API

GetUrlNotificationMetaDataUtils.java

package org.o7planning.java_14285_google_apis;

import org.o7planning.googleapis.utils.MyUtils;

import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;

public class GetUrlNotificationMetaDataUtils {

	private static final String MY_APPLICATION = "My-Application";

	/**
	 * <pre>
	 * {
	 *  url: "http://foo.com/page",
	 *  latestUpdate: {
	 *    url: "http://foo.com/page",
	 *    type: "URL_UPDATED",
	 *    notifyTime: "2017-07-31T19:30:54.524457662Z"
	 *  },
	 *  latestRemove: {
	 *    url: "http://foo.com/page",
	 *    type: "URL_DELETED",
	 *    notifyTime: "2017-08-31T19:30:54.524457662Z"
	 *  }
	 * }
	 * </pre>
	 */
	public static UrlNotificationMetadata getUrlNotifications(String applicationName, String pageUrl) throws Exception {
		HttpRequestInitializer httpRequestInitializer //
				= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING); 
		 
		HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
		//

		Indexing indexing = new Indexing.Builder(httpTransport, //
				GsonFactory.getDefaultInstance(), //  
				httpRequestInitializer) //
				.setApplicationName(applicationName) //
				.build();

		Indexing.UrlNotifications.GetMetadata getMetadataRequest = indexing.urlNotifications().getMetadata();
		getMetadataRequest.setUrl(pageUrl);

		UrlNotificationMetadata metadata = getMetadataRequest.execute(); 
		return metadata;
	}

	public static UrlNotificationMetadata getUrlNotifications(String pageUrl) throws Exception {
		return getUrlNotifications(MY_APPLICATION, pageUrl);
	}

	public static void main(String[] args) throws Exception {
		// Test:
		UrlNotificationMetadata infos = getUrlNotifications(MyTestConstants.URL_TO_TEST);
		System.out.println("Return: " + infos.toPrettyString());
	}
}

Output:

{
  url: "http://foo.com/page",
  latestUpdate: {
    url: "http://foo.com/page",
    type: "URL_UPDATED",
    notifyTime: "2017-09-31T19:30:54.524457662Z"
  },
  latestRemove: {
    url: "http://foo.com/page",
    type: "URL_DELETED",
    notifyTime: "2017-08-31T19:30:54.524457662Z"
  }
 }

7. Request URL batch indexing

Instead of submitting index requests for individual URLs, you can combine up to 100 URLs to notify Google in one request.

BatchUpdateUrlUtils.java

package org.o7planning.java_14285_google_apis;

import java.io.IOException;

import org.o7planning.googleapis.utils.MyUtils;

import com.google.api.client.googleapis.batch.BatchRequest;
import com.google.api.client.googleapis.batch.json.JsonBatchCallback;
import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.http.HttpHeaders;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.indexing.v3.Indexing;
import com.google.api.services.indexing.v3.IndexingScopes;
import com.google.api.services.indexing.v3.model.PublishUrlNotificationResponse;
import com.google.api.services.indexing.v3.model.UrlNotification;

public class BatchUpdateUrlUtils {

	private static final String MY_APPLICATION = "My-Application";

	public static void executeBatchIndexing(String applicationName, String... pageUrls) throws Exception {
		HttpRequestInitializer httpRequestInitializer //
				= MyUtils.createHttpRequestInitializer(IndexingScopes.INDEXING);
		//
		HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();

		Indexing indexing = new Indexing.Builder(httpTransport, //
				GsonFactory.getDefaultInstance(), //
				httpRequestInitializer) //
				.setApplicationName(applicationName) //
				.build();
		//

		JsonBatchCallback<PublishUrlNotificationResponse> callback = new JsonBatchCallback<PublishUrlNotificationResponse>() {
			@Override
			public void onSuccess(PublishUrlNotificationResponse res, HttpHeaders responseHeaders) {
				try {
					System.out.println(res.toPrettyString());
				} catch (IOException e) {
					e.printStackTrace();
				}
			}

			@Override
			public void onFailure(GoogleJsonError e, HttpHeaders responseHeaders) throws IOException {
				System.out.println("Error Message: " + e.getMessage());
			}
		};

		BatchRequest batch = indexing.batch();

		for (String url : pageUrls) {
			UrlNotification unf = new UrlNotification();
			unf.setUrl(url);
			unf.setType("URL_UPDATED");
			indexing.urlNotifications().publish(unf).queue(batch, callback);
		}
		batch.execute();
	}

	public static void updatePageUrlIndexing(String... pageUrls) throws Exception {
		executeBatchIndexing(MY_APPLICATION, pageUrls);
	}

	public static void main(String[] args) throws Exception {
		// Test:
		updatePageUrlIndexing(MyTestConstants.URL_TO_TEST, MyTestConstants.URL2_TO_TEST);
	}
}

Output:

{
  "urlNotificationMetadata": {
    "latestRemove": {
      "notifyTime": "2023-12-01T14:37:38.721117189Z",
      "type": "URL_DELETED",
      "url": "http://foo.com/page"
    },
    "latestUpdate": {
      "notifyTime": "2023-12-02T12:43:43.074657919Z",
      "type": "URL_UPDATED",
      "url": "http://foo.com/page"
    },
    "url": "http://foo.com/page"
  }
}
{
  "urlNotificationMetadata": { 
    "latestUpdate": {
      "notifyTime": "2023-12-02T12:43:43.074745791Z",
      "type": "URL_UPDATED",
      "url": "http://foo.com/page2"
    },
    "url": "http://foo.com/page2"
  }
}

8. Error handling

An error may arise when you send a request to Google. For example, when you have exceeded the allowed quota, or submitted a request to index a URL of a website that is not yours, etc.

{
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "message": "Permission denied. Failed to verify the URL ownership.",
      "reason": "forbidden"
    }
  ],
  "message": "Permission denied. Failed to verify the URL ownership.",
  "status": "PERMISSION_DENIED"
}

In case of error, the Google Indexing API will throw an HttpResponseException.

HandleError.java

package org.o7planning.java_14285_google_apis;

import java.util.List;

import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonError.ErrorInfo;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.HttpResponseException;
import com.google.api.services.indexing.v3.model.UrlNotificationMetadata;

public class HandleError {

	public static void main(String[] args) throws Exception {
		// Test:
		try {
			UrlNotificationMetadata infos = GetUrlNotificationMetaDataUtils
					.getUrlNotifications("https://notmywebsite.com");
			System.out.println("Return: " + infos.toPrettyString());
		} catch (Exception e) {
			if (e instanceof HttpResponseException) {
				HttpResponseException ex = (HttpResponseException) e;
				int statusCode = ex.getStatusCode();
				String content = ex.getContent();
				System.err.println("statusCode: " + statusCode);
				System.err.println("content: " + content);
				if (e instanceof GoogleJsonResponseException) {
					GoogleJsonResponseException ex2 = (GoogleJsonResponseException) e;
					GoogleJsonError detailError = ex2.getDetails();
					//
					List<ErrorInfo> errorInfos = detailError.getErrors();
					for (ErrorInfo einfo : errorInfos) {
						System.out.println("Error: domain: " + einfo.getDomain());
						System.out.println("Error: message: " + einfo.getMessage());
						System.out.println("Error: reason: " + einfo.getReason());
					}
				}
			} else {
				// Other Error
				System.out.println("Error: " + e.getMessage());
				e.printStackTrace();
			}
		}
	}
}

Output:

statusCode: 403
content: {
  "code": 403,
  "errors": [
    {
      "domain": "global",
      "message": "Permission denied. Failed to verify the URL ownership.",
      "reason": "forbidden"
    }
  ],
  "message": "Permission denied. Failed to verify the URL ownership.",
  "status": "PERMISSION_DENIED"
}
Error: domain: global
Error: message: Permission denied. Failed to verify the URL ownership.
Error: reason: forbidden

Index pages with the Java Google Indexing API

1. Requirements

2. Library

3. HttpRequestInitializer

4. Notify to update a URL

5. Notify to remove a URL

6. Get information of a URL

7. Request URL batch indexing

8. Error handling

Google

Google

Newest Articles