List, submit and delete Sitemaps with Google Search Java API

Requirements
List Sitemaps
View and download a specific Sitemap
Add a Sitemap
Delete a Sitemap

1. Requirements

The requirements for working with Google Search Java API are mentioned in the article below. This article also covers different ways to create an HttpRequestInitializer object (With API Key, OAuth Client Id or Service Account).

Get Started with the Google Search Console API

A utility class to create an HttpRequestInitializer object, which provides credentials for requests each time they are sent to Google:

MyUtils.java

package org.o7planning.googleapis.utils;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;

import org.apache.commons.io.FileUtils;

import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.json.JsonObjectParser;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.client.util.ObjectParser;
import com.google.auth.http.HttpCredentialsAdapter;
import com.google.auth.oauth2.GoogleCredentials;
import com.google.auth.oauth2.ServiceAccountCredentials;

public class MyUtils {

	public static final String SERVICE_ACCOUNT_FILE_PATH = "/Volumes/New/_gsc/test-google-search-console-key.json";

	private static byte[] serviceAccountBytes;

	public static HttpRequestInitializer createHttpRequestInitializer(String... scopes) throws IOException {
		InputStream serviceAccountInputStream = getServiceAccountInputStream();

		GoogleCredentials credentials = ServiceAccountCredentials //
				.fromStream(serviceAccountInputStream) //
				.createScoped(scopes);

		HttpRequestInitializer requestInitializer = new HttpRequestInitializer() {

			@Override
			public void initialize(HttpRequest request) throws IOException {
				HttpCredentialsAdapter adapter = new HttpCredentialsAdapter(credentials);
				adapter.initialize(request);
				//
				if (request.getParser() == null) {
					ObjectParser parser = new JsonObjectParser(GsonFactory.getDefaultInstance());
					request.setParser(parser);
				}
				//
				request.setConnectTimeout(60000); // 1 minute connect timeout
				request.setReadTimeout(60000); // 1 minute read timeout
			}

		};
		return requestInitializer;
	}

	public static synchronized InputStream getServiceAccountInputStream() throws IOException {
		if (serviceAccountBytes == null) {
			serviceAccountBytes = FileUtils.readFileToByteArray(new File(SERVICE_ACCOUNT_FILE_PATH));
		}
		return new ByteArrayInputStream(serviceAccountBytes);
	}
}

A utility class for creating a SearchConsole.Sitemaps object, an important object that is related to all the examples in this article.

MySitemapsUtils.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.IOException;
import java.security.GeneralSecurityException;

import org.o7planning.googleapis.utils.MyUtils;

import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport;
import com.google.api.client.http.HttpRequestInitializer;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.json.gson.GsonFactory;
import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.SearchConsoleScopes;

public class MySitemapsUtils {

	private static final String APPLICATION_NAME = "My-Application";

	public static SearchConsole.Sitemaps getSitemapsClient(String applicationName) //
			throws IOException, GeneralSecurityException {
		//
		// @scopes: WEBMASTERS or WEBMASTERS_READONLY
		//
		HttpRequestInitializer httpRequestInitializer = MyUtils
				.createHttpRequestInitializer(SearchConsoleScopes.WEBMASTERS);

		//
		HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();

		//
		SearchConsole client = new SearchConsole.Builder( //
				httpTransport, GsonFactory.getDefaultInstance(), httpRequestInitializer) //
				.setApplicationName(applicationName) //
				.build();

		SearchConsole.Sitemaps sitemaps = client.sitemaps();
		return sitemaps;
	}

	public static SearchConsole.Sitemaps getSitemapsClient() //
			throws IOException, GeneralSecurityException {
		return getSitemapsClient(APPLICATION_NAME);
	}
}

2. List Sitemaps

With the Google Search API, you can list sitemap files that have been submitted to Google Search Console. By default they include the sitemap files you declared in the "robots.txt" file.

Take a look at Apple's "robots.txt" file:

https://www.apple.com/robots.txt

ListSitemapExample.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.List;

import org.o7planning.googleapis.MyTestConstants;

import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.model.SitemapsListResponse;
import com.google.api.services.searchconsole.v1.model.WmxSitemap;
import com.google.api.services.searchconsole.v1.model.WmxSitemapContent;

public class ListSitemapExample {

	//
	// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
	//
	private static void listSitemaps(String siteUrl) //
			throws IOException, GeneralSecurityException {
		SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
		SearchConsole.Sitemaps.List listRequest = sitemapClient.list(siteUrl);

		SitemapsListResponse listResponse = listRequest.execute();
		System.out.println("Sitemaps.List Response: " + listResponse.toPrettyString());

		List<WmxSitemap> sitemapList = listResponse.getSitemap();

		for (WmxSitemap sitemap : sitemapList) {
			System.out.println("\nSitemap: " + sitemap.getPath());
			System.out.println("- error: " + sitemap.getErrors());
			System.out.println("- warnings: " + sitemap.getWarnings());
			System.out.println("- isPending: " + sitemap.getIsPending());
			System.out.println("- isSitemapsIndex: " + sitemap.getIsPending());
			System.out.println("- lastDownloaded: " + sitemap.getLastDownloaded());
			System.out.println("- lastSubmitted: " + sitemap.getLastSubmitted());
			//
			List<WmxSitemapContent> contentList = sitemap.getContents();
			if (contentList != null) {
				for (WmxSitemapContent content : contentList) {
					System.out.println(" - indexed: " + content.getIndexed());
					System.out.println(" - submitted: " + content.getSubmitted());
					System.out.println(" - type: " + content.getType());
				}
			}
		}
	}

	public static void main(String[] args) throws IOException, GeneralSecurityException {
		listSitemaps(MyTestConstants.siteUrl);
	}
}

Output:

Sitemaps.List Response: {
  "sitemap": [
    {
      "contents": [
        {
          "indexed": "0",
          "submitted": "890",
          "type": "web"
        }
      ],
      "errors": "0",
      "isPending": false,
      "isSitemapsIndex": false,
      "lastDownloaded": "2023-12-03T06:47:32.933Z",
      "lastSubmitted": "2023-12-03T06:47:02.495Z",
      "path": "http://foo.com/sitemap.xml",
      "type": "sitemap",
      "warnings": "0"
    }
  ]
}

Sitemap: http://foo.com/sitemap.xml
- error: 0
- warnings: 0
- isPending: false
- isSitemapsIndex: false
- lastDownloaded: 2023-12-03T06:47:32.933Z
- lastSubmitted: 2023-12-03T06:47:02.495Z
 - indexed: 0
 - submitted: 890
 - type: web

3. View and download a specific Sitemap

In this example we will retrieve a sitemap information by its URL.

The "siteUrl" parameter can be:

sc-domain:yourdomain.com
http://yourdomain.com/
http://sub.yourdomain.com/
https://langlearning.net/
https://langlearning.net/app/en/

List, add and delete Sites with Google Search Java API

GetSitemapExample.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.IOException;
import java.security.GeneralSecurityException;
import java.util.List;

import org.o7planning.googleapis.MyTestConstants;

import com.google.api.services.searchconsole.v1.SearchConsole;
import com.google.api.services.searchconsole.v1.model.WmxSitemap;
import com.google.api.services.searchconsole.v1.model.WmxSitemapContent;

public class GetSitemapExample {
	//
	// @siteUrl: https://foo.com/ (Needs suffix "/") (for a URL-prefix property)
	// @siteUrl: sc-domain:foo.com (for a Domain property)
	// @feedpath: https://foo.com/sitemap.xml
	//
	private static void getOneSitemapInfo(String siteUrl, String feedpath)
			throws IOException, GeneralSecurityException {
		SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();

		SearchConsole.Sitemaps.Get getRequest = sitemapClient.get(siteUrl, feedpath);

		WmxSitemap sitemap = getRequest.execute();

		System.out.println("\nSitemap: " + sitemap.toPrettyString());

		System.out.println("- path: " + sitemap.getPath());
		System.out.println("- error: " + sitemap.getErrors());
		System.out.println("- warnings: " + sitemap.getWarnings());
		System.out.println("- isPending: " + sitemap.getIsPending());
		System.out.println("- isSitemapsIndex: " + sitemap.getIsPending());
		System.out.println("- lastDownloaded: " + sitemap.getLastDownloaded());
		System.out.println("- lastSubmitted: " + sitemap.getLastSubmitted());
		//
		List<WmxSitemapContent> contentList = sitemap.getContents();
		for (WmxSitemapContent content : contentList) {
			System.out.println(" - indexed: " + content.getIndexed());
			System.out.println(" - submitted: " + content.getSubmitted());
			System.out.println(" - type: " + content.getType());
		}
	}

	public static void main(String[] args) throws IOException, GeneralSecurityException { 
		getOneSitemapInfo(MyTestConstants.siteUrl, MyTestConstants.feedpath);
	}
}

Output:

Sitemap: {
  "contents": [
    {
      "indexed": "0",
      "submitted": "890",
      "type": "web"
    }
  ],
  "errors": "0",
  "isPending": false,
  "isSitemapsIndex": false,
  "lastDownloaded": "2023-12-03T06:47:32.933Z",
  "lastSubmitted": "2023-12-03T06:47:02.495Z",
  "path": "https://langlearning.net/sitemap.xml",
  "type": "sitemap",
  "warnings": "0"
}
- path: https://langlearning.net/sitemap.xml
- error: 0
- warnings: 0
- isPending: false
- isSitemapsIndex: false
- lastDownloaded: 2023-12-03T06:47:32.933Z
- lastSubmitted: 2023-12-03T06:47:02.495Z
 - indexed: 0
 - submitted: 890
 - type: web

Ví dụ download một sitemap:

DownloadSitemapExample.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.security.GeneralSecurityException;

import org.o7planning.googleapis.MyTestConstants;

import com.google.api.services.searchconsole.v1.SearchConsole;

public class DownloadSitemapExample {

	//
	// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
	// @feedpath: https://foo.com/sitemap.xml
	//
	private static void downloadExample(String siteUrl, String feedpath) //
			throws IOException, GeneralSecurityException { 
		SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();

		SearchConsole.Sitemaps.Get getRequest = sitemapClient.get(siteUrl, feedpath);

		System.out.println("Sitemaps.Get Request: " + getRequest.toString());

		String outputFilePath = "/Volumes/New/Test/output/sitemap.xml";
		FileOutputStream fileOutputStream = new FileOutputStream(new File(outputFilePath));
		getRequest.executeAndDownloadTo(fileOutputStream);
		fileOutputStream.close();
		System.out.println("Done!");
	}

	public static void main(String[] args) throws IOException, GeneralSecurityException {
		downloadExample(MyTestConstants.siteUrl, MyTestConstants.feedpath);
	}
}

4. Add a Sitemap

Submitting a new sitemap file to Google means you tell Google to track this file and crawl the URLs specified within it. After this action the sitemap file you just submitted will appear on the list you get through the API. If you submit a sitemap file that is already on the list, nothing will happen.

SubmitSitemapExample.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.IOException;
import java.security.GeneralSecurityException;

import org.o7planning.googleapis.MyTestConstants;

import com.google.api.services.searchconsole.v1.SearchConsole;

public class SubmitSitemapExample {

	//
	// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
	// @feedpathToSubmit: https://foo.com/sitemap.xml
	//
	private static void submitSitemap(String siteUrl, String feedpathToSubmit) //
			throws IOException, GeneralSecurityException {
		SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();
		SearchConsole.Sitemaps.Submit submitRequest //
				= sitemapClient.submit(siteUrl, feedpathToSubmit);

		submitRequest.execute();
		System.out.println("Done!");
	}

	public static void main(String[] args) throws IOException, GeneralSecurityException {
		submitSitemap(MyTestConstants.siteUrl, MyTestConstants.feedpathToSubmit);
	}
}

5. Delete a Sitemap

Using the Google Console API you can delete a sitemap file, which will be removed from the list of sitemaps returned from the Google Search API.

Over time, Google's system will stop trying to use the sitemap file you just deleted. Note - deleting the sitemap file does not prevent the URLs specified in the sitemap from being crawled. Because Google can detect links on your website when parsing the content of each page. However, sitemaps are necessary to help Google crawl more effectively, which is often why Google recommends using this file.

In addition, an important thing for Google to stop tracking the sitemap file you just deleted is that this sitemap must return error code 404 or 410.

Example, deleting a sitemap with the Google Search Java API:

DeleteSitemapExample.java

package org.o7planning.googleapis.a14297.sitemap;

import java.io.IOException;
import java.security.GeneralSecurityException;

import org.o7planning.googleapis.MyTestConstants;

import com.google.api.services.searchconsole.v1.SearchConsole;

public class DeleteSitemapExample {

	//
	// @siteUrl: https://foo.com/ (IMPORTANT: needs suffix "/")
	// @feedpathToDelete: https://foo.com/sitemap2.xml
	//
	private static void deleteSitemap(String siteUrl, String feedpathToDelete) //
			throws IOException, GeneralSecurityException {
		SearchConsole.Sitemaps sitemapClient = MySitemapsUtils.getSitemapsClient();

		SearchConsole.Sitemaps.Delete deleteRequest //
				= sitemapClient.delete(siteUrl, feedpathToDelete);

		deleteRequest.execute();
		System.out.println("Done!");
	}

	public static void main(String[] args) throws IOException, GeneralSecurityException {
		deleteSitemap(MyTestConstants.siteUrl, MyTestConstants.feedpathToDelete);
	}
}