Java Awssdk S3 List objects
In this article, I will show you how to use Java S3Client to list objects on an S3 Bucket. Specifically we will focus on using the listObjectsV2 and listObjectsV2Paginator methods, which are new and better than using the listObject method.
In addition, you can also get basic information on each object you have listed, they include:
- File size
- The owner of the object
- Date of last modification.
- ...
First, we create the MyUtils class with a utility method to create the S3Client object, which is used in the examples of this article.
MyUtils.java
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
public class MyUtils {
public static S3Client createS3Client(Region region) {
AwsCredentials credentials = AwsBasicCredentials.create("accessKeyId", "secretAccessKey");
AwsCredentialsProvider credentialsProvider = StaticCredentialsProvider.create(credentials);
S3Client client = S3Client.builder() //
.credentialsProvider(credentialsProvider) //
.region(region) //
.build();
return client;
}
}
To create an S3Client object you need to create an AwsCredentialsProvider object, this object provides credentials that allow you to interact with AWS. See the article below to create an AwsCredentialsProvider suitable for your purposes.
1. Library
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/s3 -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>2.21.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk/s3-transfer-manager -->
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
<version>2.21.10</version>
</dependency>
<!-- https://mvnrepository.com/artifact/software.amazon.awssdk.crt/aws-crt -->
<dependency>
<groupId>software.amazon.awssdk.crt</groupId>
<artifactId>aws-crt</artifactId>
<version>0.28.0</version>
</dependency>
<!-- Log Library -->
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-api -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.9</version>
</dependency>
<!-- Log Library -->
<!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>2.0.9</version>
</dependency>
2. List objects with nextContinuationToken
By design, each user request sent to S3 Service only receives a maximum of 1000 objects. So if you want to get a list of all objects on a Bucket you have to send multiple requests. For each request you will receive a response consisting of a sublist of objects and a nextContinuationToken that creates a follow-up request for the next sublist. This seems a bit manual, see the full example:
ListObjectV2TokenExample.java
package org.o7planning.java_14211_awssdk_s3;
import java.util.List;
import org.o7planning.awssdks3.utils.MyUtils;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
public class ListObjectV2TokenExample {
private static Region myRegion = Region.EU_CENTRAL_1;
private static String myBucket = "test1.o7planning.org"; // Change to your bucket name
private static final int MAX_KEYS_PER_REQUEST = 10;
private static void listAllObjectsInBucket(S3Client s3Client, String bucketName) {
String nextContinuationToken = null;
int requestNo = 0;
do {
requestNo++;
ListObjectsV2Request.Builder requestBuilder = ListObjectsV2Request.builder() //
.bucket(bucketName) //
.maxKeys(MAX_KEYS_PER_REQUEST) // Optional (Default 1000).
.continuationToken(nextContinuationToken);
ListObjectsV2Response response = s3Client.listObjectsV2(requestBuilder.build());
List<S3Object> s3Objects = response.contents();
nextContinuationToken = response.nextContinuationToken();
System.out.println("Request " + requestNo + " returns " + s3Objects.size() + " keys");
System.out.println(" - nextContinuationToken: " + nextContinuationToken);
} while (nextContinuationToken != null);
}
public static void main(String[] args) {
S3Client s3Client = MyUtils.createS3Client(myRegion);
try {
listAllObjectsInBucket(s3Client, myBucket);
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
e.printStackTrace();
System.exit(1);
}
}
}
Output:
Request 1 returns 10 keys
- nextContinuationToken: 1qVzSfeqc5jfInTSOGJoqaWoKqXcwgEdI2u+SuejyNNhjqg38Wf8Xhz/edWke6K2N8r21lARSBC0d1FTHzhN88CBU34f/cG2RtN6iPW4Y218=
Request 2 returns 10 keys
- nextContinuationToken: 1uWdHKsz+nTpAXihAPDJJRvXc5lXavGBdd71iEvJVwzQLtcJxu9A/Gh4eWwzbW+/ZphmE6xnNDRSkJPbJpSR1LSWYxR6k2QBe
Request 3 returns 10 keys
- nextContinuationToken: 1s2ypTkPxF9QpwD5t6815Nj+SNJanjsmMBGmQVTvb0thQ4n7+LsmGBAy79tajYSI4YspaQGlbNnlwySwuWxE1ZFRgeWK13vVf
Request 4 returns 10 keys
- nextContinuationToken: 1lX92l0uLSnJs2hed1U35Rf25oorI8qrgyunlUqdDdjnDIzqC1L4EWJbLNrV1aE5FVdyey7CA/Q6O1H3YqQLLT+OPkHO6CRxAkswi7/a+opg=
Request 5 returns 10 keys
- nextContinuationToken: 11pD2mJjgIwgOe1vDSbyitV4soW789+XL90xIr/grScGh3GxE9iSD/UlzXakWgtA2+l67g1WscfcN1WHM6kHoU5LuMAAU42aIjCvr+pJHWcs=
Request 6 returns 10 keys
- nextContinuationToken: 1sqhUsdzV5B4yizBd2SEIdOpnY5NtX+78a5ZRemPMa8gtR6yu1JZhzsIrt0ARsdMY6HzGgoAM4tA=
Request 7 returns 5 keys
- nextContinuationToken: null
3. List objects with ListObjectsV2Iterable
Instead of using s3Client.listObjectsV2(request) and nextContinuationToken as in the above example, in this example we use the s3Client.listObjectsV2Paginator(request) method, which is much shorter and easier to understand.
ListObjectsV2Example1.java
package org.o7planning.java_14211_awssdk_s3;
import java.util.Iterator;
import java.util.List;
import org.o7planning.awssdks3.utils.MyUtils;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
import software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable;
public class ListObjectsV2Example1 {
private static Region myRegion = Region.EU_CENTRAL_1;
private static String myBucket = "test1.o7planning.org"; // Change to your bucket name
private static final int MAX_KEYS_PER_REQUEST = 10;
private static void listBucketObjects(S3Client s3Client, String bucketName) {
ListObjectsV2Request listV2Request = ListObjectsV2Request.builder() //
.bucket(bucketName) //
.maxKeys(MAX_KEYS_PER_REQUEST) //
.build();
ListObjectsV2Iterable listResponse = s3Client.listObjectsV2Paginator(listV2Request);
Iterator<ListObjectsV2Response> responseIterator = listResponse.iterator();
int requestNo = 0;
while (responseIterator.hasNext()) {
requestNo++;
ListObjectsV2Response response = responseIterator.next();
List<S3Object> s3Objects = response.contents();
System.out.println("Request " + requestNo + " returns " + s3Objects.size() + " keys");
}
}
public static void main(String[] args) {
S3Client s3Client = MyUtils.createS3Client(myRegion);
try {
listBucketObjects(s3Client, myBucket);
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
e.printStackTrace();
System.exit(1);
}
}
}
Output:
Request 1 returns 10 keys
Request 2 returns 10 keys
Request 3 returns 10 keys
Request 4 returns 10 keys
Request 5 returns 10 keys
Request 6 returns 10 keys
Request 7 returns 5 keys
4. List objects by prefix
In some cases you may only want to list objects with a specified prefix such as "static/images/".
ListObjectsV2Request listReq = ListObjectsV2Request.builder() //
.bucket(bucketName) //
.maxKeys(MAX_KEYS_PER_REQUEST) //
.prefix("static/images/") // Key Prefix
.build();
5. Get basic information about the object
Example: List objects in an S3 Bucket and print out the basic information of each object.
- File size
- The owner of the object
- Date of last modification.
- ...
ListObjectsV2Example2.java
package org.o7planning.java_14211_awssdk_s3;
import java.time.Instant;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import org.o7planning.awssdks3.utils.MyUtils;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
import software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable;
public class ListObjectsV2Example2 {
private static Region myRegion = Region.EU_CENTRAL_1;
private static String myBucket = "test1.o7planning.org"; // Change to your bucket name
private static final int MAX_KEYS_PER_REQUEST = 10;
private static void listBucketObjects(S3Client s3Client, String bucketName) {
ListObjectsV2Request listV2Request = ListObjectsV2Request.builder() //
.bucket(bucketName) //
.maxKeys(MAX_KEYS_PER_REQUEST) //
.build();
ListObjectsV2Iterable listResponse = s3Client.listObjectsV2Paginator(listV2Request);
Iterator<ListObjectsV2Response> responseIterator = listResponse.iterator();
while (responseIterator.hasNext()) {
ListObjectsV2Response response = responseIterator.next();
List<S3Object> s3Objects = response.contents();
//
for (S3Object s3Object : s3Objects) {
System.out.println("\n\nObject Key: " + s3Object.key());
System.out.println("Size: " + toKB(s3Object.size()) + " KB");
System.out.println("Owner: " + s3Object.owner());
//
Instant instant = s3Object.lastModified();
LocalDateTime lastModified = LocalDateTime.ofInstant(instant, ZoneOffset.UTC);
System.out.println("Last Modified: " + lastModified);
//
moreInfo(s3Client, bucketName, s3Object.key());
}
}
}
private static void moreInfo(S3Client s3Client, String bucketName, String objectKey) {
HeadObjectRequest headObjectRequest = HeadObjectRequest.builder() //
.bucket(bucketName) //
.key(objectKey) //
.build();
HeadObjectResponse headObjectResponse = s3Client.headObject(headObjectRequest);
//
System.out.println(" - contentType: " + headObjectResponse.contentType());
System.out.println(" - cacheControl: " + headObjectResponse.cacheControl());
System.out.println(" - websiteRedirectLocation: " + headObjectResponse.websiteRedirectLocation());
//
Map<String, String> metadata = headObjectResponse.metadata();
System.out.println(" User Defined Metadata: " + metadata);
for (String metaKey : metadata.keySet()) {
System.out.println(" Metadata " + metaKey + " : " + metadata.get(metaKey));
}
}
// Convert bytes to KB.
private static long toKB(long bytes) {
return bytes / 1024;
}
public static void main(String[] args) {
S3Client s3Client = MyUtils.createS3Client(myRegion);
try {
listBucketObjects(s3Client, myBucket);
} catch (S3Exception e) {
System.err.println(e.awsErrorDetails().errorMessage());
e.printStackTrace();
System.exit(1);
}
}
}
Output:
Object Key: icons/
Size: 0 KB
Owner: Owner(ID=d3c6d6ab076f6bfb589d38b23110f1890de07fbd613da51c075cc37800fd59d3)
Last Modified: 2023-11-09T17:04:35
- contentType: application/x-directory; charset=UTF-8
- cacheControl: null
- websiteRedirectLocation: null
User Defined Metadata: {}
Object Key: sample.png
Size: 2 KB
Owner: Owner(ID=d3c6d6ab076f6bfb589d38b23110f1890de07fbd613da51c075cc37800fd59d3)
Last Modified: 2023-11-09T18:06:27
- contentType: image/png
- cacheControl: null
- websiteRedirectLocation: null
User Defined Metadata: {my-meta=Some Value}
Metadata my-meta : Some Value
Object Key: static/
Size: 0 KB
Owner: Owner(ID=d3c6d6ab076f6bfb589d38b23110f1890de07fbd613da51c075cc37800fd59d3)
Last Modified: 2023-11-09T17:05:01
- contentType: application/x-directory; charset=UTF-8
- cacheControl: null
- websiteRedirectLocation: null
User Defined Metadata: {}
Object Key: static/icons/close.png
Size: 1 KB
Owner: Owner(ID=d3c6d6ab076f6bfb589d38b23110f1890de07fbd613da51c075cc37800fd59d3)
Last Modified: 2023-11-09T17:05:43
- contentType: image/png
- cacheControl: null
- websiteRedirectLocation: null
User Defined Metadata: {}
- Java Awssdk S3 Object Info
Amazon Web Services Tutorials
- Introduction to Amazon Web Services (AWS)
- Introduction to Amazon S3
- Introduction to Amazon Cloudfront and its architecture
- How to reduce Amazon Cloudfront Costs?
- Amazon CloudFront Invalidation
- Introduction to DigitalOcean Spaces
- Create DigitalOcean Spaces Bucket
- Introduction to Amazon ACM
- Java Awssdk S3 S3Client Upload object
- Create AWS accessKeyId/secretAccessKey
- Java Awssdk S3 List objects
- Host a static website on Amazon S3
- Java Awssdk CloudFront Invalidation
- DigitalOcean Spaces Create Access Key
- Java Awssdk Common Credentials Providers
- Java Awssdk ProfileCredentialsProvider
- Java Awssdk Creating and using EnvironmentVariableCredentialsProvider
- Java Awssdk Creating and using SystemPropertyCredentialsProvider
- Java Awssdk S3 Upload object with S3TransferManager
- Java Awssdk S3 S3TransferManager download object
- Java Manipulate DigitalOcean Spaces using S3TransferManager
- Java Create, list and delete S3 Bucket
- Aws Console create IAM User
- Create Amazon S3 Bucket
- Configure custom domain for Amazon S3 static website
- Create a CloudFront distribution for S3 Bucket
- Configure Amazon CloudFront Error Pages
- Amazon S3 Bucket policies
- Amazon AWS Policy Generator - policygen
- Migrate DNS service to Amazon Route 53
- Transfer domain registration to Amazon Route 53
- Request an SSL certificate from Amazon ACM
Show More