Category: Java

Exponential backoff in SQS

Amazon Web Services offers a service called Simple Queue Service (SQS) which makes it easy to decouple and scale your asynchronous system compoents. Messages in SQS will be retried on failure, and will retry once per message visibility timeout (default at 30 seconds). However, a common practice is to use exponential backoff instead of constant wait times for better flow control (see Error Retries and Exponential Backoff in AWS and the wikipedia page on Exponential backoff).

This post gives you a sample implementation on how to implement exponential backoff in your SQS consumer.

Posted Sun 17 June 2018 by Ivan Dyedov in Java (Java, AWS)

Comments

Interact with S3 Without Temp Files

There are lots of documentation and examples around uploading and downloading files to/from S3. A lot of times you don't actually want to keep around the files you upload or download, and want to delete them as soon as that process is done. You can easily do this using temp files like this:

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.GetObjectRequest;

import java.io.*;

class S3TempFileTest {

    private static final String S3_FILE_PREFIX = "s3test";
    private static final String S3_FILE_SUFFIX = ".tmp";
    private static final String S3_BUCKET_NAME = "bucket";
    private static final String S3_KEY_NAME = "key";

    private static final AmazonS3 AMAZON_S3 = new AmazonS3Client();

    public void testUploadWithTempFile() throws IOException {
        File tempFile = File.createTempFile(S3_FILE_PREFIX, S3_FILE_SUFFIX);
        // writeContent(tempFile)
        try {
            AMAZON_S3.putObject(S3_BUCKET_NAME, S3_KEY_NAME, tempFile);
        } finally {
            tempFile.delete();
        }
    }

    public void testDownloadWithTempFile() throws IOException {
        File tempFile = File.createTempFile(S3_FILE_PREFIX, S3_FILE_SUFFIX);
        try {
            AMAZON_S3.getObject(new GetObjectRequest(S3_BUCKET_NAME, S3_KEY_NAME), tempFile);
            // process(tempFile)
        } finally {
            tempFile.delete();
        }
    }
}

However, code like this is not easily testable as you're interacting with the file system, and is also not the most efficient if you're dealing with small files because writing to hard disk is slow (compared to memory IO).

Here is an example on how you could do it all in memory without using temp files. The example uses Jackson-serialized JSON content, but it applies to any kind of file.

Posted Fri 06 November 2015 by Ivan Dyedov in Java (Java, AWS)

Comments

Download Oracle Java JRE & JDK using a script

Oracle has recently disallowed direct downloads of java from their servers (without going through the browser and agreeing to their terms, which you can look at here: Oracle terms). So, if you try:

wget "http://download.oracle.com/otn-pub/java/jdk/7u4-b20/jdk-7u4-linux-x64.tar.gz"

you will receive a page with "In order to download products from Oracle Technology Network you must agree to the OTN license terms" error message.

This can be rather troublesome for setting up servers with automated scripts.

Luckily, it seems that a single cookie is all that is needed to bypass this (you still have to agree to the terms to install):

Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie

So, if you want to download jdk7u4 for 64-bit Linux (e.g., Ubuntu) using wget, you can use:

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u4-b20/jdk-7u4-linux-x64.tar.gz"

Posted Sat 05 May 2012 by Ivan Dyedov in Java (Java, Ubuntu, Linux)

Comments