Java: How to determine the correct charset encoding of a stream

java

charset-detection

encoding

stream-processing

byNikita Barsukov·Feb 26, 2025

To determine the charset of a byte stream in Java, you can decode it using a CharsetDecoder and check whether any errors occur.

Here is a handy function, which takes a ByteBuffer and a list of charset names you want to try. It will loop through all provided charsets until it finds one which does not throw a CharacterCodingException during decoding. If none do, it returns "Undetermined".

import java.nio.charset.*;

public class CharsetDetector {
    public static String detectCharset(ByteBuffer data, String[] charsets) {
        // Let the Charset Hunger Games begin!
        for (String charset : charsets) {
            try {
                data.rewind(); // Back to the future!
                Charset.forName(charset).newDecoder()
                       .onMalformedInput(CodingErrorAction.REPORT)
                       .onUnmappableCharacter(CodingErrorAction.REPORT)
                       .decode(data);
                // And we have a winner!
                return charset;
            } catch (CharacterCodingException e) {
                // This one didn't survive, on to the next one!
            }
        }
        // They're all dead, Jim.
        return "Undetermined";
    }
}

Strategies to Detect Encoding

Detecting correct charset encoding can be challenging. A few strategies and tools are available that can facilitate this process.

Using Charset Detection Libraries

The CharsetDetector from the ICU4J library, which can detect over 200 charsets, or the Mozilla's juniversalchardet are perfect for this job. They provide a set of guessers for different charset families, letting you choose the one that works best based on their confidence level.

Leverage Metadata in XML/HTML Streams

XML and HTML files sometimes include their encoding in the metadata. Checking this before starting the decoding can save you time and computational resources.

Interact with Users

When automatic detection is uncertain, an alternate strategy could be to show the user some snippets of the decoded stream in different encodings and let them choose the one that seems right.

Be Ready to Handle Exceptions

Whenever you are working with streams and charsets, keep in mind that the UnsupportedCharsetException can occur. Always include logic in your code to catch these exceptions and handle them appropriately.

Dealing with Detection on Large Streams

For large streams, consider reading only a chunk of the stream initially for charset detection. This approach is faster and typically doesn't reduce accuracy.

Dialing in on the Right Decoding

While the initial decoding is often a case of trial and error, it's vital to have a strategic approach to improve efficiency and accuracy.

Charset Detection Based on Language Patterns

You can analyse character frequency and patterns in your stream if you know the language of the content. Certain characters and sequences are more common in some languages, which can provide hints for charset detection.

User Feedback

Having a sanity check with your user when automatic methods yield uncertain results contributes to accuracy. Presenting them with snippets of the file in various predicted charsets allows them to make an informed choice based on the content's semantics.