How can I read a large text file line by line using Java?

java

streaming

lambda

performance

byNikita Barsukov·Oct 12, 2024

The most efficient approach to process a large text file line by line in Java involves the use of BufferedReader within a try-with-resources block to handle auto-closure:

try (BufferedReader reader = new BufferedReader(new FileReader("file.txt"))) {
    String line;
    while ((line = reader.readLine()) != null) {
        // Process it here, print line or, you know, make it dance 💃
    }
} catch (IOException ioEx) {
    ioEx.printStackTrace(); // Print or log your exceptions. Don't just swallow them 😵
}

The BufferedReader's readLine() is our hero here, dealing with memory management and identifying the EOF (end-of-file) with a null return. A precise and robust pattern for all your file-reading tasks.

Java 8 streams & processes

In Java 8 and beyond, Files.lines() introduces a different flow with streaming and lambda for processing large files:

try (Stream<String> lines = Files.lines(Paths.get("file.txt"))) {
    lines.forEach(System.out::println); 
    // Call me lazy, but isn't that lambda 🐏 notation cool?
} catch (IOException e) {
    e.printStackTrace(); // Houston, we have a problem 🚀!
}

This method uses internal iteration (Java's way of saying "Don't call us, we'll call you!"). It can also parallelize tasks. Just remember: streams being the responsible citizens they are, should be closed nicely, which try-with-resources handles for you.

Level up: tuning performance with buffers and encoding

The InputStreamReader, teamed up with a tuned buffer size and the right character encoding, makes an efficient team when dealing with different file types:

try (BufferedReader reader = new BufferedReader(new InputStreamReader(
         new FileInputStream("file.txt"), StandardCharsets.UTF_8), bufferSize)) {
    // Armored with a buffer, and a Reader that 'speaks' UTF-8, we are unstoppable! 🏰
}

Advanced buffer tuning matches your system capabilities and ensures your text doesn't get lost in translation due to misinterpreted encoding.

Resolving bumps on the road

Error handling requires robust tactics to intercept and manage possible issues:

try {
    // ...
} catch (IOException e) {
    // Log error and rethrow or notify
} finally {
    // Cleanup, if required. Destructor? Who said C++?
}

With try-with-resources, you won't need a finally block for resource closure. But, it can still be useful for other cleanup tasks.

Decoding your files

Different platforms have their own line ending styles: Windows uses \r\n, Unix \n. BufferedReader handles it all without a hiccup. But do pay attention to file encodings, like UTF-8 or ASCII. Explicitly defining InputStreamReader encoding prevents your text ending up as alien scribbles:

try (BufferedReader reader = new BufferedReader(new InputStreamReader(
         new FileInputStream("file.txt"), "UTF-8"))) {
    // File processing
    // Hoping not to see any 👾 alien symbols here
}

Considerations for the gurus

ASCII files are simpler, so optimizing for ASCII, if that's your jam, is 💸 money.
Use forEachOrdered() for sequential, in-order processing. No line jumping here, please! 🚧
Beware of Files.readAllLines(). It's not your friend for large files, thanks to its memory-gobbling ways.
Keep loops and stream operations light. Avoid turning them into performance hogs 🐖.