When your data is bigger than RAM, the right move isn’t to read everything into a list and then compute — it’s to process it as a stream, computing as you read. The itertools module in the standard library was built for exactly that: its functions return lazy iterators that only produce an element when you iterate to it. This post uses the three most common ones — islice, chain and groupby — to read, merge and group large data.

The Problem

A log file of several gigabytes is a familiar situation: you only need the first 10 lines to check the format, or to count errors per day. The habitual approach is to read the whole file into a list before processing it — and that’s where the trouble starts.

# Cách tốn RAM: đọc hết file vào list
lines = open("access.log").readlines() # file 2 GB -> RAM 2 GB
first_10 = lines[:10]
print(first_10)

With a file that large, the system can kill your program before it ever reaches the main processing line. Similarly, merging several log files with the + operator creates a new list on every concatenation — slow, and twice the memory.

The Solution

The three functions below replace exactly those three habits — all of them return iterators, and no element is created until the loop reaches it.

To see the first 10 lines, use islice instead of readlines():

from itertools import islice
with open("access.log") as f:
first_10 = list(islice(f, 10)) # chỉ đọc 10 dòng đầu
print(first_10)

islice(f, 10) stops right after the 10th line — the rest of the file is never read. It can also slice a middle range with islice(f, 100, 110) to sample from the middle of a large file.

To merge several files into a single stream, use chain instead of list concatenation:

from contextlib import ExitStack
from itertools import chain
names = ["access-mon.log", "access-tue.log", "access-wed.log"]
with ExitStack() as stack:
files = [stack.enter_context(open(name)) for name in names]
errors_500 = sum(1 for line in chain(*files) if " 500 " in line)
print(errors_500)

chain joins multiple iterators into a single stream without creating an intermediate list — counting 500 errors across a whole week takes just one loop, and ExitStack closes all three files when you’re done.

To group logs by date, use groupby instead of a hand-rolled loop that accumulates into a dict:

from itertools import groupby
lines = [
"2026-11-16 ERROR db timeout",
"2026-11-16 ERROR db timeout",
"2026-11-15 INFO ok",
]
for date, group in groupby(lines, key=lambda line: line[:10]):
print(date, sum(1 for _ in group))

groupby only groups consecutive elements that share a key, so the data must be sorted by that key first — forgetting to sort is the most common mistake. If the logs aren’t in order yet, sort them first with sorted(lines, key=lambda line: line[:10]) and then group.

Apply It to Your Project

In a real project, start with wherever you’re doing “read everything, then compute”: log files, CSV exports, or query result lists. Replace readlines() with a loop directly over the file object, add islice when you only need part of it, use chain when merging multiple sources, and use groupby instead of a manual dict-accumulating loop when the data is already ordered. These three functions cover most daily logs and reports.

Related Posts

Frequently Asked Questions

How is itertools different from a list comprehension?

A list comprehension builds the entire list in RAM immediately; itertools returns an iterator that only computes an element when you iterate to it. For data that fits comfortably in RAM, a list comprehension is usually slightly faster; for large data, itertools is the safe choice.

Why does my groupby group wrong, with each key appearing multiple times?

Because groupby only groups consecutive elements that share a key. Sort by the key first with sorted(…, key=…) and then call groupby, and each key will appear only once.

Can islice be used with a plain list?

Yes, islice accepts any iterable, including lists. But for a list, lines[:10] is simpler; islice shines when the source can only be read once, like a file object or a generator.

When should I use itertools instead of pandas?

When the job is just reading, merging and grouping in a stream, and you don’t want to add a heavy dependency. pandas is stronger when you need filtering, joins and complex statistics on tabular data.

💬 Have you tried this in your own project? If you hit an error or know a shorter way, leave a comment below — I read them all and will add the best ideas to the post.

References