When your data is bigger than RAM, the right move isn’t to read everything into a list and then compute — it’s to process it as a stream, computing as you read. The itertools module in the standard library was built for exactly that: its functions return lazy iterators that only produce an element when you iterate to it. This post uses the three most common ones — islice, chain and groupby — to read, merge and group large data.
The Problem
A log file of several gigabytes is a familiar situation: you only need the first 10 lines to check the format, or to count errors per day. The habitual approach is to read the whole file into a list before processing it — and that’s where the trouble starts.
# Cách tốn RAM: đọc hết file vào listlines = open("access.log").readlines() # file 2 GB -> RAM 2 GBfirst_10 = lines[:10]print(first_10)
With a file that large, the system can kill your program before it ever reaches the main processing line. Similarly, merging several log files with the + operator creates a new list on every concatenation — slow, and twice the memory.
The Solution
The three functions below replace exactly those three habits — all of them return iterators, and no element is created until the loop reaches it.
To see the first 10 lines, use islice instead of readlines():
from itertools import islicewith open("access.log") as f: first_10 = list(islice(f, 10)) # chỉ đọc 10 dòng đầuprint(first_10)
islice(f, 10) stops right after the 10th line — the rest of the file is never read. It can also slice a middle range with islice(f, 100, 110) to sample from the middle of a large file.
To merge several files into a single stream, use chain instead of list concatenation:
from contextlib import ExitStackfrom itertools import chainnames = ["access-mon.log", "access-tue.log", "access-wed.log"]with ExitStack() as stack: files = [stack.enter_context(open(name)) for name in names] errors_500 = sum(1 for line in chain(*files) if " 500 " in line)print(errors_500)
chain joins multiple iterators into a single stream without creating an intermediate list — counting 500 errors across a whole week takes just one loop, and ExitStack closes all three files when you’re done.
To group logs by date, use groupby instead of a hand-rolled loop that accumulates into a dict:
from itertools import groupbylines = [ "2026-11-16 ERROR db timeout", "2026-11-16 ERROR db timeout", "2026-11-15 INFO ok",]for date, group in groupby(lines, key=lambda line: line[:10]): print(date, sum(1 for _ in group))
groupby only groups consecutive elements that share a key, so the data must be sorted by that key first — forgetting to sort is the most common mistake. If the logs aren’t in order yet, sort them first with sorted(lines, key=lambda line: line[:10]) and then group.
Apply It to Your Project
In a real project, start with wherever you’re doing “read everything, then compute”: log files, CSV exports, or query result lists. Replace readlines() with a loop directly over the file object, add islice when you only need part of it, use chain when merging multiple sources, and use groupby instead of a manual dict-accumulating loop when the data is already ordered. These three functions cover most daily logs and reports.
Related Posts
- Python: Count and Group Data More Compactly with Counter and defaultdict in 5 Minutes
- Python: Replace os.path with pathlib — Read, Write, and Browse Files More Concisely in 5 Minutes
- Python: Write Context Managers with @contextmanager — Resources Always Closed, No Long Classes
Frequently Asked Questions
How is itertools different from a list comprehension?
A list comprehension builds the entire list in RAM immediately; itertools returns an iterator that only computes an element when you iterate to it. For data that fits comfortably in RAM, a list comprehension is usually slightly faster; for large data, itertools is the safe choice.
Why does my groupby group wrong, with each key appearing multiple times?
Because groupby only groups consecutive elements that share a key. Sort by the key first with sorted(…, key=…) and then call groupby, and each key will appear only once.
Can islice be used with a plain list?
Yes, islice accepts any iterable, including lists. But for a list, lines[:10] is simpler; islice shines when the source can only be read once, like a file object or a generator.
When should I use itertools instead of pandas?
When the job is just reading, merging and grouping in a stream, and you don’t want to add a heavy dependency. pandas is stronger when you need filtering, joins and complex statistics on tabular data.
💬 Have you tried this in your own project? If you hit an error or know a shorter way, leave a comment below — I read them all and will add the best ideas to the post.
References
- Python Documentation — itertools: https://docs.python.org/3/library/itertools.html
- Real Python — Python itertools By Example: https://realpython.com/python-itertools/
