When the “Export CSV” button has to return hundreds of thousands of rows, the familiar approach — collect everything into an HttpResponse and only then send it — holds the entire file in RAM before the client receives the first byte. StreamingHttpResponse solves exactly this problem: it sends data in small chunks as soon as they are available, so the server only keeps a few thousand rows in memory at a time. This post uses the standard pattern from the Django docs: csv.writer writing into a “fake file” (the Echo class) combined with QuerySet.iterator().
The Problem
The familiar scenario: a report page has a button to export all orders to CSV. The usual code looks like this: create an HttpResponse, hand csv.writer the response, then loop over Order.objects.all() writing each row. It looks fine, but Django has to evaluate the whole queryset, write every row into the response, and only then start sending. With 200,000 orders, the worker’s RAM spikes for the entire request, the response is slow, and a small server can easily time out or run out of memory halfway through. The problem is not the query — it is that the entire result is held back before anything is sent.
import csvfrom django.http import HttpResponsedef export_orders_csv(request): response = HttpResponse(content_type="text/csv") response["Content-Disposition"] = 'attachment; filename="orders.csv"' writer = csv.writer(response) writer.writerow(["ID", "Order code", "Total"]) for order in Order.objects.all(): writer.writerow([order.id, order.code, order.total]) return response
The Solution
Swap HttpResponse for StreamingHttpResponse. This response takes an iterator and sends each piece as soon as the iterator yields it, instead of waiting for the full content. Because csv.writer needs an object with a write() method, we use the Echo class from the Django docs: write() just returns the exact string it receives. Combined with values_list() and iterator(chunk_size=2000), the database returns rows in batches of 2,000, and itertools.chain prepends the header row to the stream without collecting all the data.
import csvimport itertoolsfrom django.http import StreamingHttpResponseclass Echo: """Fake file class: csv.writer writes here and gets the string back for streaming.""" def write(self, value): return valuedef export_orders_csv(request): rows = itertools.chain( [["ID", "Order code", "Total"]], Order.objects.values_list("id", "code", "total").iterator(chunk_size=2000), ) writer = csv.writer(Echo()) return StreamingHttpResponse( (writer.writerow(row) for row in rows), content_type="text/csv", headers={"Content-Disposition": 'attachment; filename="orders.csv"'}, )
Three things make this work: first, iterator(chunk_size=2000) skips the queryset cache and fetches data in batches from the database. Second, Echo turns csv.writer into a “string generator” instead of writing to a real file. Third, the generator expression only builds each row when it is time to send it. Two practical notes: do not call len() or count the total rows before streaming — that would evaluate the whole queryset; and if you use GZipMiddleware, disable compression for this view, because the middleware buffers the entire response to compress it, wiping out the streaming benefit.
Apply It to Your Project
Review the report-export views in your project — customer lists, transaction history, activity logs — and swap HttpResponse for this pattern wherever the data can grow over time. Set chunk_size between 1,000 and 5,000 depending on how wide each row is, then compare the worker’s RAM before and after the change using real data. If the exported file is still small (a few thousand rows), keep plain HttpResponse for simplicity; streaming only really pays off when the row count is large or unbounded.
Related Posts
- Django: Process Millions of Records Without Blowing Up RAM — Use iterator() in 5 Minutes
- Flask: Stream Large Data with stream_with_context — Clients Receive It in Chunks, Server Stays Light on RAM in 5 Minutes
- Django: Back Up and Restore Data with dumpdata and loaddata in 5 Minutes
Frequently Asked Questions
How is StreamingHttpResponse different from HttpResponse?
HttpResponse collects all content and sends it in one go, while StreamingHttpResponse sends each chunk as soon as the iterator yields it. The client gets data sooner and the server’s RAM stays low, because only a few thousand rows are held at any moment.
Can I use this pattern to export Excel (.xlsx) files?
Not directly. The xlsx format needs random-access writes, so it cannot be streamed row by row like CSV. This pattern fits CSV or JSON Lines. If you really need an xlsx file, create a temporary file on disk and return it with FileResponse.
Why use iterator(chunk_size=…) instead of looping over the queryset directly?
Looping over the queryset directly evaluates the entire result and keeps it in the queryset’s cache — exactly what we are trying to avoid. iterator() skips that cache and fetches data in batches from the database.
💬 Have you tried this in your own project? If you hit an error or have a cleaner approach, leave a comment below — I read them all and will add the best ideas to the post.
References
- Django docs — StreamingHttpResponse: https://docs.djangoproject.com/en/6.1/ref/request-response/#streaminghttpresponse-objects
- Django docs — QuerySet.iterator(): https://docs.djangoproject.com/en/6.1/ref/models/querysets/#iterator
- Python docs — csv module: https://docs.python.org/3/library/csv.html
