Lecture
In PHP, you can organize batch data processing in different ways, depending on what exactly you are working with: a database, files, or arrays.
In PHP, batch / chunk data processing is usually needed when there is a lot of data and you can't load everything into memory or process it in a single request.
Main methods:
let's take a closer look at these and other methods
When working with a database, we can load data in batches using LIMIT and OFFSET
$limit = 10; // Number of records per page
$offset = 0; // Offset (page number * $limit)
$sql = "SELECT * FROM users ORDER BY id LIMIT $limit OFFSET $offset";
$result = $pdo->query($sql)->fetchAll();
An alternative option is pagination without OFFSET
$last_id = 0;
$sql = "SELECT * FROM users WHERE id > $last_id ORDER BY id LIMIT $limit";
Simple
Suitable for small tables
OFFSET is slow on large tables
Issues if data changes while processing
Use only up to ~100k records
The idea: move along the primary key.
$limit = 100;
$lastId = 0;
do
{
$rows = $db
->query( "SELECT * FROM users WHERE id > $lastId ORDER BY id LIMIT $limit" )
->fetchAll();
foreach ($rows as $row) {
process($row);
$lastId = $row['id'];
}
} while (!empty($rows));
Very fast
Doesn't break on large volumes
Stable under concurrent changes
Requires a monotonic key (id, created_at)
The best option for a database
When working with large files, you can read them line by line instead of loading them into memory entirely.fgets() – reading line by line
$handle = fopen("large_file.txt", "r");
while (($line = fgets($handle)) !== false) {
echo $line;
}
fclose($handle);
SplFileObject – a more convenient way
$file = new SplFileObject("large_file.txt");
while (!$file->eof()) {
echo $file->fgets();
}
Use JsonMachine
Or streaming reads
foreach (JsonMachine\Items::fromFile('big.json') as $item) {
process($item);
}
If we have an array of data, we can split it into parts
$data = range(1, 100);
$chunks = array_chunk($data, 10);
foreach ($chunks as $chunk) {
print_r($chunk); // Will output arrays of 10 elements each
}
Generators let you return elements one at a time, without loading everything into memory at once.
The idea: process data record by record, but read it in batches.
function getData() {
for ($i = 1; $i <= 100; $i++) {
yield $i; // Returns one value at a time
}
}
foreach (getData() as $num) {
echo $num . "\n";
}
an example for fetching data from a database
function getUsers(PDO $db, int $chunk = 100): Generator {
$lastId = 0;
while (true) {
$stmt = $db->prepare(
"SELECT * FROM users WHERE id > :id ORDER BY id LIMIT :limit"
);
$stmt->bindValue(':id', $lastId, PDO::PARAM_INT);
$stmt->bindValue(':limit', $chunk, PDO::PARAM_INT);
$stmt->execute();
$rows = $stmt->fetchAll();
if (!$rows) {
break;
}
foreach ($rows as $row) {
$lastId = $row['id'];
yield $row;
}
}
}
Usage:
foreach (getUsers($db) as $user) {
process($user);
}
Minimal memory usage
Clean code
Generators (yield) in PHP are a powerful tool, but not a silver bullet. In the context of batch data processing / ETL / batch scripts, they have some quite real downsides.
1 You can't "rewind"
A generator is a one-shot stream.
$gen = getUsers(); foreach ($gen as $u) {} foreach ($gen as $u) {} // won't work
If you need to process the data again, the generator has to be created anew.
2 Hard to debug
you can't just var_dump($gen)
you can't view "all the data"
the call stack is harder to read
When errors occur inside a yield, debugging is more unpleasant than with arrays.
3 Errors "surface" late
$gen = getUsers(); // OK
// ...
foreach ($gen as $row) {
// ❌ the error happens here, not at creation
}
The exception occurs at the moment of iteration, not initialization — this sometimes breaks error-handling logic.
4 Not suitable if you need all the data at once
Generators are poorly suited for:
sorting
searching across the entire set
a repeated pass
totaling without intermediate state
$count = count($gen); // You need to accumulate the data — and the point of yield is lost.
5 Hard to combine with transactions
$db->beginTransaction();
foreach (getUsers($db) as $row) {
process($row); // a long process
}
$db->commit(); // a long-open transaction
Transactions can hang for minutes, which is bad for the database.
6 Overhead on every yield
every yield is a context switch
slower than a plain while + array if there isn't much data
For small volumes, generators are often slower, not faster.
7 Retries and rollbacks are harder
If:
the process crashed halfway through
the server restarted
the generator doesn't know where it was
You need to manually:
store the lastId
write checkpoints
log progress
8 Not all PHP developers know how to work with them
the code is harder to read
more maintenance errors
it's "magic" for juniors
In team development this is a real downside.
When generators are a bad choice
You need to reprocess the data
You need to sort / aggregate the entire set
There are transactions over large blocks
You need guarantees of recovery after a crash
The code is maintained by a large team
When generators are an excellent choice
Huge amounts of data
Minimal memory
Sequential processing
Read-only scenarios
ETL without strict rollbacks
yield is a memory optimization, not a processing architecture
It solves one problem excellently (memory),
but it doesn't solve:
reliability
retries
execution control
The idea: data → queue → workers
queue()->push(['user_id' => 123]);
Worker:
while ($job = queue()->pop()) { process($job); }
Scalable
Can be parallelized
Fault-tolerant
The best choice for heavy, long-running tasks
General downsides of queues (overall)
Regardless of the implementation:
Architectural complexity
It's no longer "a single PHP script", but:
a broker
producers
workers
monitoring
Harder to debug
The error could be:
in the queue
in the worker
in redelivery
Not an instant result
This is an asynchronous model — the user doesn't wait for execution
You need to monitor worker crashes
Without a supervisor / systemd everything breaks
The idea: process in pieces over time.
// take 100 unprocessed SELECT * FROM tasks WHERE processed = 0 LIMIT 100;
After processing:
UPDATE tasks SET processed = 1 WHERE id = ?;
Imports
Synchronizations
Regular jobs
php process.php --chunk=1000
You can fork:
pcntl_fork();
Used in high-load and ETL scenarios (E — Extract, T — Transform, L — Load)
| Scenario | Best option |
|---|---|
| Small volume | LIMIT + OFFSET |
| Large database | WHERE id > lastId |
| Low memory | Generators |
| Long-running tasks | Queues |
| Regular | CRON |
| Maximum speed | CLI + queue |
Comments