System call simulator: one file copy, every call

A program can't read a disk or open a file by itself; it asks the kernel with a system call. Copy a file three different ways and see every call it makes, as strace would print it, how the count changes with the buffer size, and what each call puts in the CPU's registers.

System calls
524,295
Copies via user space
2
Copy with
Common sizes
$ strace ./copy in.dat out.dat · click a line for its registers
+++ exited with 0 +++
$ strace -c
calls%syscall
262,14550.0read
262,14450.0write
20.0close
20.0openat
10.0exit_group
10.0newfstatat
524,295100total
At the syscall instruction: read
rax0 (read)
rdi3
rsipointer to the buffer
rdx4096
afterrax = 4096
Estimate from your numbers
524,295 calls × 100 ns
52.4 ms
Copying 2,147,483,648 bytes through user space
214.7 ms
Total, ignoring the disk
267.2 ms

How to use the simulator

Choose how the program copies in.dat to out.dat, the file size and, for read and write, the buffer size. The terminal shows the calls as strace prints them, with long runs of identical calls folded into one line and a count; the table beside it is what strace -c would summarise. Click a line to see the registers the program fills before the syscall instruction. The time estimate uses your own figures for the cost of one call and for memory copy speed, since both depend on the machine; it leaves out the disk.

What is a system call?

Programs run in user mode, where the CPU refuses instructions that touch hardware or other programs' memory. To open a file, read a socket or start a process, a program makes a system call: it puts a number identifying the call and up to six arguments in fixed registers and executes a special instruction that switches the CPU to kernel mode at an entry point the kernel chose. The kernel checks the arguments, does the work and returns to the program with a result.

On x86-64 Linux the number goes in rax, the arguments in rdi, rsi, rdx, r10, r8 and r9, and the instruction is syscall, which overwrites rcx and r11. The result returns in rax; a value between −4095 and −1 is an error number, which the C library turns into −1 and errno. Programs rarely do this directly: functions such as read() in the C library are thin wrappers around it.

Reading strace output

strace runs a program and prints each system call as it returns: the name, the arguments decoded as far as strace understands them (flags by name, structures by field, buffers as quoted strings), and after = the return value. openat(AT_FDCWD, "in.dat", O_RDONLY) = 3 opened a file relative to the current directory and got descriptor 3. A failure shows as = -1 ENOENT (No such file or directory). strace -c counts calls instead of listing them, and strace -f follows child processes. Current C libraries implement fstat() with the newer newfstatat call and an empty path, which is why it appears in the trace.

Buffer size and the number of calls

A read-and-write loop makes one read and one write per buffer-full, plus a final read that returns 0 at the end of the file. Copying 1 GiB:

BufferSystem calls
1 byte2,147,483,655
512 bytes4,194,311
4 KiB524,295
64 KiB32,775
128 KiB16,391
1024 KiB2,055

Every call pays a fixed price, for the switch into the kernel and back, the descriptor lookup and the bookkeeping, before it copies anything, so a program reading a byte at a time spends nearly all its time on overhead. This is why the C library's stdio functions such as getc read a block at a time into their own buffer, and why tools that move large amounts of data use buffers of tens or hundreds of kilobytes. Beyond that the count stops mattering: the bytes still have to be copied.

Fewer copies: mmap and sendfile

A read copies data from the kernel's page cache into the program's buffer, and the write copies it back into the kernel: two copies of every byte. mmap maps the source file into the program's address space, so a single write from the mapping copies each byte once; the pages are brought in by page faults as the kernel reads the mapping, which the system call trace does not show. That takes 9 calls for 1 GiB. sendfile asks the kernel to move the data between the two descriptors itself, with no copy through user space and 7 calls in all. Web servers use it to send static files to sockets, and copy_file_range does the same for file-to-file copies and can let the file system share or offload the copy.

Partial transfers and the 2 GiB limit

A read or write may move fewer bytes than asked: a read near the end of a file, from a pipe or socket with less data waiting, or a write interrupted by a signal. The return value says how many, and correct code loops until it is done. Linux also caps a single read, write or sendfile at 0x7ffff000 bytes, 2,147,479,552, even on 64-bit systems, so copying a 5 GB file with sendfile takes 3 calls; try a size above 2 GiB to see it.

About the model

The traces show the calls a small copy program makes itself; a real dynamically linked program also makes a few dozen calls at start-up to load the C library, which strace would show first. Pointers and addresses are illustrative. The number of system calls, the bytes moved and the call limit follow the Linux manual pages; the time estimate is only as good as the two figures you enter.

System call questions and answers

What is a system call?

A request from a program to the operating system kernel for something only the kernel may do, such as reading a file, sending on a socket or starting a process. On x86-64 Linux the program puts the call's number in rax and its arguments in rdi, rsi, rdx, r10, r8 and r9, executes the syscall instruction, and the CPU switches to kernel mode; the result comes back in rax.

How many system calls does copying a file take?

With read and write it is about two per buffer-full: copying 1 GiB takes 524,295 system calls with a 4 KiB buffer, 16,391 with 128 KiB and 2,147,483,655 with a 1-byte buffer. mmap with one write takes 9, and sendfile 7, counting opening, checking and closing the files.

How do I read strace output?

Each line is one system call: its name, its arguments as strace decodes them, and after = the return value. A failed call returns -1 with the error name and text, such as -1 ENOENT (No such file or directory). strace -c prints a table of how many times each call was made and how long they took instead.

What does sendfile do?

It copies data from one file descriptor to another inside the kernel, so the bytes never pass through the program's memory: no read into a buffer and write out again. Web servers use it to send static files to sockets. One call moves at most 2,147,479,552 bytes on Linux, so a program copies larger files in a loop.

Why is a bigger buffer faster?

Every system call has a fixed cost for switching into the kernel and back and for looking up the file, on top of the copying itself. A bigger buffer moves more bytes per call, so the fixed cost is paid fewer times. Past a few hundred kilobytes the gain is small, because the copying dominates.

What is the maximum size of a single read or write on Linux?

0x7ffff000 bytes, which is 2,147,479,552: read, write and sendfile transfer at most that much in one call, even on 64-bit systems, and return the number of bytes actually moved. Programs must loop until everything is transferred, which is also needed for partial reads and writes on pipes and sockets.

Further reading

Published Updated