it-swarm.cn

管道缓冲区有多大?

作为 中的注释,我对为何makefile中的“ | true”与“ || true” 用户 cjm 具有相同的效果感到困惑写道:

避免的另一个原因true是如果命令产生的输出足以填充管道缓冲区,它将阻塞等待true的读取。

我们是否有某种方法可以找出管道缓冲区的大小?

150
Kit Sunde

管道缓冲区的容量在不同系统之间有所不同(甚至在同一系统上可能有所不同)。我不确定是否有一种快速,简便和跨平台的方法来查找管道的容量。

例如,Mac OS X默认使用16384字节的容量,但是如果对管道进行大量写入,则可以切换到65336字节的容量,如果已经有太多的内核内存,则可以切换到单个系统页面的容量。由管道缓冲区使用(请参见 xnu/bsd/sys/pipe.hxnu/bsd/kern/sys_pipe.c ;由于它们来自FreeBSD,同样的行为也可能在那里发生)。

一个Linux pipe(7)手册页 说,自Linux 2.6.11起,管道容量为65536字节以及之前的单个系统页面(例如(32位)x86系统上的4096字节)。代码( include/linux/pipe_fs_i.hfs/pipe.c )似乎使用了16个系统页面(即64 KiB如果系统页面为4 KiB),但是可以通过管道上的fcntl来调整每个管道的缓冲区(最大为默认的最大容量)到1048576字节,但可以通过/proc/sys/fs/pipe-max-size)进行更改)。


这是一些bash/Perl组合,用于测试我系统上的管道容量:

#!/bin/bash
test $# -ge 1 || { echo "usage: $0 write-size [wait-time]"; exit 1; }
test $# -ge 2 || set -- "[email protected]" 1
bytes_written=$(
{
    exec 3>&1
    {
        Perl -e '
            $size = $ARGV[0];
            $block = q(a) x $size;
            $num_written = 0;
            sub report { print STDERR $num_written * $size, qq(\n); }
            report; while (defined syswrite STDOUT, $block) {
                $num_written++; report;
            }
        ' "$1" 2>&3
    } | (sleep "$2"; exec 0<&-);
} | tail -1
)
printf "write size: %10d; bytes successfully before error: %d\n" \
    "$1" "$bytes_written"

这是我在Mac OS X 10.6.7系统上以各种写入大小运行它的结果(请注意,写入大于16KiB的更改):

% /bin/bash -c 'for p in {0..18}; do /tmp/ts.sh $((2 ** $p)) 0.5; done'
write size:          1; bytes successfully before error: 16384
write size:          2; bytes successfully before error: 16384
write size:          4; bytes successfully before error: 16384
write size:          8; bytes successfully before error: 16384
write size:         16; bytes successfully before error: 16384
write size:         32; bytes successfully before error: 16384
write size:         64; bytes successfully before error: 16384
write size:        128; bytes successfully before error: 16384
write size:        256; bytes successfully before error: 16384
write size:        512; bytes successfully before error: 16384
write size:       1024; bytes successfully before error: 16384
write size:       2048; bytes successfully before error: 16384
write size:       4096; bytes successfully before error: 16384
write size:       8192; bytes successfully before error: 16384
write size:      16384; bytes successfully before error: 16384
write size:      32768; bytes successfully before error: 65536
write size:      65536; bytes successfully before error: 65536
write size:     131072; bytes successfully before error: 0
write size:     262144; bytes successfully before error: 0

Linux 3.19上的相同脚本:

/bin/bash -c 'for p in {0..18}; do /tmp/ts.sh $((2 ** $p)) 0.5; done'
write size:          1; bytes successfully before error: 65536
write size:          2; bytes successfully before error: 65536
write size:          4; bytes successfully before error: 65536
write size:          8; bytes successfully before error: 65536
write size:         16; bytes successfully before error: 65536
write size:         32; bytes successfully before error: 65536
write size:         64; bytes successfully before error: 65536
write size:        128; bytes successfully before error: 65536
write size:        256; bytes successfully before error: 65536
write size:        512; bytes successfully before error: 65536
write size:       1024; bytes successfully before error: 65536
write size:       2048; bytes successfully before error: 65536
write size:       4096; bytes successfully before error: 65536
write size:       8192; bytes successfully before error: 65536
write size:      16384; bytes successfully before error: 65536
write size:      32768; bytes successfully before error: 65536
write size:      65536; bytes successfully before error: 65536
write size:     131072; bytes successfully before error: 0
write size:     262144; bytes successfully before error: 0

注意:C头文件中定义的PIPE_BUF值(以及_PC_PIPE_BUFpathconf值)未指定管道的容量,但可以原子写入的最大字节数(请参阅 POSIXwrite(2) )。

引用来自 include/linux/pipe_fs_i.h

/* Differs from PIPE_BUF in that PIPE_SIZE is the length of the actual
   memory allocation, whereas PIPE_BUF makes atomicity guarantees.  */
147
Chris Johnsen

此Shell行也可以显示管道缓冲区的大小:

M=0; while true; do dd if=/dev/zero bs=1k count=1 2>/dev/null; \
       M=$(($M+1)); echo -en "\r$M KB" 1>&2; done | sleep 999

(将1k块发送到阻塞的管道,直到缓冲区已满)...一些测试输出:

64K (intel-debian), 32K (aix-ppc), 64K (jslinux bellard.org)      ...Ctrl+C.

使用printf的最短bash-one-liner:

M=0; while printf A; do >&2 printf "\r$((++M)) B"; done | sleep 999
36
Asain Kujovic

以下是一些仅使用Shell命令来探索实际管道缓冲区容量的其他选择:

# get pipe buffer size using Bash
yes produce_this_string_as_output | tee >(sleep 1) | wc -c

# portable version
( (sleep 1; exec yes produce_this_string_as_output) & echo $! ) | 
     (pid=$(head -1); sleep 2; kill "$pid"; wc -c </dev/stdin)

# get buffer size of named pipe
sh -c '
  rm -f fifo
  mkfifo fifo
  yes produce_this_string_as_output | tee fifo | wc -c &
  exec 3<&- 3<fifo
  sleep 1
  exec 3<&-
  rm -f fifo
'

# Mac OS X
#getconf PIPE_BUF /
#open -e /usr/include/limits.h /usr/include/sys/pipe.h
# PIPE_SIZE
# BIG_PIPE_SIZE
# SMALL_PIPE_SIZE
# PIPE_MINDIRECT
8
chan

这是Ubuntu 12.04,YMMV上的快速而肮脏的黑客

cat >pipesize.c

#include <unistd.h>
#include <errno.h>
#include </usr/include/linux/fcntl.h>
#include <stdio.h>

void main( int argc, char *argv[] ){
  int fd ;
  long pipesize ;

  if( argc>1 ){
  // if command line arg, associate a file descriptor with it
    fprintf( stderr, "sizing %s ... ", argv[1] );
    fd = open( argv[1], O_RDONLY|O_NONBLOCK );
  }else{
  // else use STDIN as the file descriptor
    fprintf( stderr, "sizing STDIN ... " );
    fd = 0 ;
  }

  fprintf( stderr, "%ld bytes\n", (long)fcntl( fd, F_GETPIPE_SZ ));
  if( errno )fprintf( stderr, "Uh oh, errno is %d\n", errno );
  if( fd )close( fd );
}

gcc -o pipesize pipesize.c

mkfifo /tmp/foo

./pipesize /tmp/foo

>sizing /tmp/foo ... 65536 bytes

date | ./pipesize

>sizing STDIN ... 65536 bytes
6
Jeff

如果您需要Python> = 3.3中的值,这是一个简单的方法(假设您可以调用dd):

from subprocess import Popen, PIPE, TimeoutExpired
p = Popen(["dd", "if=/dev/zero", "bs=1"], stdin=PIPE, stdout=PIPE)
try: 
    p.wait(timeout=1)
except TimeoutExpired: 
    p.kill()
    print(len(p.stdout.read()))
1
unhammer
$ ulimit -a | grep pipe
pipe size            (512 bytes, -p) 8

因此,在我的Linux机器上,默认情况下,我有8 * 512 = 4096字节的管道。

Solaris和许多其他系统具有类似的ulimit功能。

0
Sam Watkins