grep with continuous pipe does not work

advertisements

(maybe it is the "tcpflow" problem)

I write a script to monitoring http traffic, and I install tcpflow, then grep

it works (and you should make a http request, for example curl www.163.com)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: '

it outputs like this (continuously)

Host: config.getsync.com
Host: i.stack.imgur.com
Host: www.gravatar.com
Host: www.gravatar.com

but I can't continue to use pipe

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | cut -b 7-

does not work (nothing output)

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | grep '^Host: ' | grep H

When I replace sudo tcpflow with cat foo.txt, it works:

cat foo.txt | grep '^Host: ' | grep H

so what's wrong with pipe or grep or tcpflow ?


update:

This is my final script: https://github.com/zhengkai/config/blob/master/script/monitor_outgoing_http.sh


I think the problem is because of stdio buffering, you need to use GNU stdbuf before calling grep,

sudo tcpflow -p -c -i eth0 port 80 2>/dev/null | stdbuf -o0 grep '^Host: '

With the -o0, it basically means the output (stdout) stream from tcpflow will be unbuffered. The default behavior will be to automatically buffer up data into 40961 byte chunks before sending to next command in pipeline, which is what overriden using stdbuf


1. Refer this nice detail into the subject.