os_cp:redirects_pipes
Differences
This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
os_cp:redirects_pipes [2023/03/22 01:27] jkonczak utworzono |
os_cp:redirects_pipes [2025/04/07 21:07] (current) jkonczak added examples |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ===== Standard streams ===== | ||
| - | |||
| - | | K&R C[[https://archive.org/details/TheCProgrammingLanguageFirstEdition|[1]]] [[https://en.wikipedia.org/wiki/C_(programming_language)#K&R_C|[2]]]<code c>printf("Please type your name:\n") | ||
| - | scanf("%s", name);</code> | Python<code python>print("Please type your name:") | ||
| - | name = input()</code> | | ||
| - | |||
| - | Did you ever wonder how does a program know where to read input from and where | ||
| - | the print-like functions should output? | ||
| - | |||
| - | **In UNIX world a program expects to have three files already open upon | ||
| - | start – standard input, standard output and standard error. | ||
| - | These are called the [[https://en.wikipedia.org/wiki/Standard_streams|standard streams]].** | ||
| - | \\ | ||
| - | <small> | ||
| - | The standard input/output library of the C programming language – ''stdio.h'' – | ||
| - | bases on this concept. C was created by one of the authors of UNIX. | ||
| - | </small> | ||
| - | |||
| - | Basic I/O functions in most programming languages by default read from the | ||
| - | standard input, and output data to standard output. \\ | ||
| - | The rationale of standard error stream is to convey information on what went | ||
| - | wrong while executing a program. Programming languages usually offer dedicated | ||
| - | functions to output data to standard error. | ||
| - | |||
| - | In Unix-like, as well as POSIX-compatible systems, the operating system is | ||
| - | responsible for abstracting files away – the user should not worry about | ||
| - | details of accessing a file. | ||
| - | \\ | ||
| - | When the user wants to open a file, the user provides the file name and gets an | ||
| - | identifier – a **file descriptor** in return. \\ | ||
| - | (A file descriptor is in fact an index in an array of files maintained for the | ||
| - | process by the OS.) \\ | ||
| - | To do standard operations such as reading or writing data, the user just tells | ||
| - | which operation shall be executed, on which file descriptor, and the user shall | ||
| - | provide the details of the operations (such as where to put data read from | ||
| - | the file and how many bytes shall be read). | ||
| - | |||
| - | A child process inherits all file descriptors from its parent. | ||
| - | |||
| - | **The three standard streams are the files represented by first three file | ||
| - | descriptors – 0 is always used for standard input, 1 for stand output and 2 for | ||
| - | standard error.** | ||
| - | |||
| - | The files do not need to be ordinary files – Unix-like systems abstract almost | ||
| - | everything with a [[https://en.wikipedia.org/wiki/Everything_is_a_file|file]]. | ||
| - | \\ | ||
| - | For instance, a terminal device is a file (even if it were a real teletype). | ||
| - | |||
| - | By default, a shell opens the terminal as file 0, 1 and 2. | ||
| - | |||
| - | |||
| ===== Outputting file / text / sequences ===== | ===== Outputting file / text / sequences ===== | ||
| Line 58: | Line 7: | ||
| Name comes from //con__cat__enate//. \\ | Name comes from //con__cat__enate//. \\ | ||
| ''cat'' numbers lines with the ''-n'' switch and outputs non-printable characters | ''cat'' numbers lines with the ''-n'' switch and outputs non-printable characters | ||
| - | as ''^//x//'', ''M-//x//'', … with the ''-v'' switch. | + | as ''^//x//'', ''M-//x//'', … with the ''-v'' switch((This notation corresponds to keys one would have to press to input the byte. To see all bytes (''\t'' and ''\n'' is replaced by ''X''), try: \\ ''perl -e'for(0..15){printf"\t%x_",$_};print"\n";for$l(0..15){printf"_%x",$l;for$h(0..15){$c=$h<<4|$l;$c=88 if $c==9||$c==10;printf("\t%c",$c)}print"\n"}'|cat -v'')). |
| ''**paste** //file_1// [//file2//]...'' reads round-robin one line from each input | ''**paste** //file_1// [//file2//]...'' reads round-robin one line from each input | ||
| Line 65: | Line 14: | ||
| <small> | <small> | ||
| + | ''**join** [-t //separator//] [-1 //fieldNumberInFile1//] [-2 //fieldNumberInFile2//] //file1// //file2//'' | ||
| + | parses two sorted files and prints their lines joined on specified fields. | ||
| + | |||
| ''**fold** [-w //width//] [//file//]...'' outputs input files (or standard input) | ''**fold** [-w //width//] [//file//]...'' outputs input files (or standard input) | ||
| forcing a line break whenever a line would exceed //width// (that defaults to 80). | forcing a line break whenever a line would exceed //width// (that defaults to 80). | ||
| \\ | \\ | ||
| With the ''-s'' switch ''fold'' breaks lines on spaces (or at //width// if there are no spaces). | With the ''-s'' switch ''fold'' breaks lines on spaces (or at //width// if there are no spaces). | ||
| + | |||
| + | ''**column** [-x] [//file//]...'' works just like ''cat'' if the longest line in the //file//s (or standard input) would not fit twice within the terminal width. \\ Else, it prints the input in as many columns as fit the terminal, filling column first (or, with ''-x'', rows first). | ||
| + | \\ | ||
| + | ''**column -t** [//file//]...'' does a completely different thing: it detects columns in input (by a separator that defaults to whitespace) and outputs the input as a table. | ||
| ''**od** [-t x1]'', ''**hexdump** [-C]'', and ''**xxd**'' show binary files. | ''**od** [-t x1]'', ''**hexdump** [-C]'', and ''**xxd**'' show binary files. | ||
| </small> | </small> | ||
| + | |||
| + | ++++ Examples | {{page>so:redirects:cat&inline}} ++++ | ||
| ~~Exercise.#~~ Print any file with ''cat''. Print two files at once with ''cat''. \\ | ~~Exercise.#~~ Print any file with ''cat''. Print two files at once with ''cat''. \\ | ||
| Line 100: | Line 58: | ||
| </small> | </small> | ||
| - | ~~Exercise.#~~ Try ''echo -e 'foo\n\nbaz' '' \\ and ''echo -e '\n\n one \033[A \033[A two \033[B \033[B \n \033[1;31m red \033[0m' '' | + | ++++ Examples | {{page>so:redirects:echo&inline}} ++++ |
| + | |||
| + | ~~Exercise.#~~ Try ''echo -e 'foo\n\nbaz' '' \\ and ''echo -e '\n\n one \033[A \033[A two \033[B \033[B \n \033[1;31m red \033[0m' '' \\ <small>[[https://en.wikipedia.org/wiki/ANSI_escape_code|ANSI escape codes]] are well summarized [[https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797|here]]</small> | ||
| ~~Exercise.#~~ Try ''printf "|%4.2f|%3s|%-20s|\n|%4.2f|%3s|%-20s|\n" 3.1428 pi circumference/radius 9.8 g gravity'' | ~~Exercise.#~~ Try ''printf "|%4.2f|%3s|%-20s|\n|%4.2f|%3s|%-20s|\n" 3.1428 pi circumference/radius 9.8 g gravity'' | ||
| Line 116: | Line 76: | ||
| \\ | \\ | ||
| With the ''-w'' switch, ''seq'' makes all number of equal width (e.g., ''seq -w 8 11'' outputs 08, 09, 10 and 11). | With the ''-w'' switch, ''seq'' makes all number of equal width (e.g., ''seq -w 8 11'' outputs 08, 09, 10 and 11). | ||
| + | |||
| + | ++++ Examples | {{page>so:redirects:seq&inline}} ++++ | ||
| ~~Exercise.#~~ Generate a sequence of numbers from 1 to 15. | ~~Exercise.#~~ Generate a sequence of numbers from 1 to 15. | ||
| ~~Exercise.#~~ Generate a sequence of numbers from 64 to 1024 with step of 64. | ~~Exercise.#~~ Generate a sequence of numbers from 64 to 1024 with step of 64. | ||
| + | |||
| + | ===== Standard streams ===== | ||
| + | |||
| + | | K&R C[[https://archive.org/details/TheCProgrammingLanguageFirstEdition|[1]]] [[https://en.wikipedia.org/wiki/C_(programming_language)#K&R_C|[2]]]<code c>printf("Please type your name:\n") | ||
| + | scanf("%s", name);</code> | Python<code python>print("Please type your name:") | ||
| + | name = input()</code> | | ||
| + | |||
| + | Did you ever wonder how does a program know where to read input from and where | ||
| + | the print-like functions should output? | ||
| + | |||
| + | **In UNIX world a program expects to have three files already open upon | ||
| + | start – standard input, standard output and standard error. | ||
| + | These are called the [[https://en.wikipedia.org/wiki/Standard_streams|standard streams]].** | ||
| + | \\ | ||
| + | <small> | ||
| + | The standard input/output library of the C programming language – ''stdio.h'' – | ||
| + | bases on this concept. C was created by one of the authors of UNIX. | ||
| + | </small> | ||
| + | |||
| + | Basic I/O functions in most programming languages by default read from the | ||
| + | standard input, and output data to standard output. \\ | ||
| + | The rationale of standard error stream is to convey information on what went | ||
| + | wrong while executing a program. Programming languages usually offer dedicated | ||
| + | functions to output data to standard error. | ||
| + | |||
| + | In Unix-like, as well as POSIX-compatible systems, the operating system is | ||
| + | responsible for abstracting files away – the user should not worry about | ||
| + | details of accessing a file. | ||
| + | \\ | ||
| + | When the user wants to open a file, the user provides the file name and gets an | ||
| + | identifier – a **file descriptor** in return. \\ | ||
| + | (A file descriptor is in fact an index in an array of files maintained for the | ||
| + | process by the OS.) \\ | ||
| + | To do standard operations such as reading or writing data, the user just tells | ||
| + | which operation shall be executed, on which file descriptor, and the user shall | ||
| + | provide the details of the operations (such as where to put data read from | ||
| + | the file and how many bytes shall be read). | ||
| + | |||
| + | A child process inherits all file descriptors from its parent. | ||
| + | |||
| + | **The three standard streams are the files represented by first three file | ||
| + | descriptors – 0 is always used for standard input, 1 for stand output and 2 for | ||
| + | standard error.** | ||
| + | |||
| + | The files do not need to be ordinary files – Unix-like systems abstract almost | ||
| + | everything with a [[https://en.wikipedia.org/wiki/Everything_is_a_file|file]]. | ||
| + | \\ | ||
| + | For instance, a terminal device is a file (even if it were a real teletype). | ||
| + | |||
| + | By default, a shell opens the terminal as file 0, 1 and 2. | ||
| ===== Redirections ===== | ===== Redirections ===== | ||
| Line 161: | Line 173: | ||
| ~~Exercise.#~~ Redirect standard output and the standard error of the ''find /var/spool/'' command to the same file. | ~~Exercise.#~~ Redirect standard output and the standard error of the ''find /var/spool/'' command to the same file. | ||
| + | |||
| + | ++++ Examples | {{page>so:redirects:out-en&inline}} ++++ | ||
| === Input redirections === | === Input redirections === | ||
| Line 189: | Line 203: | ||
| ~~Exercise.#~~ Use ''bc'' to calculate ''sqrt(2.0000)'' in non-interactive mode and redirect its output to a file. | ~~Exercise.#~~ Use ''bc'' to calculate ''sqrt(2.0000)'' in non-interactive mode and redirect its output to a file. | ||
| + | |||
| + | ++++ Examples | {{page>so:redirects:in&inline}} ++++ | ||
| <small> | <small> | ||
| Line 211: | Line 227: | ||
| | ''>>'' | opens //word// for appending and replaces //file_number// with the file | | | ''>>'' | opens //word// for appending and replaces //file_number// with the file | | ||
| | ''<>'' | opens //word// for reading and writing and replaces //file_number// with the file | | | ''<>'' | opens //word// for reading and writing and replaces //file_number// with the file | | ||
| - | | ''<<'' | 1) creates a temporary file \\ 2) reads an input line \\ 3) if the line is //word//, go to step 6 \\ 4) if there are no quotes (a pair of ''"'' or <html><code>'</code></html>) in //word//, performs the expansion((E.g., ''$VAR'' is substituted with its value, ''`date`'' is replaced by output of the date command etc.)) on the line \\ 5) writes the line to the temporary file \\ 6) opens the temporary file for reading \\ 7) replace //file_number// with the file \\ The command is run once this is done | | + | | ''<<'' | 1) creates a temporary file \\ 2) reads an input line \\ 3) if the line is //word//, goes to step 7 \\ 4) if there are no quotes (a pair of ''"'' or <html><code>'</code></html>) enclosing the //word//, performs the expansion((E.g., ''$VAR'' is substituted with its value, ''`date`'' is replaced by output of the date command etc.)) on the line \\ 5) appends the line to the temporary file \\ 6) goes to step 2 \\ 7) opens the temporary file for reading \\ 8) replace //file_number// with the file \\ The command is run once this is done | |
| | ''<<-''| same as ''<<'', but after step 2 adds a step: \\ 2a) erase all leading tab characters (''\t'') \\ warning: spaces are not erased | | | ''<<-''| same as ''<<'', but after step 2 adds a step: \\ 2a) erase all leading tab characters (''\t'') \\ warning: spaces are not erased | | ||
| | ''<<<''| warning: this is a Bash extension \\ 1) creates a temporary file \\ 2) writes //word// to it \\ 3) writes a newline to it \\ 4) opens the temporary file for reading \\ 5) replace //file_number// with the file \\ The command is run once this is done | | | ''<<<''| warning: this is a Bash extension \\ 1) creates a temporary file \\ 2) writes //word// to it \\ 3) writes a newline to it \\ 4) opens the temporary file for reading \\ 5) replace //file_number// with the file \\ The command is run once this is done | | ||
| - | | ''<&'' | if //word// is a number: duplicates a readable descriptor number //word// to the number //file_number// \\ if //word/// is ''-'': closes a descriptor number //file_number// | | + | | ''<&'' | if //word// is a number: duplicates a readable descriptor number //word// to the number //file_number// \\ if //word// is ''-'': closes a descriptor number //file_number// | |
| - | | ''>&'' | if //word// is a number: duplicates a writeable descriptor number //word// to the number //file_number// \\ if //word/// is ''-'': closes a descriptor number //file_number// | | + | | ''>&'' | if //word// is a number: duplicates a writeable descriptor number //word// to the number //file_number// \\ if //word// is ''-'': closes a descriptor number //file_number// | |
| | ''&>'' | warning: this is a Bash extension \\ warning: this does not allow providing //file_number// (in fact, ''&'' is the file number) \\ opens //word// for writing, truncates it and replaces streams 1 and 2 with the file | | | ''&>'' | warning: this is a Bash extension \\ warning: this does not allow providing //file_number// (in fact, ''&'' is the file number) \\ opens //word// for writing, truncates it and replaces streams 1 and 2 with the file | | ||
| | ''&>>''| warning: this is a Bash extension \\ warning: this does not allow providing //file_number// (in fact, ''&'' is the file number) \\ opens //word// for appending and replaces streams 1 and 2 with the file | | | ''&>>''| warning: this is a Bash extension \\ warning: this does not allow providing //file_number// (in fact, ''&'' is the file number) \\ opens //word// for appending and replaces streams 1 and 2 with the file | | ||
| Line 231: | Line 247: | ||
| standard output (unless you dare to face the consequences). | standard output (unless you dare to face the consequences). | ||
| - | ~~Exercise.#~~ Swap standard input with standard output of the | + | ~~Exercise.#~~ Swap standard error with standard output of the |
| ''cat /etc/motd /etc/shadow'' command. Test if you did this correctly by adding | ''cat /etc/motd /etc/shadow'' command. Test if you did this correctly by adding | ||
| ''|rev'' at the end (that will put standard output backwards). | ''|rev'' at the end (that will put standard output backwards). | ||
| Line 305: | Line 321: | ||
| All that is written to a pipe is stored in the main memory (so it never occupies | All that is written to a pipe is stored in the main memory (so it never occupies | ||
| disk space, regardless if it is a named pipe) until some program reads it. | disk space, regardless if it is a named pipe) until some program reads it. | ||
| + | |||
| + | ~~Exercise.#~~ Run ''echo '2+2*2' ''. Then, pipe it through ''bc''. | ||
| ~~Exercise.#~~ ''echo'' some text. Then, echo the text and pipe it through ''xxd''. | ~~Exercise.#~~ ''echo'' some text. Then, echo the text and pipe it through ''xxd''. | ||
| Line 319: | Line 337: | ||
| ~~Exercise.#~~ Create a named pipe //p//. Redirect input of ''fold'' from //p// | ~~Exercise.#~~ Create a named pipe //p//. Redirect input of ''fold'' from //p// | ||
| in one terminal, and redirect output of ''ps -eF'' to //p// in another terminal. | in one terminal, and redirect output of ''ps -eF'' to //p// in another terminal. | ||
| - | \\ Then repeat the commands, running the''ps'' before the ''fold''. | + | \\ Then repeat the commands, running the ''ps'' before the ''fold''. |
| </small> | </small> | ||
| + | |||
| + | ++++ Examples | {{page>so:redirects:pipes1-en&inline}} ++++ | ||
| ===== Filters ===== | ===== Filters ===== | ||
| Line 349: | Line 369: | ||
| ~~Exercise.#~~ Run ''seq 25 > //file//''. Then run ''tail -f //file//'' in one | ~~Exercise.#~~ Run ''seq 25 > //file//''. Then run ''tail -f //file//'' in one | ||
| terminal and append (with output redirection) some data to //file//. | terminal and append (with output redirection) some data to //file//. | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:head_tail&inline}} ++++ | ||
| ==== grep ==== | ==== grep ==== | ||
| Line 395: | Line 417: | ||
| \\ | \\ | ||
| List all files containing ''ecdsa'' in ''~/.ssh''. | List all files containing ''ecdsa'' in ''~/.ssh''. | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:grep&inline}} ++++ | ||
| ==== cut ==== | ==== cut ==== | ||
| Line 401: | Line 425: | ||
| bytes (''-b //spec//'') / fields (''-f //spec//'') in each line. | bytes (''-b //spec//'') / fields (''-f //spec//'') in each line. | ||
| \\ | \\ | ||
| - | A filed is any number of characters separated by a single-character | + | A field is any number of characters separated by a single-character |
| delimiter (''-d //delim//'', defaults to tab). | delimiter (''-d //delim//'', defaults to tab). | ||
| Line 416: | Line 440: | ||
| ''egrep '^[Ee]{2}' /usr/share/myspell/en_US.dic'' a slash and all that follows | ''egrep '^[Ee]{2}' /usr/share/myspell/en_US.dic'' a slash and all that follows | ||
| it. | it. | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:cut-en&inline}} ++++ | ||
| ==== sort ==== | ==== sort ==== | ||
| Line 460: | Line 486: | ||
| ~~Exercise.#~~ Sort the file by the second column (alphabetically) and by the third column (numerically). | ~~Exercise.#~~ Sort the file by the second column (alphabetically) and by the third column (numerically). | ||
| </small> | </small> | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:sort&inline}} ++++ | ||
| ==== wc, uniq, nl ==== | ==== wc, uniq, nl ==== | ||
| Line 492: | Line 520: | ||
| \\ | \\ | ||
| Then pipe it through ''sort'' and ''uniq'' so that you see how many times each result was hit. | Then pipe it through ''sort'' and ''uniq'' so that you see how many times each result was hit. | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:wc_uniq&inline}} ++++ | ||
| <small> | <small> | ||
| Line 526: | Line 556: | ||
| * to squeeze all spaces | * to squeeze all spaces | ||
| * remove all the letters ''rwx'' | * remove all the letters ''rwx'' | ||
| + | |||
| + | ++++ Examples | {{page>so:pipes_filters:tr-en&inline}} ++++ | ||
| <small> | <small> | ||
| Line 594: | Line 626: | ||
| <small> | <small> | ||
| ~~Exercise.#~~ Open a PDF file (e.g., ''/usr/share/doc/packages/apparmor-docs/techdoc.pdf'') with less, with and without -L option. \\ | ~~Exercise.#~~ Open a PDF file (e.g., ''/usr/share/doc/packages/apparmor-docs/techdoc.pdf'') with less, with and without -L option. \\ | ||
| - | Open a ''tar'' archive with less (e.g., ''/usr/share/doc/packages/automake/amhello-1.0.tar.gz''). | + | Open a ''tar'' archive with less (e.g., ''/usr/share/doc/packages/automake/amhello-1.0.tar.gz'').\\ |
| + | View a directory (e.g., ''/usr/include'') with less. | ||
| </small> | </small> | ||
os_cp/redirects_pipes.1679444861.txt.gz · Last modified: 2023/03/22 01:27 by jkonczak
Except where otherwise noted, content on this wiki is licensed under the following license: Creative Commons Attribution-ShareAlike 4.0 International License
