Created at:

sh notes

General notes about some idiosyncrasies of sh programming

After years making shell programming trying to sticky to Bourne shell, always, but making use of some extensions when they exist (like the local keyword for declaring local variables) I decided to write this part of this document with some notes, specially on some features that are not very good documented.

The shell I use to base my tests is the NetBSD /bin/sh.

Read a file or a output of a command line-by-line

A nice way to handle lines of a previous command is to pipe the command to while read, see:: 

    m4 file.m4 | while read l; do
        do_something "$l"
    done

But it is executed in a subshell, so trying to set variables inside it and use them later won't work. Take a look at this example:: 

    last="(null)"
    cat /etc/fstab | while read l; do
        last="$l"
    done
    echo "$last"

This example will print "(null)".

Also, all examples regarding while + read have problems. They will wrongly handle lines that start with space (chopping them off). Also, read handles backslashes in a special way. Take a look at your shell's man page and the -r flag of the read builtin. A very nice page about that and other limitations and security problems in shell is in "Filenames and Pathnames in Shell: How to do it Correctly" of "David A. Wheeler Personal Home Page". We will take his advices when writing shell code.

How to do it Correctly: Filenames and Pathnames in Shell

David A. Wheeler Personal Home Page

If you want to use variables or any feature not possible when using a subshell, use the less elegant way:: 

    last="(null)"
    for l in $(cat /etc/fstab); do
        last="$l"
    done
    echo "$last"

But this will not work as we expect too, because $() (or evaluating use back quotes: `) will treat not only newlines as separators but spaces and tabs as well.

A solution that works is to change the $IFS variable that holds what are the characters considered separators when expanding command output, variable, etc. In the following example I'm setting it to the newline character. See:: 

    last="(null)"
    old_IFS="$IFS"
    IFS="
    "
    for l in $(cat /etc/fstab); do
        last="$l"
    done
    IFS="$old_IFS"
    echo "$last"

Usage of pipes with loops in shell

Linux read file line by line: for loop

A much better approach seem to read the file from stdin, using a redirection:: 

    last="(null)"
    while read l; do
        last="$l"
    done < /etc/fstab
    echo "$last"

No part of this code will run in a subshell, so any variable change of the loop will kept to the outside part.

Call functions with ` or $() to get their output

It is common to call commands and local functions with ` or $() to store their output:: 

    output="$(func)"

Although it is not a good pattern, because of shell programming limitations you might want to set a global variable inside your function. The problem is that functions inside ` and $() are executed inside a subshell. See this example:: 

    var=no
    func () {
            var=yes
            echo "output"
    }
    output="$(func)"
    echo "$output"
    echo "$var"

So, it is recommended that you look for another architecture.