Intro
My previous post took off far more than I expected, so I thought I’d write another piece on less well-known bash features.
As I said before, because I felt bash is an often-used (and under-understood) technology, I ended up writing a book on it while studying it up. It’s really gratifying to know that other people think it’s important too, despite being un-hip.
A preview of the book is available here. It focusses more than these articles on ensuring you are drilled in and understand the concepts you need to take your bash usage to a higher level. This is written more for ‘fun’.
HN discussion here.
1) ^x^y^
A gem I use all the time.
Ever typed anything like this?
$ grp somestring somefile -bash: grp: command not found
Sigh. Hit ‘up’, ‘left’ until at the ‘p’ and type ‘e’ and return.
Or do this:
$ ^rp^rep^ grep 'somestring' somefile $
One subtlety you may want to note though is:
$ grp rp somefile $ ^rp^rep^ $ grep rp somefile
If you wanted rep to be searched for, then you’ll need to dig into the man page and use a more powerful history command:
$ grp rp somefile $ !!:gs/rp/rep grep rep somefile $
I won’t try and explain this one here…
2) pushd / popd
This one comes in very handy for scripts, especially when operating within a loop.
Let’s say you’re in a for loop moving in and out of folders like this:
for d1 in $(ls -d */)
do
# Store original working directory.
original_wd="$(pwd)"
cd "$d1"
for d2 in $(ls -d */)
do
pushd "$d2"
# Do something
popd
done
# Return to original working directory
cd "${original_wd}"
done
You can rewrite the above using the pushd stack like this:
for d1 in $(ls -d *) do pushd "$d1" for d2 in $(ls -d */) do pushd "$d2" # Do something popd done popd done
Which tracks the folders you’ve pushed and popped as you go.
Note that if there’s an error in a pushd you may lose track of the stack and popd too many time. You probably want to set -e in your script as well (see previous post)
There’s also cd -, but that doesn’t ‘stack’ – it just returns you to the previous folder:
cd ~ cd /tmp cd blah cd - # Back to /tmp cd - # Back to 'blah' cd - # Back to /tmp cd - # Back to 'blah' ...
3) shopt vs set
This one bothered me for a while.
What’s the difference between set and shopt?
sets we saw before, but shopts look very similar. Just inputting shopt shows a bunch of options:
$ shopt cdable_vars off cdspell on checkhash off checkwinsize on cmdhist on compat31 off dotglob off
I found a set of answers here.
Essentially, it looks like it’s a consequence of bash (and other shells) being built on sh, and adding shopt as another way to set extra shell options.
But I’m still unsure… if you know the answer, let me know.
4) Here Docs and Here Strings
‘Here docs’ are files created inline in the shell.
The ‘trick’ is simple. Define a closing word, and the lines between that word and when it appears alone on a line become a file.
Type this:
$ cat > afile << SOMEENDSTRING > here is a doc > it has three lines > SOMEENDSTRING alone on a line will save the doc > SOMEENDSTRING $ cat afile here is a doc it has three lines SOMEENDSTRING alone on a line will save the doc $
Notice that:
- the string could be included in the file if it was not ‘alone’ on the line
- the string SOMEENDSTRING is more normally END, but that is just convention
Lesser known is the ‘here string’:
$ cat > asd <<< 'This file has one line'
5) String Variable Manipulation
You may have written code like this before, where you use tools like sed to manipulate strings:
$ VAR='HEADERMy voice is my passwordFOOTER' $ PASS="$(echo $VAR | sed 's/^HEADER(.*)FOOTER/1/')" $ echo $PASS
But you may not be aware that this is possible natively in bash.
This means that you can dispense with lots of sed and awk shenanigans.
One way to rewrite the above is:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ PASS="${VAR#HEADER}"
$ PASS="${PASS%FOOTER}"
$ echo $PASS
- The # means ‘match and remove the following pattern from the start of the string’
- The % means ‘match and remove the following pattern from the end of the string
The second method is twice as fast as the first on my machine. And (to my surprise), it was roughly the same speed as a similar python script.
If you want to use glob patterns that are greedy (see globbing here) then you double up:
$ VAR='HEADERMy voice is my passwordFOOTER'
$ echo ${VAR##HEADER*}
$ echo ${VAR%%*FOOTER}
$
6) ariable Defaults
These are very handy for knocking up scripts.
If you have a variable that’s not set, you can ‘default’ them by using this. Create a file called default.sh with these contents
#!/bin/bash
FIRST_ARG="${1:-no_first_arg}"
SECOND_ARG="${2:-no_second_arg}"
THIRD_ARG="${3:-no_third_arg}"
echo ${FIRST_ARG}
echo ${SECOND_ARG}
echo ${THIRD_ARG}
Now run chmod +x default.sh and run the script with ./default.sh first second.
Observer how the third argument’s default has been assigned, but not the first two.
You can also assign directly with ${VAR:=defaultval} (equals sign, not dash) but note that this won’t work with positional variables in scripts or functions. Try changing the above script to see how it fails.
7) Traps
The trap builtin can be used to ‘catch’ when a signal is sent to your script.
Here’s an example I use in my own cheapci script:
function cleanup() {
rm -rf "${BUILD_DIR}"
rm -f "${LOCK_FILE}"
# get rid of /tmp detritus, leaving anything accessed 2 days ago+
find "${BUILD_DIR_BASE}"/* -type d -atime +1 | rm -rf
echo "cleanup done"
}
trap cleanup TERM INT QUIT
Any attempt to CTRL-C, CTRL- or terminate the program using the TERM signal will result in cleanup being called first.
Be aware:
- Trap logic can get very tricky (eg handling signal race conditions)
- The KILL signal can’t be trapped in this way
But mostly I’ve used this for ‘cleanups’ like the above, which serve their purpose.
8) Shell Variables
It’s well worth getting to know the standard shell variables available to you. Here are some of my favourites:
RANDOM
Don’t rely on this for your cryptography stack, but you can generate random numbers eg to create temporary files in scripts:
$ echo ${RANDOM}
16313
$ # Not enough digits?
$ echo ${RANDOM}${RANDOM}
113610703
$ NEWFILE=/tmp/newfile_${RANDOM}
$ touch $NEWFILE
REPLY
No need to give a variable name for read…
$ read
my input
$ echo ${REPLY}
LINENO and SECONDS
Handy for debugging
$ echo ${LINENO}
115
$ echo ${SECONDS}; sleep 1; echo ${SECONDS}; echo $LINENO
174380
174381
116
Note that there are two ‘lines’ above, even though you used ; to separate the commands.
TMOUT
You can timeout reads, which can be really handy in some scripts
#!/bin/bash
TMOUT=5
echo You have 5 seconds to respond...
read
echo ${REPLY:-noreply}
9) Extglobs
If you’re really knee-deep in bash, then you might want to power up your globbing. You can do this by setting the extglob shell option. Here’s the setup:
shopt -s extglob
A="12345678901234567890"
B=" ${A} "
Now see if you can figure out what each of these does:
echo "B |${B}|"
echo "B#+( ) |${B#+( )}|"
echo "B#?( ) |${B#?( )}|"
echo "B#*( ) |${B#*( )}|"
echo "B##+( )|${B##+( )}|"
echo "B##*( )|${B##*( )}|"
echo "B##?( )|${B##?( )}|"
Now, potentially useful as it is, it’s hard to think of a situation where you’d absolutely want to do it this way. Normally you’d use a tool better suited to the task (like sed) or just drop bash and go to a ‘proper’ programming language like python.
10) Associative Arrays
Talking of moving to other languages, a rule of thumb I use is that if I need arrays then I drop bash to go to python (I even created a Docker container for a tool to help with this here).
What I didn’t know until I read up on it was that you can have associative arrays in bash.
Type this out for a demo:
$ declare -A MYAA=([one]=1 [two]=2 [three]=3)
$ MYAA[one]="1"
$ MYAA[two]="2"
$ echo $MYAA
$ echo ${MYAA[one]}
$ MYAA[one]="1"
$ WANT=two
$ echo ${MYAA[$WANT]}
Note that this is only available in bashes 4.x+.
This is based on some of the contents of my book Learn Bash the Hard Way
Source: https://zwischenzugs.com/2018/01/21/ten-more-things-i-wish-id-known-about-bash/