An easy thus thorough tutorial on sed, the Stream EDitor

Sed is a command line tool that can really do amazing stream manipulations: despite it certainly being a "seasoned" tool, it is very likely that there are a lot of sed one-liners inside your Company's scripts.

Having at least an understanding of it is a must if you want to be able to maintain this legacy stuff that very often is not worth the effort to rework.

And anyway, when having to deal with quick and dirty solutions that rely on shell scripts, or when writing documentation with shell commands that can be easily replaced by a copy and paste by the reader... it's still an excellent tool honestly I cannot work without.

The aim of this post is to provide an easy tutorial to quickly learn how to use sed in every situation that can be easily sorted out with a sed one-liner.

In memory of Lee E. McMahon, contributor to early versions of the Unix operating system, ... and of course in particular of the sed stream editor.

Acquainting To Sed

The Stream EDitor, broadly known as sed is a tool to perform transformations on an input stream, either a file or input from a pipeline.

Sed is not the only one utility that can perform this kind of transformations, there are other tools, more specialized, such as tr that anyway cannot do everything you can do with sed, or more general purpose, such as awk, that instead is actually a scripting language, although not powerful as perl.

So, as the rule of thumb, we can assume that it is wise to consider using sed when awk, tr or perl cannot do the job in an easier or more performing way.

Do not use sed if specialized tools specific to manage the particular format of the stream are available! For example, if you are dealing with XML, use XML Starlet instead. If you want to know more about this topic, please read my post XML in a nutshell.

Shell Invocation

The basic shell invocation of sed is as follows:

sed 'list;of;expressions' optional_file_to_process_1 optional_file_to_process_2

the above invocation tells sed to load the stream to process from two files: you can specify as many files as you need since the last arguments that are not command options are considered files to load streams from.

You can however pass the stream also by piping to sed: simply pipe the stream to sed and do not specify any file, for example:

echo 'a stream to process' | sed 'list;of;expressions'

Extended Regular Expressions Support

Although sed can handle extended regular expressions, its support is disabled by default: if you want to use them then you must explicitly enable them by specifying -r command option.

Specifying Expressions

As for the expressions to run, sed can load them for processing the input stream either:

from an expressions file
reading expressions specified as a one-liner

Probably you are most used to seeing it with one-liners, that is by the way is the most common use case when dealing with shell scripts.

Please note that it is not mandatory for one liners to be of a single expression only: you can specify more than one expression in a one-liner by separating them using a semicolon (;).

The following snippet is an example of a one-liner with two expressions:

STRING="one two three one four"
echo ${STRING}|sed 's/one/1/g;s/two/2/g'

the outcome is as follows:

1 2 three 1 four

it executes both of the following expressions:

s/one/1/g
s/two/2/g

An alternate way is to provide each expression as a single argument using the -e command option:

STRING="one two three one four"
echo ${STRING}|sed -e 's/one/1/g' -e's/two/2/g'

Let's see now an example of how to do the same thing using a expressions file:

first simply create the foo.sed expressions file with the following contents:

# an example expression file
s/one/1/g
s/two/2/g

Lines with the '#' character as the first non-white-space character are considered comments. Some, but not all, sed versions consider the text after the '#' character as comment.

then run sed using -f command option to specify the expressions file to be processed:

STRING="one two three one four"
echo ${STRING} | sed -f foo.sed

the outcome is of course same as the before one:

1 2 three 1 four

note that same way as you can do with -e option, if necessary you can specify multiple expressions files supplying them using additional -f option.

It is even possible to combine expressions files and one-liner by within the same command by using -e command option to provide one-liners:

echo ${STRING} | sed -f foo.sed -e 's/three/3/' -f bar.sed

It's up to you to decide the style of invocation to use, depending on your use case and to the readability you want your code to have.

In-Place File Modification

Of course sed has been developed to process input streams, but it is mostly used to modify files.

As an example, let's configure chronyd to use the servers of the Swiss NTP pool - our target file is "/etc/chrony.conf": our task is simply replacing (change) the line that specifies the ntp pool to use.

To accomplish this task, we need a expression that focuses on the line that begins with "pool" - we can express this using the following regex:

^pool

then the expression must

specify that we want to change (replace) the whole line: this is achieved by specifying the c command
specify the argument of the c command, that is the new text

The whole change command along with its argument is as follows:

c server 0.ch.pool.ntp.org\nserver 1.ch.pool.ntp.org\nserver 2.ch.pool.ntp.org\nserver 3.ch.pool.ntp.org

since the new text spreads across more than one line, we need to insert the line feed control character "\n" to mark the end of each line.

So, to see everything as a whole, the complete shell command is:

sed '/^pool /c \server 0.ch.pool.ntp.org\nserver 1.ch.pool.ntp.org\nserver 2.ch.pool.ntp.org\nserver 3.ch.pool.ntp.org' /etc/chrony.conf

if you run it you only get the modified stream printed to the standard output: this is the default behavior of sed indeed.

Have a look at the "/etc/chrony.conf" file: it has still not been modified yet.

If we really want sed to perform an in-place modification of a stream that comes from a file, we must explicitly specify it by supplying the -i option.

Let's try the same command with the -i option:

sudo sed -i '/^pool /c \server 0.ch.pool.ntp.org\nserver 1.ch.pool.ntp.org\nserver 2.ch.pool.ntp.org\nserver 3.ch.pool.ntp.org' /etc/chrony.conf

this time the command does not print anything to the standard output and performs the file modification.

If you were really configuring chrony with the above sed one-liner, you must also restart chronyd systemd unit so to have it loading the new configuration.

Getting The Hang Of It

Since the purpose of sed is to modify the stream, it is reasonable that it prints the whole modified stream by default. However this is boring especially when dealing with long files: you have to scroll back the whole stream to check if your expression is actually working as expected.

So this is the very first thing to learn with sed:

find out a handy way to guess and check expressions that match lines and text in the stream.

My personal advice is to rely on the -n sed option, that disables the printing of the stream, and use the "p" option of the command used in the sed expression.

Please re-read the above sentence carefully: the "p" I'm talking about is not an option of the sed command line utility itself, rather than an option of the command specified inside the expression.

as an example, let say that we have a dedicated partition for the filesystem of temporary files mounted on the "/tmp" and that our "/etc/fstab" contains an entry like the following one:

LABEL=tmp /tmp xfs defaults,noatime 0 0

let's see how to configure to mount the partition on "/tmp" with the following restrictions:

preventing the execution of files (noexec mount option)
preventing the use of the setuid POSIX bit (nosuid mount option)
preventing the creation of device file (nodev mount option)

this require to:

select the line containing the " /tmp " pattern
replace default options with "default,noexec,nodev"

the complete sed one-liner is:

sed  '/ \/tmp / s/default[^ ]/default,noexec,nodev,nosuid/' /etc/fstab

It is necessary to escape with a back-slash "\" the slash "/"of the "/tmp" path to avoid confusion with the slashes that enclose the pattern of the matching clause that triggers the "s" command.

As you see it prints the whole modified fstab, ... that of course gets modified only if our patterns do match.

Printing Only What It Gets Processed

We can now disable printing the whole stream to standard output ( -n option of the sed command line utility) and enable printing only of the contents of the for modified lines ("p" option of the s command) as follows:

sed -n '/ \/tmp / s/default[^ ]/default,noexec,nodev,nosuid/p' /etc/fstab

this time the output is only:

LABEL=tmp /tmp xfs default,noexec,nodev,nosuid,noatime 0 0

now we clearly see that, ... it's not working as expected: we wanted to replace the mount options, ... but the noatime option is still there.

This means that there's something wrong in the regular expression we wrote. Let's fix it:

sed -n '/ \/tmp / s/default[^ ]*/default,noexec,nodev,nosuid/p' /etc/fstab

this time the output is only:

LABEL=tmp /tmp xfs default,noexec,nodev,nosuid 0 00

so now it's really working as expected.

This means that we can modify the file, so remove the -n and "p" options we just added.

Mind the trick of disabling the printing to the standard output and enabling the printing only of matched commands: when dealing with large files, this is far more easy than having to scroll back the whole output to see if our sed expression worked as expected.

Create A Backup Of The Original File

Now, to perform the actual modification to the file, specify the -i command line parameter with a trailing ".bak":

sed -i.bak '/ \/tmp / s/default[^ ]*/default,noexec,nodev,nosuid/' /etc/fstab

this causes sed to perform an in-place modification of the file and to create within the same directory of the original file a backup file with the same name of the source file but with a trailing ".bak".

Although it is not really necessary, it is wise to provide this option followed by a suffix: when a suffix is specified a backup copy of the original file is taken, and this is certainly handy to rollback things if necessary.

So let's have a look to the differences from "/etc/fstab" and "/etc/fstab.bak":

diff /etc/fstab /etc/fstab.bak

the output is:

15c15
< LABEL=tmp /tmp xfs default,noexec,nodev,nosuid 0 0
---
> LABEL=tmp /tmp xfs defaults,noatime 0 0

Pattern Space

Sed reads the stream line by line: each time a line is read, it removes the trailing newline and puts its contents in the so-called pattern space. This buffer:

gets overridden at each read cycle
is the source of the stream processed by the commands
is also used to store the outcome of the commands, which replaces its original contents. If there are other commands or expressions, they of course get the outcome of the previous command or expression as the source stream

sed by default is line-oriented - bare this this is why you'll never find a line feed within the pattern space. The consequence is that you cannot have a regular expression that matches a pattern that spans onto two lines.

Anyway you can control how sed loads the pattern space using the following commands:

n read the next line from the stream and put it into the pattern space (overwriting the current contents)
N read the next line from the stream and append it to the pattern space

For instance, if you have a record that is more than a single line in length each, you can force sed to load the number of additional lines you need to the pattern space by using the N command.

For example, let's create the "os.txt" file with the following contents:

Id=1
Name=CentOS Linux
Version=8
Id=2
Name=CentOS Linux
Version=7
Id=3
Name=CentOS Linux
Version=6

Our goal is to replace the value of the "Name" field with the string "Rocky Linux" only for the record with "Version=8"

As you can see, each record is three lines in length and contains the fields "Id", "Name" and "Version".

Since the first line gets loaded in the patter space at each read cycle, we must add the N command two times so to load the next two lines and have the whole record loaded in the pattern space:

N;N

we must use an expression that matches the "Version=8" pattern as triggering criteria for the commands:

/Version=8/

and use a command that replaces of the value of the "Name" field with the string "Rocky Linux" as follows:

s/Name=[^ \n]*/Name=Rocky Linux/

let's put everything together: this is the whole sed shell command:

sed 'N;N;/Version=8/ s/Name=[^ \n]*/Name=Rocky Linux/' os.txt

the output is:

Id=1
Name=Rocky Linux Linux
Version=8
Id=2
Name=CentOS Linux
Version=7
Id=3
Name=CentOS Linux
Version=6

Hold Space

Sed lets you use an additional buffer called hold space: conversely from the pattern space that gets overwritten by the next line of the stream at each read, the hold space persists until the end.

It's purpose is providing a buffer where if necessary you can store data you can recall when needed.

Sed provides a few commands to exploit the hold space:

h copy pattern space to hold space (overwriting the current contents)
H append pattern space to hold space
g copy hold space to pattern space (overwriting the current contents)
G append hold space to pattern space

as an example to play with, modify the "os.txt" file we created a while ago so that it looks like as follows:

Id=1,Name=CentOS
Linux,Version=8
Id=2,Name=CentOS Linux,Version=7
Id=3,Name=CentOS Linux,Version=6

our goal is the same: replace the string "CentOS Linux" on the row with "Version=8" with the new string "Rocky Linux".

The straightforward problem is that this time we do not have a record with a fixed number of lines: we have to run a replace command with a regular expression that must match across two lines, ... and now you know that sed by default is line oriented, so it cannot work unless we use a trick.

Since the size of the file is small, we can:

load the file as a whole on the hold space (we append it line by line at each read using the H command)
after the last read - note the use of the dollar ($) character, that means the end of the stream - replace the content of the pattern space with the contents of the hold space
after the last read, perform the substitution using the s command - since we have the whole file in the pattern space, we can match the regular expression across more than just one line

Mind the size of the amount of data you send to the hold space - this is an extreme example where we load everything in the hold buffer, but usually you can do more tailored things.

the whole sed expression is as follows:

sed -n 'H;$g;$s/CentOS[ ]*\n*[ ]*Linux,Version=8/Rocky Linux,Version=8/g;$p' os.txt

the output is:

Id=1,Name=Rocky Linux,Version=8
Id=2,Name=CentOS Linux,Version=7
Id=3,Name=CentOS Linux,Version=6

since sed prints the contents of the pattern space at each read cycle, we must disable the printing (-n option of the sed command line) and add the "p" option to the s command so to print the outcome of the processing: since we have processed the whole file, the whole file content gets printed.

While making trials with copying/appending data into hold space and pattern space, you may benefit using the l command: it "lists out the current line in a visually unambiguous form".

Sed Expression Syntax

The very first thing to learn to work with sed is the basic syntax of a sed expression:

/LINE_MATCHING_CLAUSE/ statement

as you see from above, an expression is made of:

an optional LINE_MATCHING_CLAUSE
the statement

the LINE_MATCHING_CLAUSE is the filter that must match so as to have the statement run against the contents currently loaded into the pattern space.

If the LINE_MATCHING_CLAUSE is omitted, then the match always succeeds and so any the statement is always run.

A sed statement can either be:

a command, along with its options if necessary. If necessary, you can even specify a list of commands
another sed expression - so another statement preceded by another LINE_MATCHING_CLAUSE

when the statement is a command, the expression looks like as follows:

/LINE_MATCHING_CLAUSE/ command/OPTIONS/

when the statement is another expression, the straightforward syntax template is:

/LINE_MATCHING_CLAUSE/ expression

so things get a more complicated, since the expression can really be a lot of things: see Nesting Expressions to know how to deal with this.

Nesting Commands

we can of course nest a list of commands within the same LINE_MATCHING_CLAUSE by:

enclosing the list by curly braces {}
separate list items using semicolons (;)

The syntax is as follows:

/LINE_MATCHING_CLAUSE/ {command/OPTIONS/;command/OPTIONS/}

mind that:

each command runs only if the LINE_MATCHING_CLAUSE matches
nested commands run in the order they are listed

that is: if LINE_MATCHING_CLAUSE does not match, no command of the list is run.

Let's see an example of nested commands within the same LINE_MATCHING_CLAUSE - let's pretend that we have the "/etc/exports.d/apps.exports" NFS exports file with the following contents:

/srv/nfs/foo 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)
/srv/nfs/bar 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)
/srv/nfs/baz 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)

and we need to alter only to the "/srv/nfs/bar" share as follows:

add the "no_all_squash" NFS export option
modify the subnet mask of the subnet that is granted access to the share to "255.255.240.0"

Please mind that since this is a tutorial and we are just experimenting, we don't want to modify the file contents: we just want to see the outcome of the line that is affected by the modification. That's why we disable printing of the whole stream and print only the modified stream.

The whole sed command looks like as follows:

sed -n '/^\/srv\/nfs\/bar / {s/\(rw[^)]\)/\1no_all_squash,/;s/255\.255\.255\.0/255.255.240.0/p}' /etc/exports.d/apps.exports

where:

LINE_MATCHING_CLAUSE: "^\/srv\/nfs\/bar "
1^st command: "s/$rw[^)]$/\1no_all_squash,/"
2^nd command: "s/255\.255\.255\.0/255.255.240.0/"

Let's run it - the output must be as follows:

/srv/nfs/bar 192.168.0.0/255.255.240.0(rw,no_all_squash,root_squash,sec=krb5:krb5i:krb5p)

if there would be no output or an error message, then it means that our one liner failed somewhere.

Nesting Expressions

Same way we can nest commands within the same LINE_MATCHING_CLAUSE, we can also nest expressions by using curly braces {}

The syntax is as follows:

/LINE_MATCHING_CLAUSE/ {/LINE_MATCHING_CLAUSE/ statement}

of course you can nest more things as needed:

/LINE_MATCHING_CLAUSE/ {/LINE_MATCHING_CLAUSE/ {/LINE_MATCHING_CLAUSE/ statement}}

The behavior is the same of a nested if block, so mind that, in order to run the statement:

all LINE_MATCHING_CLAUSE must match
because of the nesting the outer LINE_MATCHING_CLAUSE must match before the inner ones

Using multiple expressions

Sometimes we want to specify multiple expressions having the LINE_MATCHING_CLAUSE separately evaluated, avoiding that an unmatched clause prevents the other expressions from running.

As we already saw, besides supplying multiple expressions using the -e parameter for each of them you can also simply list them separated by a semicolon ;

/LINE_MATCHING_CLAUSE/ statement;/LINE_MATCHING_CLAUSE/ statement;

Line Matching Clauses

This kind of clause lets you specify a per-line matching criteria, that can either be a line number or a regular expression.

For the reader's convenience, in all of the following examples, we intentionally disable output to stdout (-noption) and use the p command to print lines that match.

Match Line By line Number

This is the first and most straightforward line matching clause: for example the following expression prints the 7^th line of the "/usr/share/doc/curl/README" file:

sed -n '7p' /usr/share/doc/curl/README

where:

7 is the line number
p is the print command, that causes sed to print the contents of the line

the output is as follows:

README

Match Line By Regular Expression Pattern

More often we want to focus on lines that match a specific pattern - we already saw this use case in action:

sed -n '/ \/tmp / s/default[^ ]*/default,noexec,nodev,nosuid/p' /etc/fstab

since the clause is not a number, sed guesses that we are not interested in a certain line number: this time sed finds the " \/tmp " string enclosed between slashes (/) - this means that the clause is a pattern expressed using a regular expression.

Beware of matching is case-sensitive: if you need not to distinguish between upper-case and lowercase you must specify the "I" option right after the last slash enclosing the pattern. So the pattern would become something like "/ \/tmp /I ".

So the above sed expression contains a matching pattern clause that limits the substitute commands to only the lines that contain the " /tmp " pattern. If the line matches, then the substitute command is run replacing the mount options with the string "default,noexec,nodev,nosuid".

The output would be something like as follows:

LABEL=tmp /tmp xfs default,noexec,nodev,nosuid 0 00

Line Interval Matching Clauses

For the reader's convenience, in all of the following examples, we intentionally disable output to stdout (-noption) and use the p command to print lines that match.

This kind of clause lets you specify the begin and the end criteria that identify the interval of interesting lines: please mind that the begin and the end criteria can either be the line number or a regular expression pattern.

The begin and the end criteria that identify the interval of interesting lines are separated by a comma character (,).

Match Lines By An Interval Of Line Numbers

This is the most straightforward clause we can use to match an interval of lines: simply specify the number of the line where the interval begins and where the interval ends.

For example, to match the lines from 1 to 6:

sed -n '1,6p' /usr/share/doc/curl/README

where:

1,6 is the interval
p is the print command, that causes sed to print the contents of the line

The output is as follows:

      _ _ ____  _
 ___| | | | _ \ | |
/ __| | | | |_) | |
|  (__| |_| | _<| |___
\___|\___/|_| \_\_____|

Match Lines By Start And End Markers

This applies when we want to focus on a marker based interval: it is a quite common use case when having to configure files structured into stanzas.

As an example, let's modify "Nobody-User" and "Nobody-Group" values into the "[Mapping]" stanza of the "/etc/idmapd.conf" file.

Let's start by typing an expression that prints the lines that we are interested to modify:

sed -n '/\[Mapping\]/,/^\[/p' /etc/idmapd.conf

the output is as follows:

[Mapping]

#Nobody-User = nobody
#Nobody-Group = nobody

[Translation]

it prints just what we expect it to match, so the expression is good. We can now complete the sed expression by adding the substitution commands - we can put them right before the print command, so to have it printed to the standard output to quickly check if it is working as expected:

sed -n '/\[Mapping\]/,/^\[/{s/^[ ]*#[ ]*Nobody-User[ ]*=.*/Nobody-User = nfsnobody/;s/^[ ]*#[ ]*Nobody-Group[ ]*=.*/Nobody-Group = nfsnobody/;p}' /etc/idmapd.conf

the output is as follows:

[Mapping]

Nobody-User = nfsnobody
Nobody-Group = nfsnobody

[Translation]

that is exactly what we want to achieve, so our trials end here: we have the right expression.

Now, if we really want to go on and actually modify "/etc/imapd.conf" file, we must turn it into:

sed -i.bak '/\[Mapping\]/,/^\[/{s/^[ ]*#[ ]*Nobody-User[ ]*=.*/Nobody-User = nfsnobody/;s/^[ ]*#[ ]*Nobody-Group[ ]*=.*/Nobody-Group = nfsnobody/}' /etc/idmapd.conf

From A Marker To The End Of The File

You may need to extend the line interval up to the end of the file: simply specify "$" as end marker.

For example, to start matching as soon as the "[Static]"stanza is reached until the end of the file:

sed -n '/\[Static\]/,$ p' /etc/idmapd.conf

Match An Interval Of Lines Mixing Line Number and Marker

This applies to only these two use cases:

select an interval of lines from a line number to a marker
select an interval of lines from a marker to a line number

A straightforward use case is picking the interval of lines from the first line to a marker.

From The Beginning Of The File To A Marker

You must specify the number 1 as the beginning, and the pattern as the end.

For example, to match since the beginning of the file up to the "[Static]"word that delimits the beginning of the Static stanza is reached:

sed -n '1,/\[Static\]/ p' /etc/idmapd.conf

Negating The Line Matching Clause

A command can be configured so to process when the matching clause is false: it is enough to prepend an exclamation mark (!) to the command - look at the delete (d) command to see an example of this in action.

Sed Commands - The Most Used Ones

After getting acquainted to the matching clauses that can be used to trigger commands, it has come the time to see the most used sed commands:

In the following examples I intentionally omit the -i parameter to avoid readers running them by mistake without having a clear understanding of what we are doing. Of course, if you really want to have the command to actually modify the original files, you must add -i option and use sudo if administrative rights are needed.

Append (a)

The lowercase a command is used to append text after the lines that match LINE_MATCHING_CLAUSE.

For example:

sudo sed '/Subsystem[ ]*.*/a \\nMatch User foosvc\n PermitTTY no\n X11Forwarding no' /etc/ssh/sshd_config

add a Match rule to SSHd config file with directives that prevent the foosvc user to set a TTY or to request X11 Forwarding.

Insert (i)

The lowercase i command is used to insert the supplied text before the lines that match LINE_MATCHING_CLAUSE.

The following example adds the "pam_time.so" module include to "/etc/pam.d/login":

sed '/^[ ]*account[ ]*required[ ]*/i account required pam_time.so' /etc/pam.d/login

The outcome is enabling time-based logins – hint: if you really want to use it you should also configure "/etc/security/time.conf".

Change (c)

The lowercase c command is used to replace the whole lines that match LINE_MATCHING_CLAUSE with the supplied text.

The following example replaces the parameters supplied to the password quality check pam module ("pam_pwquality.so"):

sed '/password[ ]*.*pam_pwquality\.so/c password requisite pam_pwquality.so try_first_pass local_users_only retry=3 minlen=8 minclass=3 authtok_type' /etc/pam.d/system-auth

This PAM module is often configured to set the passwd command to reject new passwords that do not meet the corporate password policy.

Delete (d)

The lowercase d command is used to delete the lines that match LINE_MATCHING_CLAUSE.

The following example removes "pam_xauth.so" from the "/etc/pam.d/su" file, thus disabling the automatic forwarding of xauth-keys from the switch to the switched user..

sed '/pam_xauth\.so/d' /etc/pam.d/su

It is of course possible to trigger the delete command only when a line does not match the LINE_MATCHING_CLAUSE: for example, let's pretend that we have the same "/etc/exports.d/apps.exports" NFS exports file we have already seen with the following contents:

/srv/nfs/foo 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)
/srv/nfs/bar 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)
/srv/nfs/baz 192.168.0.0/255.255.255.0(rw,root_squash,sec=krb5:krb5i:krb5p)

and we need to delete all the lines but the "/srv/nfs/bar", so as to preserve it as the only NFS share.

The sed expression to achieve this is:

sed '/^\/srv\/nfs\/bar / !d' /etc/exports.d/apps.exports

Note how the exclamation mark (!) triggers the command only when the LINE_MATCHING_CLAUSE is false.

Substitute (s)

The lowercase s command is used to substitute the text matching the substitute pattern in the lines that match LINE_MATCHING_CLAUSE.

It requires some arguments using the following syntax:

s/regex/replacement/flags

mind that:

slash character (/) is the default delimiter. This can be a little bit unhandy if any of the parameters contain backslashes, since it would require you to escape these backlashes so as to avoid confusion with the ones used as delimiters. You can overcome this by simply specifying another delimiter such as the plus(+) character, as we already did in one of our previous examples
regex is the regular expression used to find the text to be replaced inside the line
replacement is the text that replaces the matched text.
flags are optional flags that can be specified to alter s command default behavior.

Example of flags are:

i – enable case-insensitive matching of the supplied regex.

For example:

echo $STRING|sed 's/oNe/1/i'

matches the regex - the output is:

1 two three one four

g – apply to all the matches of the matched lines: the default behavior is to apply only to the first match of every matched line.

For example:

STRING="one two three one four"
echo $STRING|sed 's/one/1/'

the output is:

1 two three one four

since we did not supply the g option, so only the first match gets substituted.

Let's retry the same expression, but with the g option this time:

echo $STRING|sed 's/one/1/g'

the output is:

1 two three 1 four

number - skip the first number of matches - for example:

echo $STRING|sed 's/one/1/2'

the output is:

one two three 1 four

The following example locates the line used to implicitly trust users in the "wheel" group, that is commented by default, and "uncomments" it (substitute # with nothing)

sed '/pam_wheel.so trust/s/^[ ]*#//' /etc/pam.d/su

Back-references

Back-references are regular expression commands which refer to a previous part of the matched regular expression. The s command does support them and it lets us store up to 10 values that match the regex referenced by the number of the occurrence preceded by a backslash.

For example:

STRING='Dimensions: H="111", W="222", D="333"'
echo $STRING| sed 's/.*H="\([0-9]*\).*W="\([0-9]*\).*D="\([0-9]*\)"/Height=\1; Depth=\3; Width=\2/'

the output is:

V1=111; V3=333; V2=222

this example add "minlen=8" to the existing options supplied to pam_pwquality using a back-reference

sed '/password[ ]*.*pam_pwquality\.so/s/\(.*\)*/\1 minlen=8/' /etc/pam.d/system-auth

let's see a more complex example: this time we want to uncomment every NFSv4 directive in the UMICH_SCHEMA stanza of "/etc/idmapd.conf" file - let's begin by figuring out an expression that fit our needs:

sed -n '/\[UMICH_SCHEMA\]/,$ s/#\(NFSv4.*\)/\1/p' /etc/idmapd.conf

the outcome is:

NFSv4_person_objectclass = NFSv4RemotePerson
NFSv4_name_attr = NFSv4Name
NFSv4_uid_attr = UIDNumber
NFSv4_acctname_attr = uid
NFSv4_group_objectclass = NFSv4RemoteGroup
NFSv4_gid_attr = GIDNumber
NFSv4_group_attr = NFSv4Name
NFSv4_member_attr = memberUID

since it looks good, we can perform the change as follows:

sed -i '/\[UMICH_SCHEMA\]/,$ s/#\(NFSv4.*\)/\1/' /etc/idmapd.conf

look at the "/etc/idmapd.conf" file: every NFSv4 directive of that stanza must be uncommented now.

More On Sed Commands

As we already saw, there are other sed commands out of the ones explained above, such as the p command that prints the matching stream. Anyway I'm omitting them in this post since it's quite unlikely that you really need to use them unless in quite rare use cases.

Footnotes

Here it ends this tutorial on SED: I hope that after reading it you agree that despite it being a "seasoned" tool, it is something that can be a very valid ally to quickly create shell scripts that modify streams.

Of course nowadays, since we are more prone to develop Python scripts than shell scripts, it is less likely having to deal with it, but anyway mind that sometimes it is more cost-effective to write a shell script with sed than blindly starting writing Python code.

That's why I think that we are still far from the time when it will be over.

Please mind that this post does not explain everything about sed: there are a lot of nuances I omitted since it's quite unlikely that you really need to use them, and anyway I think that if things gets more complex than the ones I described, it's better to use something else, and maybe consider writing the script using a more powerful language such as Python.

Writing a post like this takes hours. I'm doing it for the only pleasure of sharing knowledge and thoughts, but all of this does not come for free: it is a time consuming volunteering task. This blog is not affiliated to anybody, does not show advertisements nor sells data of visitors. The only goal of this blog is to make ideas flow. So please, if you liked this post, spend a little of your time to share it on Linkedin or Twitter using the buttons below: seeing that posts are actually read is the only way I have to understand if I'm really sharing thoughts or if I'm just wasting time and I'd better give up.

Sed – the Stream EDitor

Acquainting To Sed

Shell Invocation

Extended Regular Expressions Support

Specifying Expressions

In-Place File Modification

Getting The Hang Of It

Printing Only What It Gets Processed

Create A Backup Of The Original File

Pattern Space

Hold Space

Sed Expression Syntax

Nesting Commands

Nesting Expressions

Using multiple expressions

Line Matching Clauses

Match Line By line Number

Match Line By Regular Expression Pattern

Line Interval Matching Clauses

Match Lines By An Interval Of Line Numbers

Match Lines By Start And End Markers

From A Marker To The End Of The File

Match An Interval Of Lines Mixing Line Number and Marker

From The Beginning Of The File To A Marker

Negating The Line Matching Clause

Sed Commands - The Most Used Ones

Append (a)

Insert (i)

Change (c)

Delete (d)

Substitute (s)

Back-references

More On Sed Commands

Footnotes

Leave a Reply Cancel Reply