Basic I/O Monitoring on Linux

The technique we discuss here is basic, it gives a good overview and is very easy to use. So let get focused… We will use iostat utility. In case you need you know where to find more about it — right, man pages.

So we will use the following form of the command:

iostat -x [-d]

• -x option displays extended statistics. You definitely want it.
• -d is optional. It removes CPU utilization to avoid cluttering the output. If you leave it out, you will get the following couple lines in addition:

  avg-cpu: %user %nice %sys %iowait %idle

  6.79 0.00 3.79 16.97 72.46

• is the number of seconds iostat waits between each report. Without a specified interval, iostat displays statistics since the system was up then exits, which is not useful in our case. Specifying the number of seconds causes iostat to print periodic reports where IO statistics are averaged for the time period since previous report. I.e., specifying 5 makes iostat dump 5 seconds of average IO characteristics, every 5 seconds until it’s stopped.

If you have many devices and you want to watch for only some of them, you can also specify device names on command line:

iostat -x -d sda 5

Now let’s get to the most interesting part — what those cryptic extended statistics are. (For readability, I formatted the report above so that the last two lines are in fact a continuation of the first two.)

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s

sda 0.00 12.57 10.18 9.78 134.13 178.84 67.07

wkB/s avgrq-sz avgqu-sz await svctm %util

89.42 15.68 0.28 14.16 8.88 17.72

• r/s and w/s— respectively, the number of read and write requests issued by processes to the OS for a device.
• rsec/s and wsec/s — sectors read/written (each sector 512 bytes).
• rkB/s and wkB/s — kilobytes read/written.
• avgrq-sz — average sectors per request (for both reads and writes). Do the math — (rsec + wsec) / (r + w) = (134.13+178.84)/(10.18+9.78)=15.6798597
  If you want it in kilobytes, divide by 2.
  If you want it separate for reads and writes — do you own math using rkB/s and wkB/s.
• avgqu-sz — average queue length for this device.
• await — average response time (ms) of IO requests to a device. The name is a bit confusing as this is the total response time including wait time in the requests queue (let call it qutim), and service time that device was working servicing the requests (see next column — svctim).

  So the formula is await = qutim + svctim.

• svctim — average time (ms) a device was servicing requests. This is a component of total response time of IO requests.
• %util — this is a pretty confusing value. The man page defines it as, Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. A bit difficult to digest. Perhaps it’s better to think of it as percentage of time the device was servicing requests as opposed to being idle. To understand it better here is the formula:

  utilization = ( (read requests + write requests) * service time in ms / 1000 ms ) * 100%
  or
  %util = ( r + w ) * svctim /10 = ( 10.18 + 9.78 ) * 8.88 = 17.72448

Traditionally, it’s common to assume that the closer to 100% utilization a device is, the more saturated it is. This might be true when the system device corresponds to a single physical disk. However, with devices representing a LUN of a modern storage box, the story might be completely different.

Rather than looking at device utilization, there is another way to estimate how loaded a device is. Look at the non-existent column I mentioned above — qutim — the average time a request is spending in the queue. If it’s insignificant, compare it to svctim — the IO device is not saturated. When it becomes comparable to svctim and goes above it, then requests are queued longer and a major part of response time is actually time spent waiting in the queue.

The figure in the await column should be as close to that in the svctim column as possible. If await goes much above svctim, watch out! The IO device is probably overloaded.

There is much to say about IO monitoring and interpreting results. Perhaps this is only the first of a series of posts about IO statistics. At Pythian we often come across different environments with specific characteristics and various requirements that our clients have. So stay tune — more to come.

Interpreting iostat Output

In this post I am going to explore how extended iostat statistics can be useful to a system administrator beyond a binary “Disk is bottleneck / Disk is not bottleneck.” Before we can get to any of that however, we must make sure we have a basic background knowledge of the Disk IO Subsystem.

Linux Disk IO Subsystem:

I am not a kernel hacker, so this overview might be flawed in parts but hopefully it is accurate enough to give the background needed for analyzing the output of iostat.

Layer	Unit	Typical Unit Size
User Space System Calls	read() , write()
Virtual File System Switch (VFS)	Block	4096 Bytes
Disk Caches	Page
Filesystem (For example ext3)	Blocks	4096 Bytes (Can be set at FS creation)
Generic Block Layer	Page Frames / Block IO Operations (bio)
I/O Scheduler Layer	bios per block device (Which this layer may combine)
Block Device Driver	Segment	512 Bytes
Hard Disk	Sector	512 Bytes

There are two basic system calls, read() and write(), that a user process can make to read data from a file system. In the kernel these are handled by the Linux Virtual Filesystem Switch (VFS). VFS is an abstraction to all file systems so they look the same to the user space and it also handles the interface between the file system and the block device layer. The caching layer provides caching of disk reads and writes in terms of memory pages. The generic block layer breaks down IO operations that might involve many different non-contiguous blocks into multiple IO operations. The I/O scheduling layer takes these IO operations and schedules them based on order on disk, priority, and/or direction. Lastly, the device driver handles interfacing with the hardware for the actual operations in terms of disk sectors which are usually 512 bytes.

A Little Bit on Page Caching:

The page cache caches pages of data that do or will reside on disk. Therefore before it writes data to disk it puts it in memory, and before it reads data from disk it checks to see if it is in memory already (With the exception of Direct IO). Writing pages out to disk actually gets deferred. This is done to increase performance so writes can be grouped together more efficiently. When a page of disk data gets changed and needs to be written out to disk it is called “dirty”. Since it is dangerous to keep pages in memory for too long in case of a system shutdown the kernel’s pdflush threads scan for dirty pages and then flushes them out to disk. Linux will actually try to use as much memory as it can for caching files which is why the top command usually shows so much used memory. When you want to see how much memory is free for processes you can run the free command and look at the ‘-/+ buffers/cache’.

iostat output:

So with this background lets look at some of the output of iostat and tie it together with our background knowledge. Iostat can break down the statistics at both the partition level and then device level, however in this post I am going to focus on the device level.

The Overview Statistics: “Is it Saturated or Not?”

From iostat there are two summary statistics which are Input/Output CPU wait time (iowait) and device utilization which are both expressed in terms of percentages.

iowait is from the CPU’s perspective and it is the percentage of time that the CPU spent waiting for a IO device to be ready. Another way to look at iowait is the amount of time that the CPU could have been doing something but couldn’t because all the processes were waiting on the disk or the network devices.

Device utilization is covered throughly by Alex Gorbahev in Basic I/O Monitoring on Linux. He summarizes it as “The percentage of time the device spent servicing requests as opposed to being idle.”

iostat and caching:

It is import to note that iostat shows requests to the device (or partition) and not read and write requests from user space. So in the table above iostat is reading below the disk cache layer. Therefore, iostat says noting about your cache hit ratio for block devices. So it is possible that disk IO problems might be able to be resolved by memory upgrades. From my research there is no way to pull out a cache hit/miss ratio out of Linux easily when it comes to block devices which is a bit disappointing. One suggestion from serverfault is to install a kernel with debuging symbols and use SystemTap to trace the VFS events and tie them together with the block layer events. I intend to explore this but I would prefer to see a way to get this data from /proc or /sys.

iostat Output for Random and Sequential Reads:

One of the main things to do when examining disk IO is to determine if the disk access patterns are sequential or random. This information can aid in our disk choices. When operations are random the seek time of the disk becomes more important. This is because physically the drive head has to jump around. Seek time is the measurement of the speed at which the heads can do this. For small random reads solid state disks can be a huge advantage.

So in fio I have created two different simple tests to run. The first is sequential reading, and the second is random reading. During these tests I ran iostat -x 3 throughout the test.

Snapshot of Random Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     0.00  172.67    0.00  1381.33     0.00     8.00     0.99    5.76   5.76  99.47

Snapshot of Sequential Read Test:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda              13.00     0.00  367.00    0.00 151893.33     0.00   413.88     2.46    6.71   2.72 100.00

What is more important to me for this is not just what these numbers are but what, in the context of of random vs sequential reading and in context of the IO subsystem, they mean.

The first two columns, rrqm/s and wrqm/s, are read and write requests merged per second. In my above diagram of the Linux Block IO subsystem above I mentioned that that the scheduler can combine operations. This can be done when multiple operations are physically adjacent to each other on the device. So in sequential operation it would make sense to often see a large number of merges. In the snapshot of the random reads, we see no merges. However, the merging layer feels a little bit like “magic” and I don’t believe it is the best indicator of if the patterns are random or sequential.

The next 5 columns are read and write requests to the device (r/s, w/s), followed by the amount of sectors read and written from the device (rsec/s, wsec/s), and then the size of each request (avgrq-sz). In the random test there are 172 reads that result in 1,381 sectors being read in. In the sequential test there are 367 read request to 151,893 sectors being read. So in the random test we get about 8 sectors per request and in the sequential test we get 413 sectors per read. If you look closely, this happens to be the same number as avgrq-sz which does this math for us (Sectors Read / Read Operations). However it is worth noting that this is how it is calculated as the average request size does not differentiate between reads and writes. From these tests a low sector write/read to request ratio or small request sizes seem to indicate a random IO profile. I believe this to be a better indicator than the amount of merges as to whether or not there is random or sequential disk patterns.

The final 4 columns are the average queue length of requests to the device (avgqu-sz), how long requests took to be serviced including their time in the queue (await), how long requests took to be serviced by the device after they left the queue (svctm), and lastly the utilization percentage which I already mentioned in the overview statistics section. In the above example random requests take longer for the disk to service as expected because of the seek time. However, the queue itself ends up being shorter which I am unable to explain. Utilization, in more detail, is the service time in ms * total IO operations / 1000 ms. This gives the percentage of how busy the single disk was during the given time slice. I believe for a given utilization level a higher number of operations is probably indicative of a sequential pattern.

I have run various variations on the above. They include a mixture of reads and writes for both random and sequential data as well as sequential and random writes. For the writes I got similar results as far as the ratios were concerned and queue and services time were higher.

In the end it seems average request size is the key to show if the disk usage patterns are random or not since this is post merging. Taking this into the context of the layers above this might not mirror what an application is doing. This is because a read or write operations coming from user space might operate on a fragmented file in which case the generic block layer will break it up and it appears as random disk activity.

Conclusion:

As far as I am concerned this is only a start in interpreting IO statistics. I think these tests need to be repeated, perhaps with different tools to generate the disk IO, as my interpretations might just be totally off. Also, a pretty big limitation of what I did is that my work was all on a single disk and these numbers might have different results under various RAID configurations. I feel the inability to measure the cache hit ratio of reads on a block device is a significant shortcoming that I would love to see addressed since from a system administrators perspective the solution to certain IO problems might be to throw more memory at the problem.

Lastly, I want to make a point about these sort of low level statistics in general. Everything needs to monitored from the an application perspective as well. These statistics can be misleading and are most useful when they can be correlated with the data that actually matters to users of the applications, for example, response time from the user perspective. These also need to be monitored over time because you want to be able to see changes for capacity planning as well as to give them context to past performance when problems arise.

Further Reading:
http://www.igvita.com/2009/06/23/measuring-optimizing-io-performance/
Understanding The Linux Kernel, Chapter 14
http://www.ufsdump.org/papers/io-tuning.pdf
http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/
http://www.kernel.org/doc/Documentation/iostats.txt

Compound Interest Calculation

Formula for calculating compound interest:

Where,

• P = principal amount (initial investment)
• r = annual nominal interest rate (as a decimal)
• n = number of times the interest is compounded per year
• t = number of years
• A = amount after time t

Example usage:

An amount of $1500.00 is deposited in a bank paying an annual interest rate of 4.3%, compounded quarterly. Find the balance after 6 years.

A. Using the formula above, with P = 1500, r = 4.3/100 = 0.043, n = 4, and t = 6:

So, the balance after 6 years is approximately $1,938.op.

rurlhttpcolonslashslashendotwikipediadotorgslashwikislashCompound_interest

LDAP Authentication In Linux

This howto will show you howto store your users in LDAP and authenticate some of the services against it. I will not show howto install particular packages, as it is distribution/system dependant. I will focus on “pure” configuration of all componenets needed to have LDAP authentication/storage of users. The howto assumes somehow, that you are migrating from a regular passwd/shadow authentication, but it is also suitable for people who do it from scratch.

Requirements

OpenLDAP
pam_ldap
nss_ldap
PADL migrationtools

Introducion

The thing we want to achieve is to have our users stored in LDAP, authenticated against LDAP ( direct or pam ) and have some tool to manage this in a human understandable way.

This way we can use all software, which has ldap support or fallback to PAM ldap module, which will act as a PAM->LDAP gateway.

Configuring OpenLDAP

OpenLDAP consists of slapd and slurpd daemon. This howto covers one LDAP server without a replication, so we will focus only on slapd. I also assume you installed and initialized your openldap installation (depends on system/disribution). If so, let’s go to configuration part.

On my system (Gentoo), openldap’s configuration is stored in /etc/openldap, we are interested in/etc/openldap/slapd.conf file. But first we have to generate a password for LDAP administrator, to put it into the config file:

# slappasswd -h {md5}

The config looks like this:

# vim /etc/openldap/slapd.conf

include         /etc/openldap/schema/core.schema

include         /etc/openldap/schema/cosine.schema

include         /etc/openldap/schema/inetorgperson.schema

include         /etc/openldap/schema/nis.schema

allow bind_v2

pidfile         /var/run/openldap/slapd.pid

argsfile        /var/run/openldap/slapd.args

modulepath      /usr/lib/openldap/openldap

access to attrs=userPassword

       by dn="uid=root,ou=People,dc=hackadmin,dc=com" write

       by dn="cn=Manager,dc=hackadmin,dc=com" write

       by anonymous auth

       by self write

       by * none

access to dn.base="" by * read

access to *

        by dn="cn=Manager,dc=hackadmin,dc=com" write

        by * read

database        bdb

suffix          "dc=hackadmin,dc=com"

rootdn          "cn=Manager,dc=hackadmin,dc=com"
rootpw          {MD5}Tk1sMytv5ipjr+Vhcf03JQ==

directory       /var/lib/openldap-data

index   objectClass     eq

Remember to change suffix and paths to your needs.

These are basic options with some basic ACLs needed to change passwrods by user. If you want more functionality, please read the manual about openLDAP. Now when we have a proper config for slapd, we can start the daemon :

# /etc/init.d/ldap start

# chkconfig ldap on

Now we can test if openldap is running and working properly. We do not have any data yet in the directory, but we can try to bind as cn=Manager,dc=domain,dc=com. When you are asked for password, you should use the one you generated (of course the plain text version of it :

# ldapsearch -D “cn=Manager,dc=hackadmin,dc=com” -W

Migrate/Add data to the directory

Now when we have a running LDAP server, we have to fill it with data, either create or migrate entries. I will show you howto migrate existing entries from regular /etc/passwd, /etc/shadow , /etc/groups

The first step is to configure mogrationtools to your needs. The configuration file on gentoo is located in/usr/share/migrationtools/migrate_common.ph.

Generally you need to change only these:

$DEFAULT_BASE = "dc=hackadmin,dc=com";

$EXTENDED_SCHEMA = 1;

Now you are ready to migrate the data (actually it works even without the export command):

export ETC_SHADOW=/etc/shadow

# ./migrate_base.pl > /tmp/base.ldif
# ./migrate_group.pl /etc/group /tmp/group.ldif
# ./migrate_hosts.pl /etc/hosts /tmp/hosts.ldif
# ./migrate_passwd.pl /etc/passwd /tmp/passwd.ldif

Now we have the data in the format understood by LDAP server. Please open one the files with text editor to get used to the syntax. After that we can add the data from ldifs.

# ldapadd -D “cn=Manager,dc=domain,dc=com” -W -f /tmp/base.ldif

# ldapadd -D “cn=Manager,dc=domain,dc=com” -W -f /tmp/group.ldif

# ldapadd -D “cn=Manager,dc=domain,dc=com” -W -f /tmp/passwd.ldif

# ldapadd -D “cn=Manager,dc=domain,dc=com” -W -f /tmp/hosts.ldif

You can try searching for some data:

# ldapsearch uid=foouser

Client configuration

By client I mean the machine, which connects to LDAP server to get users and authorize. It can be also the machine, the ldap server runs on. In both cases we have to edit three files : /etc/ldap.conf, /etc/nsswitch.conf and /etc/pam.d/system-auth

Let’s start woth ldap.conf, the ldap’s client:

BASE    dc=hackadmin, dc=com

scope sub

suffix          "dc=hackadmin,dc=com"

## when you want to change user's password by root

rootbinddn cn=Manager,dc=hackadmin,dc=com

## there are needed when your ldap dies

timelimit 5

bind_timelimit 5

uri ldap://ldap.hackadmin.com/

pam_password exop

ldap_version 3

pam_filter objectclass=posixAccount

pam_login_attribute uid

pam_member_attribute memberuid

nss_base_passwd ou=Computers,dc=cognifide,dc=pl

nss_base_passwd ou=People,dc=cognifide,dc=pl

nss_base_shadow ou=People,dc=cognifide,dc=pl

nss_base_group  ou=Group,dc=cognifide,dc=pl

nss_base_hosts  ou=Hosts,dc=cognifide,dc=pl

Now it is time for nsswitch.conf and pam

Add these to nsswitch.conf:

passwd: files ldap

shadow: files ldap

group:  files ldap

And change the system-auth (or hatever you have like login, sshd etc) to :

auth       required     pam_env.so

auth       sufficient   pam_unix.so likeauth nullok

auth       sufficient   pam_ldap.so use_first_pass

auth       required     pam_deny.so

account    sufficient   pam_unix.so

account    sufficient   pam_ldap.so

account    required     pam_ldap.so

password   required     pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3

password   sufficient   pam_unix.so nullok md5 shadow use_authtok

password   sufficient   pam_ldap.so use_first_pass

password   required     pam_deny.so

session    required     pam_limits.so

session    required     pam_unix.so

session    optional     pam_ldap.so

Time to test it. The best tool for it is a good old getent. Pick a user from your system and issue:

# getent passwd | grep foouser

You should get the result twice, if so the nss_ldap works fine. The pam part can be tested by deleting a user from the /etc/passwd and trying to log in through ssh.

Apache mod_auth_ldap

To have LDAP authorization in apache, you have to load mod_auth_ldap module

LoadModule mm_auth_ldap_module modules/mod_auth_ldap.so

Now it is enought to make .htaccess like that:

AuthName "Restricted"
AuthType Basic
AuthLDAPURL ldap://ldap.hackadmin.com:389/ou=People,dc=hackadmin,dc=com?uid
AuthLDAPBindDN "cn=Manager,dc=hackadmin,dc=com"
AuthLDAPBindPassword "your_secret_secret_password_to_ldap_admin"
require valid-user

Note that this method can be also used for webdav subversion authorization

Administration tools for ldap

There are few tool I recommend using to administrate OpenLDAP server

phpldapadmin - web based tool
ldapvi - vim browsing
PADL migrationtools - migrationtools
IDEALX sambaldap tools - samba ldap tools

rurlwwwdothackadmin.comdot2010slash03slash05slashldap-authentication-in-linuxslash

Creating home directories on Linux hosts with pam_mkhomedir

I have been converting a number of hosts to LDAP authentication. I’m currently creating user home directories on each server, which has a number of pros and cons. One of the cons is that a newly provisioned user won’t have a home directory, all will be assigned “/” as their home directory when they login. This is less than ideal, since most users will need a place to modify files and customize their environment. To simplify my life, I have been playing around with autodir and pam_mkhomedir. Both solutions provide an automated way to create user home directories, and are pretty easy to set up.

To configure pam_mkhomedir, you can add add the following line to the session management section of /etc/pam.d/system-auth:

session     optional      pam_mkhomedir.so

After the module is enabled, users should see a “Creating directory” line when they login to a server for the first time:

$ ssh test@foo
test@foo’s password:
Creating directory ‘/home/test’.

In addition to creating the home directory specified in the passwd file (or in the homeDirectory attribute if you are using LDAP), the mkhomedir module will also populate the user’s home directory with the files in /etc/skel:

$ ls -la /etc/skel

total 40
drwxr-xr-x.   4 root root  4096 2009-07-07 13:56 .
drwxr-xr-x. 113 root root 12288 2009-07-16 11:08 ..
-rw-r--r--.   1 root root    18 2009-04-08 06:46 .bash_logout
-rw-r--r--.   1 root root   176 2009-04-08 06:46 .bash_profile
-rw-r--r--.   1 root root   124 2009-04-08 06:46 .bashrc
drwxr-xr-x.   2 root root  4096 2009-03-17 20:54 .gnome2
drwxr-xr-x.   4 root root  4096 2009-07-07 13:44 .mozilla
-rw-r--r--.   1 root root   658 2009-03-02 12:18 .zshrc

Adding to the base set of files that are placed in each user’s home directory is as simple as copying one or more files into /etc/skel, or modifying the existing files. I will touch on the autodir module in a follow up post.

Using cut – Shellscript string manipulation

This post is designed to be a refresher, reference or quick intro into how to manipulate strings with the cut command in bash.Some times its useful to take the output of a command and reformat it. I sometimes do this for aesthetic purposes or tor format for use as input into another command.

Cut has options to cut by bytes (-b), characters (-c) or fields (-f). I normally cut by character or field but byte can come in handy some times.

The options to cut by are below.

N N’th byte, character or field, counted from 1

N- from N’th byte, character or field, to end of line

N-M from N’th to M’th (included) byte, character or field

-M from first to M’th (included) byte, character or field

The options pretty much explain themselves but I have included some simple examples below:

Cutting by characters (command on top, output below)

echo "123456789" | cut -c -5

12345

echo "123456789" | cut -c 5-

56789

echo "123456789" | cut -c 3-7

34567

echo "123456789" | cut -c 5

5

Sometimes output from a command is delimited so a cut by characters will not work. Take the example below:

echo -e "1\t2\t3\t4\t5" |cut -c 5-7

3 4

To echo a tab you have to use the -e switch to enable echo to process back slashed characters. If the desired output is 3\t4 then this would work great if the strings were always 1 character but if anywhere before field 3 a character was added the output would be completely changed as followed:

echo -e "1a\t2b\t3c\t4d\t5e" | cut -c 5-7

b 3

This is resolved by cutting by fields.

Cutting by fields

The syntax to cut by fields is the same as characters or bytes. The two examples below display different output but are both displaying the same fields (Fields 3 Through to the end of line.)

echo -e "1\t2\t3\t4\t5" | cut -f 3-

3 4 5

echo -e "1a\t2a\t3a\t4a\t5a" | cut -f 3-

3a 4a 5a

The default delimiter is a tab, if the output is delimited another way a custom delimiter can be specified with the -d option. It can be just about any printable character, just make sure that the character is escaped (back slashed) if needed. In the example below I cut the string up using the pipe as the delimiter.

echo "1|2|3|4|5" | cut -f 3- -d \|

3|4|5

One great feature of cut is that the delimiter that was used for input can be changed by the output of cut. In the example below I change the format of the string from a dash delimited output and change it to a comma.

echo -e "1a-2a-3a-4a-5a" | cut -f 3- -d – --output-delimiter=,

3a,4a,5a

Formatting with Cut Example

Sometimes certain Linux applications such as uptime do not have options to format the output. Cut can be used to pull out the information that is desired.

Normal up-time Command:

owen@the-linux-blog:~$ uptime

19:18:40 up 1 day, 22:15, 4 users, load average: 0.45, 0.10, 0.03

Time with up-time displayed:

owen@the-linux-blog:~$ uptime |cut -d , -f 1,2 | cut -c 2-

19:19:36 up 1 day, 22:22

For the above example I pipe the output of uptime to cut and tell it I want to split it with a comma , delimiter. I then choose fields 1 and 2. The output from that cut is piped into another cut that removes the spaces in front of the output.

Load averages extracted from uptime:

owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-

load average: 0.42, 0.10, 0.03

This is about the same as the previous example except the fields changed. Instead of fields 1 and 2 I told it to display fields 4 through the end. The output from that is piped to another cut which removes the three spaces that were after the comma in "4 users, " by starting at the 3rd character.

The great thing about cutting by fields is that no matter if the field length changes the data stays the same. Take the example below. I now have 17 users logged in which would have broke the output if I had used -c (since there is an extra character due to a double digit number of users being logged in.)

owen@the-linux-blog:~$ uptime

19:25:11 up 1 day, 22:28, 17 users, load average: 0.00, 0.06, 0.04

owen@the-linux-blog:~$ uptime |cut -d , -f 4- | cut -c 3-

load average: 0.00, 0.06, 0.04

That just about covers everything for the cut command. Now you know about it you can use cut to chop up all types of strings. It is one of the many great tools available for string manipulation in bash. If you can remember what cut does it will make your shell scripting easier, you don’t need to memorize the syntax because all of the information on how to use cut is available here, in the man pages and all over the web.

Extracting sub string in Bash

Extracting sub string at Bash is very easy, let say you have a phone number 012-4567890 and you just wanna extract 4567890 out, you can do as below.


num="016-4567890";
echo ${num:5:7}
567890

O/P is 567890

5 is the starting point and 7 is the string length for sub string.