How to free Linux Kernel page cache and/or inode and dentry caches

Kernels 2.6.16 and newer provide a mechanism to have the kernel drop the page cache and/or inode and dentry caches on command, which can help free up a lot of memory.

Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:
echo 1 > /proc/sys/vm/drop_caches

To free dentries and inodes:
echo 2 > /proc/sys/vm/drop_caches

To free pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation, and dirty objects are not free-able, the user should run "sync" first in order to make sure all cached objects are freed.

Linux bond or team multiple network interfaces into single

Finally today I had implemented NIC bounding (bind both NIC so that it works as a single device).My idea is to improve performance by pumping out more data from both NIC without using any other method.

Linux allows binding multiple network interfaces into a single channel/NIC using special kernel module called bonding. "The Linux bonding driver provides a method for aggregating multiple network interfaces into a single logical "bonded" interface. The behavior of the bonded interfaces depends upon the mode; generally speaking, modes provide either hot standby or load balancing services. Additionally, link integrity monitoring may be performed."

Note:-What is bonding?

Bonding allows you to aggregate multiple ports into a single group, effectively combining the bandwidth into a single connection. Bonding also allows you to create multi-gigabit pipes to transport traffic through the highest traffic areas of your network. For example, you can aggregate three megabits ports (1 mb each) into a three-megabits trunk port. That is equivalent with having one interface with three megabits speed.

Setting up bounding is easy with RHEL v5.0.and above

Step #1:

Create a bond0 configuration file

Red Hat Linux stores network configuration in /etc/sysconfig/network-scripts/ directory. First, you need to create bond0 config file:

Code:

# vi /etc/sysconfig/network-scripts/ifcfg-bond0

Append following lines to it:

DEVICE=bond0

IPADDR=192.168.1.59

NETWORK=192.168.1.0

NETMASK=255.255.255.0

USERCTL=no

BOOTPROTO=none

ONBOOT=yes

Note:Replace above IP address with your actual IP address. Save file and exit to shell prompt

Step #2:

Modify eth0 and eth1 config files:

Open both configuration using vi text editor and make sure file read as follows for eth0 interface

# vi /etc/sysconfig/network-scripts/ifcfg-eth0

Modify/append directive as follows:

DEVICE=eth0

USERCTL=no

ONBOOT=yes

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

Open eth1 configuration file using vi text editor:

# vi /etc/sysconfig/network-scripts/ifcfg-eth1

Make sure file read as follows for eth1 interface:

DEVICE=eth1

USERCTL=no

ONBOOT=yes

MASTER=bond0

SLAVE=yes

BOOTPROTO=none

Save file and exit to shell prompt

Step # 3:

Load bond driver/module

Make sure bonding module is loaded when the channel-bonding interface (bond0) is brought up. You need to modify kernel modules configuration file:

# vi /etc/modprobe.conf

Append following two lines:

alias bond0 bonding

options bond0 mode=balance-alb miimon=100

Note:-Save file and exit to shell prompt. You can learn more about all bounding options at the end of this document

Step # 4:

Test configuration

First, load the bonding module:

# modprobe bonding

Restart networking service in order to bring up bond0 interface:

# service network restart

Verify everything is working:

# less /proc/net/bonding/bond0

Output:

Bonding Mode: load balancing (round-robin)

MII Status: up

MII Polling Interval (ms): 0

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: eth0

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:0c:29:XX:XX:X1

Slave Interface: eth1

MII Status: up

Link Failure Count: 0

Permanent HW addr: 00:0c:29:XX:XX:X2

List all interfaces:

# ifconfig

Output:

bond0 Link encap:Ethernet HWaddr 00:0C:29:XX:XX:XX

inet addr:192.168.1.59 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1

RX packets:2804 errors:0 dropped:0 overruns:0 frame:0

TX packets:1879 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:250825 (244.9 KiB) TX bytes:244683 (238.9 KiB)

eth0 Link encap:Ethernet HWaddr 00:0C:29:XX:XX:XX

inet addr:192.168.1.59 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:2809 errors:0 dropped:0 overruns:0 frame:0

TX packets:1390 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:251161 (245.2 KiB) TX bytes:180289 (176.0 KiB)

Interrupt:11 Base address:0x1400

eth1 Link encap:Ethernet HWaddr 00:0C:29:XX:XX:XX

inet addr:192.168.1.59 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::20c:29ff:fec6:be59/64 Scope:Link

UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1

RX packets:4 errors:0 dropped:0 overruns:0 frame:0

TX packets:502 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:258 (258.0 b) TX bytes:66516 (64.9 KiB)

Interrupt:10 Base address:0x1480

Note:-If the administration tools of your distribution do not support master/slave

notation in configuration of network interfaces, you will need to configure

the bonding device with the following commands manually:

# /sbin/ifconfig bond0 192.168.1.59 up

# /sbin/ifenslave bond0 eth0

# /sbin/ifenslave bond0 eth1

Que:-What are the other MODE options in modprobe .conf file

Ans:-You can set up your bond interface according to your needs. Changing one parameters (mode=X) you can have the following bonding types:

mode=0 (balance-rr)

Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

mode=1 (active-backup)

Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode.

mode=2 (balance-xor)

XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance.

mode=3 (broadcast)

Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

mode=4 (802.3ad)

IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.

mode=5 (balance-tlb)

Adaptive transmit load balancing: channel bonding that does not require any special switch support. The outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

mode=6 (balance-alb)

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load balancing is achieved by ARP negotiation. The bonding driver intercepts the ARP Replies sent by the local system on their way out and overwrites the source hardware address with the unique hardware address of one of the slaves in the bond such that different peers use different hardware addresses for the server.

PS

1) Displaying top CPU_consuming processes:

ps aux | head -1; ps aux | sort -rn | head -10

2) Displaying top 10 memory-consuming processes:

ps aux | head -1; ps aux | sort -rn | head

3) Displaying process in order of being penalized:

ps -eakl | head -1; ps -eakl | sort -rn

4) Displaying process in order of priority:

ps -eakl | sort -n | head

5) Displaying process in order of nice value

ps -eakl | sort -n

6) Displaying the process in order of time

ps vx | head -1;ps vx | grep -v PID | sort -rn | head -10

7) Displaying the process in order of real memory use

ps vx | head -1; ps vx | grep -v PID | sort -rn | head -10

8) Displaying the process in order of I/O

ps vx | head -1; ps vx | grep -v PID | sort -rn | head -10

9) Displaying WLM classes

ps -a -o pid, user, class, pcpu, pmem, args

10) Determinimg process ID of wait processes:

ps vg | head -1; ps vg | grep -w wait

11) Wait process bound to CPU

ps -mo THREAD -p

How to List perl modules installed on my system?

List installed perl module
To display the list enter the following command:
$ instmodsh
Output:

Available commands are:
l - List all installed modulList installed perl module
To display the list enter the following command:
$ instmodsh
Output:

Available commands are:
l - List all installed modules
m - Select a module
q - Quit the program
cmd?

At cmd? prompt type l to list all installed modules:
cmd? les
m - Select a module
q - Quit the program
cmd?

At cmd? prompt type l to list all installed modules:
cmd? l

Sed Grouping and BackReference - PART-2

Example :

echo "[asd] [qwe] [zxc]"
[asd] [qwe] [zxc]

echo "[asd] [qwe] [zxc]" | sed -e "s/\[\(\(.\)*\)\]/\<\1\>/g"


echo "[asd] [qwe] [zxc]" | sed -e "s/\[\([^[]*\)\]/\<\1\>/g"


part in red color makes the difference. :P

Sed Grouping and BackReference - PART-1

Grouping can be used in sed like normal regular expression. A group is opened with “\(” and closed with “\)”.Grouping can be used in combination with back-referencing.

Back-reference is the re-use of a part of a Regular Expression selected by grouping. Back-references in sed can be used in both a Regular Expression and in the replacement part of the substitute command.

Example 1: Get only the first path in each line

$ sed 's/\(\/[^:]*\).*/\1/g' path.txt
/usr/kbos/bin
/usr/local/sbin
/opt/omni/lbin

In the above example, \(\/[^:]*\) matches the path available before first : comes. \1 replaces the first matched group.

Example 2: Multigrouping

In the file path.txt change the order of field in the last line of the file.

$ sed '$s@\([^:]*\):\([^:]*\):\([^:]*\)@\3:\2:\1@g' path.txt
/usr/kbos/bin:/usr/local/bin:/usr/jbin:/usr/bin:/usr/sas/bin
/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/omni/bin:
/root/bin:/opt/omni/sbin:/opt/omni/lbin

In the above command $ specifies substitution to happen only for the last line.Output shows that the order of the path values in the last line has been reversed.

Example 3: Get the list of usernames in /etc/passwd file

This sed example displays only the first field from the /etc/passwd file.

$sed 's/\([^:]*\).*/\1/' /etc/passwd
root
bin
daemon
adm
lp
sync
shutdown

Example 4: Parenthesize first character of each word

This sed example prints the first character of every word in paranthesis.

$ echo "Welcome To The Geek Stuff" | sed 's/\(\b[A-Z]\)/\(\1\)/g'
(W)elcome (T)o (T)he (G)eek (S)tuff

Example 5: Commify the simple number.

Let us create file called numbers which has list of numbers. The below sed command example is used to commify the numbers till thousands.

$ cat  numbers
1234
12121
3434
123

$sed 's/\(^\|[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g' numbers
1,234
12,121
3,434
123

Basic I/O Monitoring on Linux

The technique we discuss here is basic, it gives a good overview and is very easy to use. So let get focused… We will use iostat utility. In case you need you know where to find more about it — right, man pages.

So we will use the following form of the command:

iostat -x [-d]

• -x option displays extended statistics. You definitely want it.
• -d is optional. It removes CPU utilization to avoid cluttering the output. If you leave it out, you will get the following couple lines in addition:

  avg-cpu: %user %nice %sys %iowait %idle

  6.79 0.00 3.79 16.97 72.46

• is the number of seconds iostat waits between each report. Without a specified interval, iostat displays statistics since the system was up then exits, which is not useful in our case. Specifying the number of seconds causes iostat to print periodic reports where IO statistics are averaged for the time period since previous report. I.e., specifying 5 makes iostat dump 5 seconds of average IO characteristics, every 5 seconds until it’s stopped.

If you have many devices and you want to watch for only some of them, you can also specify device names on command line:

iostat -x -d sda 5

Now let’s get to the most interesting part — what those cryptic extended statistics are. (For readability, I formatted the report above so that the last two lines are in fact a continuation of the first two.)

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s

sda 0.00 12.57 10.18 9.78 134.13 178.84 67.07

wkB/s avgrq-sz avgqu-sz await svctm %util

89.42 15.68 0.28 14.16 8.88 17.72

• r/s and w/s— respectively, the number of read and write requests issued by processes to the OS for a device.
• rsec/s and wsec/s — sectors read/written (each sector 512 bytes).
• rkB/s and wkB/s — kilobytes read/written.
• avgrq-sz — average sectors per request (for both reads and writes). Do the math — (rsec + wsec) / (r + w) = (134.13+178.84)/(10.18+9.78)=15.6798597
  If you want it in kilobytes, divide by 2.
  If you want it separate for reads and writes — do you own math using rkB/s and wkB/s.
• avgqu-sz — average queue length for this device.
• await — average response time (ms) of IO requests to a device. The name is a bit confusing as this is the total response time including wait time in the requests queue (let call it qutim), and service time that device was working servicing the requests (see next column — svctim).

  So the formula is await = qutim + svctim.

• svctim — average time (ms) a device was servicing requests. This is a component of total response time of IO requests.
• %util — this is a pretty confusing value. The man page defines it as, Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. A bit difficult to digest. Perhaps it’s better to think of it as percentage of time the device was servicing requests as opposed to being idle. To understand it better here is the formula:

  utilization = ( (read requests + write requests) * service time in ms / 1000 ms ) * 100%
  or
  %util = ( r + w ) * svctim /10 = ( 10.18 + 9.78 ) * 8.88 = 17.72448

Traditionally, it’s common to assume that the closer to 100% utilization a device is, the more saturated it is. This might be true when the system device corresponds to a single physical disk. However, with devices representing a LUN of a modern storage box, the story might be completely different.

Rather than looking at device utilization, there is another way to estimate how loaded a device is. Look at the non-existent column I mentioned above — qutim — the average time a request is spending in the queue. If it’s insignificant, compare it to svctim — the IO device is not saturated. When it becomes comparable to svctim and goes above it, then requests are queued longer and a major part of response time is actually time spent waiting in the queue.

The figure in the await column should be as close to that in the svctim column as possible. If await goes much above svctim, watch out! The IO device is probably overloaded.

There is much to say about IO monitoring and interpreting results. Perhaps this is only the first of a series of posts about IO statistics. At Pythian we often come across different environments with specific characteristics and various requirements that our clients have. So stay tune — more to come.