Tuesday, July 30, 2013

Linux Swap Space & Swappiness

What is a SWAP Space?
Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the pre-configured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.
Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.
However, swapping does have a downside. Compared to memory, disks are very slow. Memory speeds can be measured in nanoseconds, while disks are measured in milliseconds, so accessing the disk can be tens of thousands times slower than accessing physical memory. The more swapping that occurs, the slower your system will be. Sometimes excessive swapping or thrashing occurs where a page is swapped out and then very soon swapped in and then swapped out again and so on. In such situations the system is struggling to find free memory and keep applications running at the same time. In this case only adding more RAM will help.
Linux has two forms of swap space: the swap partition and the swap file. The swap partition is an independent section of the hard disk used solely for swapping; no other files can reside there. The swap file is a special file in the filesystem that resides amongst your system and data files.

How big should my swap space be?

Many people follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. This rule is nonsense. On a modern system, that's a LOT of swap, most people prefer that their systems never swap. You don't want your system to ever run out of RAM+swap, but you usually would rather have enough RAM in the system so it doesn't need to swap.

Red hat recommends setting as follows for RHEL 5:

The reality is the amount of swap space a system needs is not really a function of the amount of RAM it has but rather the memory workload that is running on that system. A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. In this case the system will simply lock the anonymous and system V shared memory into RAM and use the remaining RAM for caching file system data so when memory is exhausted the kernel only reclaims pagecache memory.

Considering that: 

1) At installation time when configuring the swap space there is no easy way to predetermine the memory a workload will require

2) The more RAM a system has the less swap space it typically needs, a better swap space

Systems with 4GB of ram or less require a minimum of 2GB of swap space
Systems with 4GB to 16GB of ram require a minimum of 4GB of swap space
Systems with 16GB to 64GB of ram require a minimum of 8GB of swap space
Systems with 64GB to 256GB of ram require a minimum of 16GB of swap space


Note: Swap space will just keep operation running for a while on heavy duty servers by swapping process.It is possible to run a Linux system without a swap space, and the system will run well if you have a large amount of memory -- but if you run out of physical memory then the system will crash, as it has nothing else it can do, so it is advisable to have a swap space, especially since disk space is relatively cheap.
The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.
The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle.

My RHEL6 OS is showing below swappiness values.

cat /proc/sys/vm/swappiness
60


"Swappiness" is a property for the Linux kernel that changes the balance between swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value means the kernel will try to avoid swapping as much as possible where a higher value instead will make the kernel aggressively try to use swap space. The default value is 60, and for most desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may improve interactivity.
In short:

vm.swappiness = 0 - it will swap only to avoid an out of memory condition
vm.swappiness = 60 - default value
vm.swappiness = 100 - it will swap aggressively
###################################################################
To temporarily set the swappiness in Linux, write the desired value (e.g. 10) to /proc/sys/vm/swappiness using the following command, running as root user:
echo 10 > /proc/sys/vm/swappiness
###################################################################
Permanent changes are made in /etc/sysctl.conf via the following configuration line
vm.swappiness = 10
###################################################################
Thanks,
Nishith N.Vyas

Tuesday, June 11, 2013

Squid Access Log Customization (Date & Time Stamp)

Subject: Adding date & time in squid access log file

By default, squid don't allow you to view "date & time stamp" of currently accessed websites & google searches. But, you may edit "/etc/squid/squid.conf" file to allow the same by modifying below lines.

I. Open your squid.conf and search for the line that starts with access_log or cache_access_log and looks like:


access_log /var/log/squid/access.log squid


II. Note the last keyword (in this case squid, but it could be common, combined, squidmime)

III. Then look for the line that starts with logformat. In my case, it has shown below.

logformat squid  %ts.%03tu %6tr %>a %Ss/%03Hs %


By default, it is commented. Just need to replace above shown line with below.

logformat squid %tl %6tr %>a %Ss/%03Hs %

Note: You can take a look at squid.conf.default that comes with squid and it's well documented to see other available options for customizing your logs.

Finally, restart/reload squid service & execute below command to verify the configuration of "Date & Time" stamp. Sample example is as given below.

tail -f /var/log/squid/access.log

11/Jun/2013:13:53:47 +0530     22 10.101.19.147 TCP_MEM_HIT/200 2677 GET http://i8.dainikbhaskar.com/thumbnail/69x60/web2images/www.divyabhaskar.co.in/2013/06/10/5548_joke-7.jpg - NONE/- image/jpeg

11/Jun/2013:13:53:49 +0530    569 10.101.19.147 TCP_MISS/200 4881 GET http://i8.dainikbhaskar.com/thumbnail/120x104/web2images/www.divyabhaskar.co.in/2013/06/10/4459_yadav8.jpg - DIRECT/80.150.193.186 image/jpeg




Regards,
Nishith N.Vyas



Friday, May 31, 2013

Run VLC Player as "root" user in Fedora/RHEL/CentOS

Subject: Run "VLC Player" as a "root" user in Fedora/RHEL/CentOS 5x/6x

This hand's on assumes that you have installed VLC media player on your respective Linux platform & having "rpmforge & rpmfusion" repository in "/etc/yum.repos.d/" directory.

Perform below hand's now.

1) Install "hexedit" utility in linux using yum. The command is,

yum install hexedit

2) After successful installation, open "/usr/bin/vlc" using hexeditor as given below.

hexedit /usr/bin/vlc

3) Now, Press "TAB" key

4) Now, Press "Ctrl + S" key to search ASCII string.

5) In the given search box, type "geteuid" & replace it with "getppid"

6) Now, Press "Ctrl + X" to save the file.

Finally, your VLC player is ready to run as "root" user. Verify it & comments are always welcomed,if this post is useful to you.

This hand's on has been tested on "CentOS & RHEL 6.0" & will work with Fedora editions too.


Nishith N.Vyas



Friday, March 15, 2013

Block Ultrasurf in RedHat Linux Squid

Subject:
Block "Ultrasurf" in RedHat Linux Squid. Successfully tested on "2.6.STABLE6-4.el5" version.

Hello All,

Prior to do anything in squid configuration, it is necessary to understand about "what ultrasurf is" & why it is popular.!!!!

Ultrasurf is a product of Ultrareach Internet Corporation. Originally created to help internet users in China find security and freedom online, Ultrasurf has now become one of the world's most popular anti-censorship, pro-privacy software, with millions of people using it to bypass internet censorship and protect their online privacy.

Visit https://ultrasurf.us/ for more information.

Coming directly onto "practical", kindly add below lines in "/etc/squid/squid.conf" file.


#####Create New ACL#####
acl UltraSurf port 9666
acl ipacl url_regex http://[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*
acl numeric_IPs url_regex ^[0-9]+.[0-9]+.[0-9]+.[0-9]+

#####Deny all ACL#####
http_access deny UltraSurf
http_access deny ipacl
http_access deny numeric_IPs
http_access deny all   --- This is the default line.This is for your reference only to add all lines before this only.

Note: Above configuration is useful if "ultrasurf" is installed in LAN Computers. But, still anyone can download "ultrasurf" from internet. So, it is better to block the "ultrasurf name & domain" using below ACL

##################################
acl blockregexurl url_regex -i ultrasurf
acl block-site dstdomain .ultrasurf.us

http_access deny blockregexurl
http_access deny block-site
##################################

Note: Now, Squid will block all Internet requests, having "ip address" in URL. So, you need to add another ACL to allow certain IP Addresses, In use in your LAN segment for "Hosting or R&D purpose", if any.


##################################
acl bypass_ultrasurf_ip dstdomain 172.24.23.1 192.168.1.2 59.167.154.20

http_access allow bypass_ultrasurf_ip  --- Put this line above the "deny ultrasurf" lines only.

##################################

SUM UP:
This configuration has been tested to block "ultrasurf & all it's alternatives" as given below.

* tor
* GTunnel
* FreeGate
* Tunnelier


Thanks,
Nishith N.Vyas








IBM JFS (Journaling File System) Introduction

IBM JFS is a 128 bit file system available in AIX Operating System. Currently available versions are JFS v1 & v2.

JFS means "keep track of file system changes", before committing them to the main file system. So, in the event of "sudden power failure or system crash", such file systems are quicker to bring back online & less likely to become corrupted. 

JFS & i-nodes
JFS allows you to specify the number of disk i-nodes created within a file system in case more or fewer than the default number of disk i-nodes is desired.
NBPI = The number of disk i-nodes at file system creation is specified in a value called as the number of bytes per i-node or NBPI
For example, an NBPI value of 1024 causes a disk i-node to be created for every 1024 bytes of file system disk space. 

Another way to look at this is that a small NBPI value (512 for instance) results in a large number of i-nodes, while a large NBPI value (such as 16,384) results in a small number of i-nodes.


For JFS file systems, one i-node is created for every NBPI bytes of allocation group space allocated to the file system. The total number of i-nodes in a file system limits the total number of files and the total size of the file system. An allocation group can be partially allocated, though the full number of i-nodes per allocation group is still allocated. NBPI is inversely proportional to the total number of i-nodes in a file system.
The JFS restricts all file systems to 16M (224) i-nodes
The set of allowable NBPI values vary according to the allocation group size (agsize). 
The default is 8 MB. The allowable NBPI values are 512, 1024, 2048, 4096, 8192, and 16,384 with an agsize of 8 MB. A larger agsize can be used. 
The allowable values for agsize are 8, 16, 32, and 64. The range of allowable NBPI values scales up as agsize increases. If the agsize is doubled to 16 MB, the range of NBPI values also double: 1024, 2048, 4096, 8193, 16384, and 32768.
JFS2 & i-nodes
JFS2 allocates i-nodes as needed.
If there is room in the file system for additional i-nodes, they are automatically allocated. Therefore, the number of i-nodes available is limited by the size of the file system itself.

Nishith N.Vyas

Thursday, March 14, 2013

Understanding Load Average in LINUX/UNIX/AIX


The term “load average” is used in many "Linux/UNIX/AIX" Operating Systems as a major utility.
Everybody knows that the numbers the term “load average” refers to, usually three numbers, somehow represent the load on the system’s CPU. In this post I’ll try making this three numbers clearer and understandable.

The easiest way to see the “load average” of your system is by "uptime" command.

It also appears in "top" command in Linux & "topas" command in UNIX/AIX. 

In all three cases the load average refers to a group of three numbers. For example, in the following output of "uptime",

10:41:47 up 5 days, 48 min, 1 user, load average: 0.82, 0.71, 0.66
The last three numbers are the “load average”. Each number represent the systems load as a moving average over 1, 5 and 15 minutes respectively. Now, the important thing is to understand what is being averaged, the load metric.

The metric that represent the load at a given point in time is how many process are queued for running at each given time (including the process that is currently being ran). Generally speaking, on a single core machine, this can be looked at as CPU utilization percentage when multiplied by 100. 

For example if I had a load-average of 0.50 in the last minute, this means that over the last minute half of the time the CPU was idle as it had no running process. 

On the other hand if I had load average of 2.50 it means that over the last minute an average of 1.5 process were waiting to their turn to run. So, the CPU was overloaded by 150%.

On a multi-core (Like Core 2 Duo, IBM Power Servers, HP Itanium Servers) systems things are a bit different, but in order to avoid unnecessary complications one can usually divide the load-average by the number of cores and treat the result as the load average of single core machine.

For example, let’s say the load average of a two-core machine was 3.00 2.00 0.50

This means that over the last minute we had an average of three runnable process (3.00), this means that one process, in average, was queued as there are two core in the machine that can run to process at a time. So the machine was overloaded had a load of 150% its capability. 

Over the last 5 minutes the load average of 2.00 means that we roughly had 2 process running each time, so the machine was fully utilized but wasn’t overloaded by work. 

Over last 15 minutes the load-average of 0.50 means that we could handle 4 time that load without overloading the CPU, we only had (0.50/2)*100=25% CPU utilization in that 15 minutes.
Hope now everybody is clear about "Load Average Term".

Nishith N.Vyas

Wednesday, April 11, 2012

Set User Password using single command line in Linux

To set "Linux User" password using single command line.

On your shell prompt, type below command.
echo -e "Hello\nHello" | passwd nishith

Here,
-e : effect
\n : New Line
Hello : Mentioned twice as "password & retype password" ;)
nishith : User Name

Change the password of "nishith" users on a bunch of servers i.e. 10-20 servers.

Let's say server ip address range is from "192.168.10.1 to 192.168.10.20"

On your shell prompt, type below command.

for ((i=1,i<=20;i++); do ssh 192.168.10.$i 'echo -e "Hello\nHello" | passwd nishith'; done;

Create one user & set it's initial password remotely.

ssh 'useradd ; echo -e "passwdofuser\npasswordofuser" | passwd newuser'

Example:

ssh root@192.168.10.10 'useradd nishith1; echo -e "Hello1\nHello1" | passwd nishith1'

It will ask "root" password of 192.168.10.10 & rest will be taken care by the command itself.


Enjoy Linux

Friday, March 30, 2012

Linux Boot Process Stages

Hello,

Herewith I am posting "Linux Boot Process" in such an easy way so that you can understand it properly.

1) First stage is obviously called "POST"
(POST = Power On Self Test, Means if you get "beep" sound, the computer hardware is working properly i.e. your electronic circuit)

2) BIOS

3) MBR = Master Boot Record. Having size of 512 Bytes. The Byte distribution inside MBR is,

446 Bytes = Primary Boot Loader
64 Bytes = Partition Table
2 Bytes = MBR Validation Check

4) Loading Linux Loader called GRUB (GRand Unified Boot Loader)
(Please note that /etc/grub/grub.conf is linked with /boot/grub/grub.conf"

5) Kernel Loads. The sub processes are,

* Mounts the "/root" file system
* PID of this process would be "1"
* "initrd" (Initial RAM Disk), used by kernel as a temporary "root" file system until kernel is boot properly & the real "root" file system is mounted.

"initrd" contains necessary drivers to load.

6) "init" process loads entries mentioned inside "/etc/inittab" & "/etc/rc.sysinit" files.

7) "runlevel" loads the respective files & service mentioned inside "/etc/rc.d/rc*.d/
* There are total 6 runlevel are available by default.
* If you set "runlevel 5" in "/etc/inittab", then it will load all drivers & services available inside
"/etc/rc.d/rc5.d/

"S" letter shows (Service to start during "startup")
"K" letter shows (Service to kill during "startup")

That's it.

So, in short Linux boot stages are,
POST, BIOS,MBR,Kernel,INIT & Runlevel

EOF

Thursday, March 15, 2012

Linux Server Monitoring Commands You Really Need To Know

Hello Friends,

Want to know what's really going on with your server? Then you need to know these essential commands. Once you've mastered them, you'll be well on your way to being an expert Linux system administrator.

Depending on the Linux distribution, you can run pull up much of the information that these shell commands can give you from a GUI program. SUSE Linux, for example, has an excellent, graphical configuration and management tool, YaST, and KDE's KDE System Guard is also excellent.

However, it's a Linux administrator truism that you should run a GUI on a server only when you absolutely must. That's because Linux GUI's take up system resources that could be better used elsewhere. So, while using a GUI program is fine for basic server health check ups, if you want to know what's really happening, turn off the GUI and use these tools from the Linux command shell.

This also means that you should only start a GUI on a server when it's required; don’t leave it running. For optimum performance, a Linux server should run at "runlevel 3", which fully supports networking and multiple users but doesn't start the GUI when the machine boots. If you really need a graphical desktop, you can always get one by running "startx" from a shell prompt.

Note: If your server starts by booting into a graphical desktop, you need to change this. To do so, head to a terminal window, "su" to the root user, and use your favourite editor on /etc/inittab.

Once there, find the initdefault line and change it from id:5:initdefault: toid:3:initdefault:

If there is no inittab file, create it, and add the id:3 line. Save and exit. The next time you boot into your server it will boot into "runlevel 3". If you don't want to reboot after this change, you can also set your server's run level immediately with the command: init 3

Once your server is running at init 3, you can start using the following shell programs to see what's happening inside your server.

iostat

The iostat command shows in detail what your storage subsystem is up to. You usually use iostat to monitor how well your storage sub-systems are working in general and to spot slow input/output problems before your clients notice that the server is running slowly. Trust me, you want to spot these problems before your users do!

meminfo and free

Meminfo gives you a detailed list of what's going on in memory. Typically you access meminfo's data by using another program such as cat or grep. For example,

cat /proc/meminfo

gives you the details of what's going on in your server’s memory at any given moment.

For a quick “just the facts” look at memory, you can use the free command. In short,free gives you the overview; meminfo gives you the details.

mpstat

The mpstat command reports on the activities of each of the available CPUs on a multi-processor server. These days, thanks to multi-core processors, that’s almost all servers.mpstat also reports on the average activities of all your server's CPUs. It enables you to display overall CPU statistics per system or per processor. This overview can alert you to possible application problems before they get to the point of annoying users.

netstat

Netstat, like ps, is a Linux tool that administrators use every day. It displays a lot of network related information, such as socket usage, routing, interface, protocol, network statistics, and more. Some of the most commonly used options are:

-a Show all socket information

-r Show routing information

-i Show network interface statistics

-s Show network protocol statistics

nmon

Nmon, short for Nigel's Monitor, is a popular open-source tool to monitor Linux systems performance. Nmon watches the performance information for several subsystems, such as processor utilization, memory utilization, run queue information, disk I/O statistics, network I/O statistics, paging activity, and process metrics. You can then view nmon's real-time system measurements via its curses “graphical” interface.

sjvn_LinuxServerMonitoring_nmon.png

To run nmon, you start the tool from the shell. Once up, you select the subsystems to monitor by typing in its one-key commands. For example, to get CPU, memory, and disk statistics, you type c, m, and d. You can also use nmon with the -f flag to save performance statistics to a CSV file for later analysis.

For day to day server monitoring I find nmon to be the single most useful program in my Linux system management tool-kit.

pmap

The pmap command reports the amount of memory that your server's processes are using. You can use this tool to determine which processes on the server are being allocated memory and whether any of these processes are being piggy with RAM.

ps and pstree

The ps and pstree commands are two of the Linux administrator’s best friends. They both provide a list of all currently running processes. Ps tells you how much memory and processor time the server’s programs are using. Pstree shows less information, but highlights which processes are the children of other processes. Armed with this information, you can spot out–of-control processes and kill them off with Linux's “take no prisoners” kill command.

sar

The sar program is a Swiss-army knife of a system monitoring tool. The sar command is actually made up of three programs: sar, which displays the data, and sa1 and sa2, which collect and store it. Once installed, sar creates a detailed overview of CPU utilization, memory paging, network I/O and transfer statistics, process creation activity, and storage device activity. The big difference between sar and nmon is that the former is better at long-term system monitoring, while I find nmon to be better at giving me a quick read on my server's status.

strace

strace is often thought of a programmer's debugging tool, but it's more than that. It intercepts and records the system calls that are called by a process. This makes it a useful diagnostic, instructional, and debugging tool. For example, you can use strace to find out which configuration file a program is actually using when it starts up.

Strace does have one flaw though. When it's checking out a specific process, that process' performance will fall through the floor. Thus, I only use strace when I already have a darned good reason to think that that program is causing trouble.

tcpdump

Tcpdump is a simple, robust network monitoring utility. Its basic protocol analyzing capability enables you to get a rough view of what is happening on your network. To really dig into what's going on with your network however, you'll want to use Wireshark (see below).

top

The top command shows what's going on with your active processes. By default, it displays the most CPU-intensive tasks running on the server and updates the list every five seconds. You can sort the processes by PID (Process ID); age, newest first; time, by cumulative time; and resident memory usage and total time it's been using the CPU since startup. I find this a fast and easy way to see if any process is starting to go out of control and about to get into trouble.

uptime

Use uptime to see how long the server has been running and how many users are logged on. It also gives you an overview of the average server load. The optimal value of the load is 1 or less, which means that each process has immediate access to the CPU and there are no CPU cycles lost.

vmstat

For the most part, you use vmstat to monitor what's going on with virtual memory. Linux constantly uses virtual memory to get the best possible storage performance.

If your applications are taking up too much memory you get excessive page-outs — programs moving from RAM to your system's swap space, which is on the hard drive. Your server can reach a point where it's spending more time managing memory paging than running your applications, a condition called thrashing. When your computer is thrashing, its performance falls through the floor. Vmstat, which can display either average data or actual samples, can help you spot memory pig programs and processes before they bring your server to a crawl.

Wireshark

Wireshark, formerly known as Ethereal (and still often referred to that way), is tcpdump's big brother, though it is more sophisticated and with far more advanced protocol analyzing and reporting. Wireshark has both a GUI interface and a shell interface. If you do any serious network administration, you must use ethereal.

Note: If you're using Wireshark/ethereal, I highly recommend Chris Sander's Practical Packet Analysis, a great book on how to get the most out of this useful program.


Have A Good Day Ahead...