Tuesday, July 30, 2013

Linux Swap Space & Swappiness

What is a SWAP Space?
Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the pre-configured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.
Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.
However, swapping does have a downside. Compared to memory, disks are very slow. Memory speeds can be measured in nanoseconds, while disks are measured in milliseconds, so accessing the disk can be tens of thousands times slower than accessing physical memory. The more swapping that occurs, the slower your system will be. Sometimes excessive swapping or thrashing occurs where a page is swapped out and then very soon swapped in and then swapped out again and so on. In such situations the system is struggling to find free memory and keep applications running at the same time. In this case only adding more RAM will help.
Linux has two forms of swap space: the swap partition and the swap file. The swap partition is an independent section of the hard disk used solely for swapping; no other files can reside there. The swap file is a special file in the filesystem that resides amongst your system and data files.

How big should my swap space be?

Many people follow an old rule of thumb that your swap partition should be twice the size of your main system RAM. This rule is nonsense. On a modern system, that's a LOT of swap, most people prefer that their systems never swap. You don't want your system to ever run out of RAM+swap, but you usually would rather have enough RAM in the system so it doesn't need to swap.

Red hat recommends setting as follows for RHEL 5:

The reality is the amount of swap space a system needs is not really a function of the amount of RAM it has but rather the memory workload that is running on that system. A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. In this case the system will simply lock the anonymous and system V shared memory into RAM and use the remaining RAM for caching file system data so when memory is exhausted the kernel only reclaims pagecache memory.

Considering that

1) At installation time when configuring the swap space there is no easy way to predetermine the memory a workload will require

2) The more RAM a system has the less swap space it typically needs, a better swap space

Systems with 4GB of ram or less require a minimum of 2GB of swap space
Systems with 4GB to 16GB of ram require a minimum of 4GB of swap space
Systems with 16GB to 64GB of ram require a minimum of 8GB of swap space
Systems with 64GB to 256GB of ram require a minimum of 16GB of swap space


Note: Swap space will just keep operation running for a while on heavy duty servers by swapping process.It is possible to run a Linux system without a swap space, and the system will run well if you have a large amount of memory -- but if you run out of physical memory then the system will crash, as it has nothing else it can do, so it is advisable to have a swap space, especially since disk space is relatively cheap.
The key question is how much? Older versions of Unix-type operating systems (such as Sun OS and Ultrix) demanded a swap space of two to three times that of physical memory. Modern implementations (such as Linux) don't require that much, but they can use it if you configure it. A rule of thumb is as follows: 1) for a desktop system, use a swap space of double system memory, as it will allow you to run a large number of applications (many of which may will be idle and easily swapped), making more RAM available for the active applications; 2) for a server, have a smaller amount of swap available (say half of physical memory) so that you have some flexibility for swapping when needed, but monitor the amount of swap space used and upgrade your RAM if necessary; 3) for older desktop machines (with say only 128MB), use as much swap space as you can spare, even up to 1GB.
The Linux 2.6 kernel added a new kernel parameter called swappiness to let administrators tweak the way Linux swaps. It is a number from 0 to 100. In essence, higher values lead to more pages being swapped, and lower values lead to more applications being kept in memory, even if they are idle.

My RHEL6 OS is showing below swappiness values.

cat /proc/sys/vm/swappiness
60


"Swappiness" is a property for the Linux kernel that changes the balance between swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value means the kernel will try to avoid swapping as much as possible where a higher value instead will make the kernel aggressively try to use swap space. The default value is 60, and for most desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may improve interactivity.
In short:

vm.swappiness = 0 - it will swap only to avoid an out of memory condition
vm.swappiness = 60 - default value
vm.swappiness = 100 - it will swap aggressively
###################################################################
To temporarily set the swappiness in Linux, write the desired value (e.g. 10) to /proc/sys/vm/swappiness using the following command, running as root user:
echo 10 > /proc/sys/vm/swappiness
###################################################################
Permanent changes are made in /etc/sysctl.conf via the following configuration line
vm.swappiness = 10
###################################################################
Thanks,
Nishith N.Vyas

Tuesday, June 11, 2013

Squid Access Log Customization (Date & Time Stamp)

Subject: Adding date & time in squid access log file

By default, squid don't allow you to view "date & time stamp" of currently accessed websites & google searches. But, you may edit "/etc/squid/squid.conf" file to allow the same by modifying below lines.

I. Open your squid.conf and search for the line that starts with access_log or cache_access_log and looks like:


access_log /var/log/squid/access.log squid


II. Note the last keyword (in this case squid, but it could be common, combined, squidmime)

III. Then look for the line that starts with logformat. In my case, it has shown below.

logformat squid  %ts.%03tu %6tr %>a %Ss/%03Hs %


By default, it is commented. Just need to replace above shown line with below.

logformat squid %tl %6tr %>a %Ss/%03Hs %

Note: You can take a look at squid.conf.default that comes with squid and it's well documented to see other available options for customizing your logs.

Finally, restart/reload squid service & execute below command to verify the configuration of "Date & Time" stamp. Sample example is as given below.

tail -f /var/log/squid/access.log

11/Jun/2013:13:53:47 +0530     22 10.101.19.147 TCP_MEM_HIT/200 2677 GET http://i8.dainikbhaskar.com/thumbnail/69x60/web2images/www.divyabhaskar.co.in/2013/06/10/5548_joke-7.jpg - NONE/- image/jpeg

11/Jun/2013:13:53:49 +0530    569 10.101.19.147 TCP_MISS/200 4881 GET http://i8.dainikbhaskar.com/thumbnail/120x104/web2images/www.divyabhaskar.co.in/2013/06/10/4459_yadav8.jpg - DIRECT/80.150.193.186 image/jpeg




Regards,
Nishith N.Vyas



Friday, May 31, 2013

Run VLC Player as "root" user in Fedora/RHEL/CentOS

Subject: Run "VLC Player" as a "root" user in Fedora/RHEL/CentOS 5x/6x

This hand's on assumes that you have installed VLC media player on your respective Linux platform & having "rpmforge & rpmfusion" repository in "/etc/yum.repos.d/" directory.

Perform below hand's now.

1) Install "hexedit" utility in linux using yum. The command is,

yum install hexedit

2) After successful installation, open "/usr/bin/vlc" using hexeditor as given below.

hexedit /usr/bin/vlc

3) Now, Press "TAB" key

4) Now, Press "Ctrl + S" key to search ASCII string.

5) In the given search box, type "geteuid" & replace it with "getppid"

6) Now, Press "Ctrl + X" to save the file.

Finally, your VLC player is ready to run as "root" user. Verify it & comments are always welcomed,if this post is useful to you.

This hand's on has been tested on "CentOS & RHEL 6.0" & will work with Fedora editions too.


Nishith N.Vyas



Friday, March 15, 2013

Block Ultrasurf in RedHat Linux Squid

Subject:
Block "Ultrasurf" in RedHat Linux Squid. Successfully tested on "2.6.STABLE6-4.el5" version.

Hello All,

Prior to do anything in squid configuration, it is necessary to understand about "what ultrasurf is" & why it is popular.!!!!

Ultrasurf is a product of Ultrareach Internet Corporation. Originally created to help internet users in China find security and freedom online, Ultrasurf has now become one of the world's most popular anti-censorship, pro-privacy software, with millions of people using it to bypass internet censorship and protect their online privacy.

Visit https://ultrasurf.us/ for more information.

Coming directly onto "practical", kindly add below lines in "/etc/squid/squid.conf" file.


#####Create New ACL#####
acl UltraSurf port 9666
acl ipacl url_regex http://[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*
acl numeric_IPs url_regex ^[0-9]+.[0-9]+.[0-9]+.[0-9]+

#####Deny all ACL#####
http_access deny UltraSurf
http_access deny ipacl
http_access deny numeric_IPs
http_access deny all   --- This is the default line.This is for your reference only to add all lines before this only.

Note: Above configuration is useful if "ultrasurf" is installed in LAN Computers. But, still anyone can download "ultrasurf" from internet. So, it is better to block the "ultrasurf name & domain" using below ACL

##################################
acl blockregexurl url_regex -i ultrasurf
acl block-site dstdomain .ultrasurf.us

http_access deny blockregexurl
http_access deny block-site
##################################

Note: Now, Squid will block all Internet requests, having "ip address" in URL. So, you need to add another ACL to allow certain IP Addresses, In use in your LAN segment for "Hosting or R&D purpose", if any.


##################################
acl bypass_ultrasurf_ip dstdomain 172.24.23.1 192.168.1.2 59.167.154.20

http_access allow bypass_ultrasurf_ip  --- Put this line above the "deny ultrasurf" lines only.

##################################

SUM UP:
This configuration has been tested to block "ultrasurf & all it's alternatives" as given below.

* tor
* GTunnel
* FreeGate
* Tunnelier


Thanks,
Nishith N.Vyas








IBM JFS (Journaling File System) Introduction

IBM JFS is a 128 bit file system available in AIX Operating System. Currently available versions are JFS v1 & v2.

JFS means "keep track of file system changes", before committing them to the main file system. So, in the event of "sudden power failure or system crash", such file systems are quicker to bring back online & less likely to become corrupted. 

JFS & i-nodes
JFS allows you to specify the number of disk i-nodes created within a file system in case more or fewer than the default number of disk i-nodes is desired.
NBPI = The number of disk i-nodes at file system creation is specified in a value called as the number of bytes per i-node or NBPI
For example, an NBPI value of 1024 causes a disk i-node to be created for every 1024 bytes of file system disk space. 

Another way to look at this is that a small NBPI value (512 for instance) results in a large number of i-nodes, while a large NBPI value (such as 16,384) results in a small number of i-nodes.


For JFS file systems, one i-node is created for every NBPI bytes of allocation group space allocated to the file system. The total number of i-nodes in a file system limits the total number of files and the total size of the file system. An allocation group can be partially allocated, though the full number of i-nodes per allocation group is still allocated. NBPI is inversely proportional to the total number of i-nodes in a file system.
The JFS restricts all file systems to 16M (224) i-nodes
The set of allowable NBPI values vary according to the allocation group size (agsize). 
The default is 8 MB. The allowable NBPI values are 512, 1024, 2048, 4096, 8192, and 16,384 with an agsize of 8 MB. A larger agsize can be used. 
The allowable values for agsize are 8, 16, 32, and 64. The range of allowable NBPI values scales up as agsize increases. If the agsize is doubled to 16 MB, the range of NBPI values also double: 1024, 2048, 4096, 8193, 16384, and 32768.
JFS2 & i-nodes
JFS2 allocates i-nodes as needed.
If there is room in the file system for additional i-nodes, they are automatically allocated. Therefore, the number of i-nodes available is limited by the size of the file system itself.

Nishith N.Vyas

Thursday, March 14, 2013

Understanding Load Average in LINUX/UNIX/AIX


The term “load average” is used in many "Linux/UNIX/AIX" Operating Systems as a major utility.
Everybody knows that the numbers the term “load average” refers to, usually three numbers, somehow represent the load on the system’s CPU. In this post I’ll try making this three numbers clearer and understandable.

The easiest way to see the “load average” of your system is by "uptime" command.

It also appears in "top" command in Linux & "topas" command in UNIX/AIX. 

In all three cases the load average refers to a group of three numbers. For example, in the following output of "uptime",

10:41:47 up 5 days, 48 min, 1 user, load average: 0.82, 0.71, 0.66
The last three numbers are the “load average”. Each number represent the systems load as a moving average over 1, 5 and 15 minutes respectively. Now, the important thing is to understand what is being averaged, the load metric.

The metric that represent the load at a given point in time is how many process are queued for running at each given time (including the process that is currently being ran). Generally speaking, on a single core machine, this can be looked at as CPU utilization percentage when multiplied by 100. 

For example if I had a load-average of 0.50 in the last minute, this means that over the last minute half of the time the CPU was idle as it had no running process

On the other hand if I had load average of 2.50 it means that over the last minute an average of 1.5 process were waiting to their turn to run. So, the CPU was overloaded by 150%.

On a multi-core (Like Core 2 Duo, IBM Power Servers, HP Itanium Servers) systems things are a bit different, but in order to avoid unnecessary complications one can usually divide the load-average by the number of cores and treat the result as the load average of single core machine.

For example, let’s say the load average of a two-core machine was 3.00 2.00 0.50

This means that over the last minute we had an average of three runnable process (3.00), this means that one process, in average, was queued as there are two core in the machine that can run to process at a time. So the machine was overloaded had a load of 150% its capability. 

Over the last 5 minutes the load average of 2.00 means that we roughly had 2 process running each time, so the machine was fully utilized but wasn’t overloaded by work

Over last 15 minutes the load-average of 0.50 means that we could handle 4 time that load without overloading the CPU, we only had (0.50/2)*100=25% CPU utilization in that 15 minutes.
Hope now everybody is clear about "Load Average Term".

Nishith N.Vyas