Well today I was needing to see some packages that arrived on apache. In order to see I have to use tcpdump, but you will only some short information if you are using basic.
In order to see HTTP header you have to use something like this:
tcpdump -n -vvvs 1024 -l -A host matrafox.info
If you want to see more information you have to increase the size, something like this
tcpdump -n -vvvs 1500-l -A host matrafox.info
Well all know that this packet is not included into centos distro . To install and use php-mcrypt you have to install and use epel repository.
So here we go
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm
yum update
yum install php-mcrypt*
So .. my problem is that I see some error into dmesg log about ata2 , however I don’t know what hard drive is mapped to that ata. So , how can we find this ?
[16380082.599294] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[16380082.599340] ata2.00: failed command: FLUSH CACHE EXT
[16380082.599369] ata2.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
[16380082.599370] res 40/00:00:80:2b:0a/00:00:70:00:00/40 Emask 0x4 (timeout)
[16380082.599435] ata2.00: status: { DRDY }
[16380082.599461] ata2: hard resetting link
[16380082.919117] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[16380082.921248] ata2.00: configured for UDMA/133
[16380082.921279] ata2: EH complete
Well to find out we have to look into /sys file-system :
ls -l /sys/block/sd{a,b}
lrwxrwxrwx 1 root root 0 Jan 17 08:01 /sys/block/sda -> ../devices/pci0000:00/0000:00:1f.2/host0/target0:0:0/0:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Jan 17 08:01 /sys/block/sdb -> ../devices/pci0000:00/0000:00:1f.2/host1/target1:0:0/1:0:0:0/block/sdb
To get that ID, Instead of looking into the target structure, we’re looking into the scsi_host structure. It has a directory for the host and in it a file unique_id. That’s the ID appended to ata.
for host in 0 1 ; do echo -n host${host}:; cat /sys/devices/pci0000:00/0000:00:1f.2/host${host}/scsi_host/host${host}/unique_id; done
host0:1
host1:2
So the answer is on ata2 I have sdb mounted.
Also I found this php code on internet
One of the problem that is related to my latest post is the Wait IO problem. So if in top or other state command you see WA then is a problem with your IO.
But how to figure out what is the problem ?
Well first of all you can take a look on how your server is going with the waiting.
You can run vmstat to see what is happening.
vmstat 2
procs ———–memory———- —swap– —–io—- –system– —–cpu—–
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 252848 1252264 2449400 20381888 0 0 25 77 3 1 6 2 58 29 0
0 1 252848 1241816 2449408 20381956 0 0 34 148 3102 2892 10 3 84 3 0
0 0 252848 1257000 2449408 20382036 0 0 0 468 1336 1201 3 1 33 58 0
3 0 252848 1243228 2449412 20382332 0 0 140 134 2278 1955 7 2 90 1 0
0 1 252848 1245444 2449420 20382460 0 0 68 304 2558 2255 7 2 40 42 0
1 1 252848 1231920 2449424 20382648 0 0 0 122 1708 1356 4 1 61 28 0
From here we are interested into the pre-last column the wa . From what we see there are all the time some values there, this mean the server is waiting the IO . Now we see that we have some problem.
But here you don’t see anything about what is causing this. You only check and see if you are having IO problem .
Please note, in this result, the sum of columns id (Idle) and wa (Wait IO) is almost 100, this is leading to some problem on configuration . In most of the cases is from ext4 journal.
How to find the cause ?
Well we can check with ps auxf command .
ps auxf |more
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2 0.0 0.0 0 0 ? S Nov10 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Nov10 1:26 \_ [migration/0]
root 4 0.0 0.0 0 0 ? S Nov10 1:04 \_ [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S Nov10 0:00 \_ [migration/0]
root 6 0.0 0.0 0 0 ? S Nov10 0:01 \_ [watchdog/0]
root 7 0.0 0.0 0 0 ? S Nov10 0:07 \_ [migration/1]
root 8 0.0 0.0 0 0 ? S Nov10 0:00 \_ [migration/1]
root 9 0.0 0.0 0 0 ? S Nov10 3:33 \_ [ksoftirqd/1]
root 10 0.0 0.0 0 0 ? S Nov10 0:01 \_ [watchdog/1]
root 11 0.0 0.0 0 0 ? S Nov10 0:03 \_ [migration/2]
root 12 0.0 0.0 0 0 ? S Nov10 0:00 \_ [migration/2]
root 13 0.0 0.0 0 0 ? S Nov10 6:11 \_ [ksoftirqd/2]
root 14 0.0 0.0 0 0 ? S Nov10 0:01 \_ [watchdog/2]
root 15 0.0 0.0 0 0 ? S Nov10 0:02 \_ [migration/3]
root 16 0.0 0.0 0 0 ? S Nov10 0:00 \_ [migration/3]
And here we are interested about STAT column .
The stat can have this values:
D Uninterruptible sleep (usually IO)
R Running or runnable (on run queue)
S Interruptible sleep (waiting for an event to complete)
T Stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z Defunct (“zombie”) process, terminated but not reaped by its parent.
Additional characters may be displayed:
< high-priority (not nice to other users)
N low-priority (nice to other users)
L has pages locked into memory (for real-time and custom IO)
s is a session leader
l is multi-threaded (using CLONE_THREAD, like NPTL pthreads do)
+ is in the foreground process group
So from here we deduct that if the stat have "D" then is a problem with the process.
To cut down process that is "eating" your CPU time, you can use this command which samples all process with D flag in every second:
while true; do date; ps auxf | awk ‘{if($8==”D”) print $0;}’; sleep 1; done
Tue Dec 3 13:22:06 CET 2013
Tue Dec 3 13:22:07 CET 2013
Tue Dec 3 13:22:08 CET 2013
Tue Dec 3 13:22:09 CET 2013
Tue Dec 3 13:22:10 CET 2013
root 500 0.0 0.0 0 0 ? D Nov10 2:14 \_ [jbd2/sda6-8]
nobody 14675 0.1 0.1 134932 39620 ? D 13:20 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
Tue Dec 3 13:22:11 CET 2013
Tue Dec 3 13:22:12 CET 2013
root 981 0.0 0.0 0 0 ? D Nov10 3:13 \_ [kjournald]
Tue Dec 3 13:22:13 CET 2013
Tue Dec 3 13:22:14 CET 2013
From the result you can see that [jbd2/sda6-8] [kjournald] is creating some problem.
Also you may use iotop or atop to see some stats.
You can check only those process with this command .
while true; do ps auxf | grep D | grep -E “(jbd2\/dm\.*|kdmflush)”; sleep 1; done
Ok, if the problem is related to the ext4 journal I think you should consider to remove that. If your server is on development you don’t need it, also if is in production then you may better use a raid for having your information in good condition.
Regards
Well yesterday I receive daily e-mail and saw that my raid is failing .
cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
2102464 blocks [2/2] [UU]
md1 : active raid1 sdb2[1] sda2[0]
264960 blocks [2/2] [UU]
md2 : active raid1 sdb3[2](F) sda3[0]
1462766336 blocks [2/1] [U_]
So that mean that sdb3 is marked as failed drive and U_ mean that raid is degraded.
Well from this point I remove the sdb1 and sdb2 from raid but before I mark them as failed
mdadm --manage /dev/md1 --fail /dev/sdb2
mdadm --manage /dev/md0 --fail /dev/sdb1
mdadm /dev/md0 -r /dev/sdb1
mdadm /dev/md1 -r /dev/sdb2
mdadm /dev/md2 -r /dev/sdb3
After replacement of hard drive I have to recreate the same partition on new sdb and add it to raid.
sfdisk -d /dev/sda | sfdisk /dev/sdb
mdadm /dev/md0 -a /dev/sdb1
mdadm /dev/md1 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb3
Now watch how your raid is recovering
watch cat /proc/mdstat
Every 2.0s: cat /proc/mdstat Tue Sep 4 09:52:52 2012
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sda1[0]
2102464 blocks [2/2] [UU]
md1 : active raid1 sdb2[1] sda2[0]
264960 blocks [2/2] [UU]
md2 : active raid1 sdb3[2] sda3[0]
1462766336 blocks [2/1] [U_]
[===>.................] recovery = 16.0% (234480192/1462766336) finish=412.8min speed=49580K/sec
unused devices:
However the speed my by low so how to increase that speed ?
cat /proc/sys/dev/raid/speed_limit_max
200000
cat /proc/sys/dev/raid/speed_limit_min
1000
Now I increase the min limit to 50000
echo 50000 >/proc/sys/dev/raid/speed_limit_min
Now if you watch cat /proc/mdstat again you will see that your speed is improved and your time get low.
Powered by WordPress