转自:
http://wiki.isyi.com/ 感谢作者提供如此精彩的文章！
目录
复制内容到剪贴板
代码:
* 1 配置目标
* 2 环境说明
* 3 安装软件
o 3.1 安装 linux-kernel-headers
o 3.2 安装gcc
o 3.3 安装gcc-3.4
o 3.4 安装libc6-dev
o 3.5 安装libaio1 libaio-dev
o 3.6 安装bzip2
o 3.7 安装ntpdate
o 3.8 安装make
o 3.9 安装gawk
* 4 设置环境
o 4.1 增加打开文件数
o 4.2 设置gcc
* 5 系统优化
o 5.1 修改头文件
o 5.2 优化网络
o 5.3 调整磁盘属性
o 5.4 去除不必要的服务
o 5.5 重启服务器
* 6 配置缓存
o 6.1 安装squid
o 6.2 配置squid
o 6.3 squid运行脚本
* 7 运行观测
o 7.1 启动squid
o 7.2 top 信息
o 7.3 squid运行信息
o 7.4 并发连接
o 7.5 其他
o 7.6 截取日志工具
o 7.7 gawk分割日志效率测试
o 7.8 gcc-4.1编译squid的出错信息
o 7.9 没有初始化就运行squid的错误信息
o 7.10 squid初始化时，io等待非常高 配置目标
由于跑动态网站的服务器没有网通入口，需要做一个代理，让用户可以通过有网通入口的服务器来访问动态网站。开始只是考虑用Lighttpd来做单纯的代理。后来考虑到动态网站里面有大量的图片、javascript和htm静态文件。如果仅仅使用Lighttpd代理，没有缓存，用户量大了，后面动态服务器承受不了那么大的压力。
用Squid吧。不仅仅可以代理，还可以将静态资源文件缓存起来，一来可以让公网用户可以快速访问，二来缓存可以提高访问速度，三来可以减轻动态服务器的压力。一举三得，何乐而不为呢？
[编辑] 环境说明
准备配置成为缓存服务器的硬件配置：
品牌：HP DL360G4p
CPU： Xeon 3.0G
内存：2G
硬盘：73G x 2
由于配置了raid卡，安装系统的时候，不创建raid磁盘，无法找到硬盘驱动，无法安装下去。最终只好两块硬盘，各创建个raid0。
操作系统： Debian GNU/Linux 4.0
分区：
磁盘1
/ 1G
swap 物理内存x2 G
/usr 4G
/var 4G
/tmp 2G
/opt/cache1 20G
/opt/ 剩余空间
磁盘2
swap 物理内存x2 G
/opt/cache2 20G
/home 剩余空间
安装软件
刚安装好的系统，需要安装其他一些相关的工具包或者头文件。
* linux-kernel-headers
* gcc
* gcc-3.4
复制内容到剪贴板
代码:
* libc6-dev
* libaio1
* libaio-dev
* bzip2
* ntpdate
* make
* gawk 为什么要安装gcc-3.4呢？因为squid在gcc-4.1下编译会出现错误，编译不通过。在gcc-3.4下编译成功，运行很稳定。
安装 linux-kernel-headers
复制内容到剪贴板
代码:
# apt-get install linux-kernel-headers
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
linux-kernel-headers
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 1875kB of archives.
After unpacking 11.2MB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main linux-kernel-headers 2.6.18-7 [1875kB]
Fetched 1875kB in 0s (4674kB/s)
tar: ./md5sums: time stamp 2007-02-12 11:09:09 is 1453134.820269366 s in the future
tar: ./control: time stamp 2007-02-12 11:09:09 is 1453134.820065338 s in the future
tar: ./preinst: time stamp 2007-02-12 11:09:07 is 1453132.819973995 s in the future
tar: .: time stamp 2007-02-12 11:09:09 is 1453134.819924311 s in the future
Selecting previously deselected package linux-kernel-headers.
(Reading database ... 20027 files and directories currently installed.)
Unpacking linux-kernel-headers (from .../linux-kernel-headers_2.6.18-7_i386.deb) ...
Setting up linux-kernel-headers (2.6.18-7) ...安装gcc
复制内容到剪贴板
代码:
# apt-get install gcc
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
binutils cpp cpp-4.1 gcc-4.1 libssp0
Suggested packages:
binutils-doc cpp-doc gcc-4.1-locales make manpages-dev autoconf automake1.9 libtool flex bison gdb gcc-doc gcc-4.1-doc libc6-dev-amd64 lib64gcc1 lib64ssp0
Recommended packages:
libc6-dev libc-dev libmudflap0-dev
The following NEW packages will be installed:
binutils cpp cpp-4.1 gcc gcc-4.1 libssp0
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 5419kB of archives.
After unpacking 14.0MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 [url]http://202.205.109.38[/url] etch/main binutils 2.17-3 [2605kB]
Get:2 [url]http://202.205.109.38[/url] etch/main cpp-4.1 4.1.1-21 [2331kB]
Get:3 [url]http://202.205.109.38[/url] etch/main cpp 4:4.1.1-15 [11.6kB]
Get:4 [url]http://202.205.109.38[/url] etch/main libssp0 4.1.1-21 [4492B]
Get:5 [url]http://202.205.109.38[/url] etch/main gcc-4.1 4.1.1-21 [461kB]
Get:6 [url]http://202.205.109.38[/url] etch/main gcc 4:4.1.1-15 [5052B]
Fetched 5419kB in 1s (3129kB/s)
Selecting previously deselected package binutils.
(Reading database ... 19779 files and directories currently installed.)
Unpacking binutils (from .../binutils_2.17-3_i386.deb) ...
Selecting previously deselected package cpp-4.1.
Unpacking cpp-4.1 (from .../cpp-4.1_4.1.1-21_i386.deb) ...
Selecting previously deselected package cpp.
Unpacking cpp (from .../cpp_4%3a4.1.1-15_i386.deb) ...
Selecting previously deselected package libssp0.
Unpacking libssp0 (from .../libssp0_4.1.1-21_i386.deb) ...
Selecting previously deselected package gcc-4.1.
Unpacking gcc-4.1 (from .../gcc-4.1_4.1.1-21_i386.deb) ...
Selecting previously deselected package gcc.
Unpacking gcc (from .../gcc_4%3a4.1.1-15_i386.deb) ...
Setting up binutils (2.17-3) ...
Setting up cpp-4.1 (4.1.1-21) ...
Setting up cpp (4.1.1-15) ...
Setting up libssp0 (4.1.1-21) ...
Setting up gcc-4.1 (4.1.1-21) ...
Setting up gcc (4.1.1-15) ...安装gcc-3.4
复制内容到剪贴板
代码:
# apt-get install gcc-3.4
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
cpp-3.4 gcc-3.4-base
Suggested packages:
gcc-3.4-doc libc6-dev-amd64 lib64gcc1
The following NEW packages will be installed:
cpp-3.4 gcc-3.4 gcc-3.4-base
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 3309kB of archives.
After unpacking 8770kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 [url]http://202.205.109.38[/url] etch/main gcc-3.4-base 3.4.6-5 [164kB]
Get:2 [url]http://202.205.109.38[/url] etch/main cpp-3.4 3.4.6-5 [1735kB]
Get:3 [url]http://202.205.109.38[/url] etch/main gcc-3.4 3.4.6-5 [1409kB]
Fetched 3309kB in 0s (3387kB/s)
Selecting previously deselected package gcc-3.4-base.
(Reading database ... 22350 files and directories currently installed.)
Unpacking gcc-3.4-base (from .../gcc-3.4-base_3.4.6-5_i386.deb) ...
Selecting previously deselected package cpp-3.4.
Unpacking cpp-3.4 (from .../cpp-3.4_3.4.6-5_i386.deb) ...
Selecting previously deselected package gcc-3.4.
Unpacking gcc-3.4 (from .../gcc-3.4_3.4.6-5_i386.deb) ...
Setting up gcc-3.4-base (3.4.6-5) ...
Setting up cpp-3.4 (3.4.6-5) ...
Setting up gcc-3.4 (3.4.6-5) ...[编辑] 安装libc6-dev
复制内容到剪贴板
代码:
# apt-get install libc6-dev
Reading package lists... Done
Building dependency tree... Done
Suggested packages:
glibc-doc manpages-dev
The following NEW packages will be installed:
libc6-dev
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 2716kB of archives.
After unpacking 10.9MB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main libc6-dev 2.3.6.ds1-13 [2716kB]
Fetched 2716kB in 0s (4172kB/s)
tar: ./md5sums: time stamp 2007-02-21 23:14:09 is 2274104.304737386 s in the future
tar: ./control: time stamp 2007-02-21 23:14:09 is 2274104.304570096 s in the future
tar: .: time stamp 2007-02-21 23:14:09 is 2274104.304515509 s in the future
Selecting previously deselected package libc6-dev.
(Reading database ... 21752 files and directories currently installed.)
Unpacking libc6-dev (from .../libc6-dev_2.3.6.ds1-13_i386.deb) ...
Setting up libc6-dev (2.3.6.ds1-13) ...安装libaio1 libaio-dev
复制内容到剪贴板
代码:
# apt-get install libaio1 libaio-dev
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
libaio-dev libaio1
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 35.9kB of archives.
After unpacking 238kB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main libaio1 0.3.106-3 [5770B]
Get:2 [url]http://202.205.109.38[/url] etch/main libaio-dev 0.3.106-3 [30.1kB]
Fetched 35.9kB in 0s (442kB/s)
Selecting previously deselected package libaio1.
(Reading database ... 22223 files and directories currently installed.)
Unpacking libaio1 (from .../libaio1_0.3.106-3_i386.deb) ...
Selecting previously deselected package libaio-dev.
Unpacking libaio-dev (from .../libaio-dev_0.3.106-3_i386.deb) ...
Setting up libaio1 (0.3.106-3) ...
Setting up libaio-dev (0.3.106-3) ...安装bzip2
复制内容到剪贴板
代码:
# apt-get install bzip2
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
bzip2
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 269kB of archives.
After unpacking 446kB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main bzip2 1.0.3-6 [269kB]
Fetched 269kB in 0s (1377kB/s)
Selecting previously deselected package bzip2.
(Reading database ... 22259 files and directories currently installed.)
Unpacking bzip2 (from .../bzip2_1.0.3-6_i386.deb) ...
Setting up bzip2 (1.0.3-6) ...安装ntpdate
复制内容到剪贴板
代码:
# apt-get install ntpdate
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
ntpdate
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 57.7kB of archives.
After unpacking 213kB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main ntpdate 1:4.2.2.p4+dfsg-2 [57.7kB]
Fetched 57.7kB in 0s (382kB/s)
Selecting previously deselected package ntpdate.
(Reading database ... 22303 files and directories currently installed.)
Unpacking ntpdate (from .../ntpdate_1%3a4.2.2.p4+dfsg-2_i386.deb) ...
Setting up ntpdate (4.2.2.p4+dfsg-2) ...安装make
复制内容到剪贴板
代码:
# apt-get install make
Reading package lists... Done
Building dependency tree... Done
Suggested packages:
make-doc-non-dfsg
The following NEW packages will be installed:
make
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 382kB of archives.
After unpacking 1610kB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main make 3.81-2 [382kB]
Fetched 382kB in 0s (1652kB/s)
Selecting previously deselected package make.
(Reading database ... 22316 files and directories currently installed.)
Unpacking make (from .../archives/make_3.81-2_i386.deb) ...
Setting up make (3.81-2) ...安装gawk
复制内容到剪贴板
代码:
# apt-get install gawk
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
gawk
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 694kB of archives.
After unpacking 2011kB of additional disk space will be used.
Get:1 [url]http://202.205.109.38[/url] etch/main gawk 1:3.1.5.dfsg-4 [694kB]
Fetched 694kB in 0s (2744kB/s)
Selecting previously deselected package gawk.
(Reading database ... 22432 files and directories currently installed.)
Unpacking gawk (from .../gawk_1%3a3.1.5.dfsg-4_i386.deb) ...
Setting up gawk (3.1.5.dfsg-4) ...设置环境
增加打开文件数
系统默认的ulimit里，限制open file 只有1024个。要配置一台高并发的缓存服务器，是远远不够的。
* 设置前：
复制内容到剪贴板
代码:
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
* 设置open file为65535
# ulimit -HSn 65535* 设置后
复制内容到剪贴板
代码:
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited设置gcc
因为Debian 4.0 默认的gcc 是gcc-4.1，而编译squid需要使用gcc-3.4，需要我们手工指定编译器版本。另外增加一些优化参数，告诉gcc在编译的时候，使用优化编译。
复制内容到剪贴板
代码:
# export CC='gcc-3.4'
# export CFLAGS="-O6 -mtune=pentium -fomit-frame-pointer"
# export CXX=gcc-3.4
# export CXXFLAGS="-O6 -mtune=pentium -fomit-frame-pointer -felide-constructors -fno-exceptions -fno-rtti"系统优化
修改头文件
* 修改/usr/include/linux/limits.h
复制内容到剪贴板
代码:
# vi /usr/include/linux/limits.h
#define NR_OPEN 10240
/******/
#define OPEN_MAX 10240 /* # open files a process may have */
* 修改/usr/include/bits/types.h
# vi /usr/include/bits/types.h
#注意：此文件中并无此定义，添加到文件的末尾；
####
#undef __STD_TYPE
#define __FD_SETSIZE 32768 #添加此句.
#endif /* bits/types.h */优化网络
* 设置优化参数
复制内容到剪贴板
代码:
# vi /etc/sysctl.conf
net.ipv4.tcp_fin_timeout=1
net.ipv4.tcp_tw_recycle=1
# TCP memory
net.core.rmem_max=16777216
net.core.rmem_default=16777216
net.core.netdev_max_backlog=262144
net.core.somaxconn=262144
net.ipv4.tcp_syncookies=1
net.ipv4.tcp_max_orphans=262144
net.ipv4.tcp_max_syn_backlog=262144
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_syn_retries=2* 查看当前的设定值
复制内容到剪贴板
代码:
# cat /etc/sysctl.conf |grep -v "^#"|grep net|awk -F\= '{print $1}'|xargs sysctl
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_tw_recycle = 0
net.core.rmem_max = 131071
net.core.rmem_default = 109568
net.core.netdev_max_backlog = 1000
net.core.somaxconn = 128
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 32768
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syn_retries = 5 上面的值，是系统默认的。
* 应用优化值
在/etc/sysctl.conf设定的值，在重启系统后，可以自动生效。如果想不重启系统也能生效，在Debian下我找不到更好的办法来在不重启系统的情况下，要使得sysctl.conf的配置生效，我用的是下面的方法：
复制内容到剪贴板
代码:
# cat /etc/sysctl.conf |grep -v "^#"|grep net|xargs sysctl -w
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2* 查看优化后的设定值
复制内容到剪贴板
代码:
# cat /etc/sysctl.conf |grep -v "^#"|grep net|awk -F\= '{print $1}'|xargs sysctl
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
net.core.rmem_max = 16777216
net.core.rmem_default = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2调整磁盘属性
作为缓存目录的分区，不需要记录atime状态。因此我们在/etc/fstab 文件里设定 noatime 选项。
复制内容到剪贴板
代码:
# vi /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/dev/cciss/c0d0p1 / ext3 defaults,errors=remount-ro 0 1
/dev/cciss/c0d1p3 /home ext3 defaults 0 2
/dev/cciss/c0d0p8 /opt ext3 defaults 0 2
/dev/cciss/c0d0p7 /opt/cache1 ext3 rw,noatime 0 2
/dev/cciss/c0d1p2 /opt/cache2 ext3 rw,noatime 0 2
/dev/cciss/c0d0p6 /tmp ext3 defaults 0 2
/dev/cciss/c0d0p3 /usr ext3 defaults 0 2
/dev/cciss/c0d0p5 /var ext3 defaults 0 2
/dev/cciss/c0d0p2 none swap sw 0 0
/dev/cciss/c0d1p1 none swap sw 0 0
/dev/hda /media/cdrom0 udf,iso9660 user,noauto 0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0去除不必要的服务
复制内容到剪贴板
代码:
# cd /etc/rc2.d/
# mv S89atd K89atd
# mv S18portmap K18portmap
# mv S20exim4 K20exim4
# mv S20openbsd-inetd K20openbsd-inetd
# mv S21nfs-common K21nfs-common
# mv S20acpid K20acpid提示：
在进行这步之前，先确认当前系统默认运行级别。Debian默认是2，Redhat默认是3。
确认方法：
# cat /etc/inittab |grep initdefault
id:2:initdefault:
其中id:2，是表示默认运行级是2。
[编辑] 重启服务器
# reboot
配置缓存
安装squid
* 创建squid帐户
复制内容到剪贴板
代码:
# useradd -s /bin/false squid* 创建常用目录
复制内容到剪贴板
代码:
# cd /opt
# mkdir -p {distfiles,etc,modules,bin,sbin,var/log,var/run}
# mkdir -p /opt/var/log/squid
# mkdir -p /home/weblogs
# chown -R squid:squid /opt/var/log/squid /home/weblogs
# chown -R squid:squid /opt/cache*
# cd distfiles* 下载squid
# wget
http://www.squid-cache.org/Versi ... .6.STABLE13.tar.bz2
* 解压squid
# tar -jxf squid-2.6.STABLE13.tar.bz2
* 编译安装
复制内容到剪贴板
代码:
# cd squid-2.6.STABLE13
# ./configure --prefix=/opt/modules/squid2 \
--with-pthreads \
--enable-storeio=diskd,coss,ufs,aufs,null \
--enable-async-io=128 \
--enable-icmp \
--enable-removal-policies=heap,lru \
--enable-x-accelerator-vary \
--enable-useragent-log \
--enable-gnuregex \
--enable-epoll \
--enable-coss-aio-ops \
--enable-snmp \
--enable-carp \
--enable-referer-log \
--enable-htcp \
--enable-cache-digests \
--enable-default-err-language=Simplify_Chinese \
--enable-cahce-digests \
--enable-kill-parent-hack \
--enable-underscore \
--disable-wccp \
--disable-delay-pools \
--disable-arp-acl \
--disable-ident-lookups \
--disable-hostname-checks \
--with-coss-membuf-size=4194304 \
--with-maxfd=32768 \
--with-large-files
# make
# make install提示：
我测试过使用--with-coss-membuf-size的值列表：
复制内容到剪贴板
代码:
536870912 = 1048576 × 512
268435456 = 1048576 × 256
134217728 = 1048576 × 128
67108864 = 1048576 × 64
33554432 = 1048576 × 32
16777216 = 1048576 × 16
8388608 = 1048576 × 8
4194304 = 1048576 × 4在4M、8M的时候，是最稳定的。如果是512M 的时候，squid进程很快就占满了内存，并且运行一会就挂了。
[编辑] 配置squid
复制内容到剪贴板
代码:
# rm /opt/modules/squid2/etc/squid.conf
# vi /opt/modules/squid2/etc/squid.conf
#squid.conf for fcache
#LastUpdate:2007-06-13
visible_hostname c001.ccn.eol.cn
#每台服务器的hostid都要对应ID表中的值.
########################################################################################
#####################################################################################
#
cache_peer 202.205.109.51 parent 80 0 no-netdb-exchange no-query originserver name=pcache1
cache_peer 202.205.109.28 parent 80 0 no-netdb-exchange no-query originserver name=pcache2
cache_peer_domain pcache1 .gkcx.eol.cn
cache_peer_domain pcache2 ask.eol.cn
dead_peer_timeout 600 seconds
acl accelerateddom dstdomain .eol.cn .edu.cn
#
cache_mgr yangmm[AT]cernet.com
#
http_port 80 vhost vport defaultsite=www.eol.cn
icp_port 3130
htcp_port 4827
#
#
#
cache_mem 1536 MB
cache_replacement_policy heap GDSF
memory_replacement_policy heap LRU
maximum_object_size 8192 KB
minimum_object_size 0 KB
maximum_object_size_in_memory 7168 KB
redirect_rewrites_host_header off
#cache_dir diskd /opt/var/spool/squid 1024 32 256
cache_dir coss /opt/cache1/coss1 4096 max-size=1000000 block-size=512
cache_dir coss /opt/cache1/coss2 4096 max-size=1000000 block-size=512
cache_dir coss /opt/cache2/coss1 4096 max-size=1000000 block-size=512
cache_dir coss /opt/cache2/coss2 4096 max-size=1000000 block-size=512
#cache_dir diskd /opt/cache1 2048 32 256
#cache_dir diskd /opt/cache2 2048 32 256
##Apache-style squid logs
logformat combined %>a %ui %un [%tl +0800] "%rm %ru HTTP/%rv" %Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
access_log /home/weblogs/access.log combined
#cache_access_log none
query_icmp on
buffered_logs on
logfile_rotate 10
cache_log /opt/var/log/squid/cache.log
cache_swap_log /opt/var/log/squid/cache_swap_log
cache_store_log none
#cache_store_log /opt/var/log/squid/store.log
#
#emulate_httpd_log on
log_fqdn off
check_hostnames off
dns_timeout 2 minutes
#mix
#dns_nameservers 202.106.0.20
#####
hosts_file /etc/hosts
refresh_pattern . 5 40% 30
#
#parseConfigFile: line 62 unrecognized: 'reference_age 2 hours'fs
#reference_age 2 hours
#
negative_ttl 25 seconds
positive_dns_ttl 60 seconds
negative_dns_ttl 30 seconds
half_closed_clients off
client_lifetime 240 seconds
pconn_timeout 120 seconds
#ident_timeout 1 seconds
shutdown_lifetime 15 seconds
#
digest_rebuild_period 300 seconds
#
# ACCESS CONTROLS
#Recommended minimum configuration:
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443
acl Safe_ports port 80 81 6080 85 # http
acl purge method PURGE
acl CONNECT method CONNECT
acl TRACE method TRACE
#We recommend you to use the following two lines.
acl QUERY urlpath_regex cgi-bin \? asp php php3 cgi squid-internal-periodic
cache deny QUERY
hierarchy_stoplist cgi-bin asp php php3 cgi
acl pcachedom dstdomain pcache.ccn.eol.cn
#set this for php
always_direct allow QUERY
always_direct allow pcachedom
#set this for fcache
#never_direct allow ALL
#
# access arc
# Allow requests when they are to the accelerated machine AND to the
# right port with right protocol
#Recommended minimum configuration:
#
# Only allow cachemgr access from localhost
http_access allow manager localhost
http_access deny manager
# Only allow purge requests from localhost
#http_access allow purge localhost
http_access deny purge
# Deny requests to unknown ports
http_access deny !Safe_ports
# Deny CONNECT to other than SSL ports
http_access deny CONNECT !SSL_ports
http_access deny TRACE
#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
# Example rule allowing access from your local networks. Adapt
http_access allow localhost
###my domain only
acl acceleratedProtocol protocol HTTP
acl acceleratedPort port 6080
acl acceleratedPort port 81
acl acceleratedPort port 80
acl acceleratedPort port 85
http_access allow acceleratedProtocol acceleratedPort accelerateddom
#http_access allow acceleratedProtocol acceleratedPort accelerateddst
##
### OPTIONS WHICH AFFECT THE NEIGHBOR SELECTION ALGORITHM
#3s
icp_query_timeout 3000
#
# And finally deny all other access to this proxy
http_access deny all
http_reply_access allow all
reply_body_max_size 0 allow all
cache_effective_user squid
cache_effective_group squid
memory_pools on
#max_open_disk_fds 32768
max_open_disk_fds 65535
snmp_port 3401
acl mysnmps src 211.151.91.97/255.255.255.255
acl snmppublic snmp_community c001.ccn.eol.cn
snmp_access allow snmppublic localhost
snmp_access allow snmppublic mysnmps
snmp_access deny all
offline_mode off
# coredump_dir none
redirector_bypass off
high_response_time_warning 3000
high_page_fault_warning 3000
#ie_refresh on
sleep_after_fork 1000
pid_filename /opt/var/run/squid2-server.pid
#end of squid.confsquid运行脚本
复制内容到剪贴板
代码:
# chmod +x /opt/sbin/squid-server.sh
# ln -s /opt/sbin/squid-server.sh /etc/init.d/squid
# ln -s /etc/init.d/squid /etc/rc2.d/S22squid
# ln -s /opt/modules/squid2/sbin/squid /usr/sbin/运行观测
启动squid
# /opt/sbin/squid-server.sh start
提示：
1、启动的时候会初始化缓存目录，这个过程会比较长。可能是因为做了raid的原因，IO效率比较低。
2、这个脚本有监控功能，自动在后台监控squid进程，如果squid死掉，会自动将squid启动起来。
top 信息
复制内容到剪贴板
代码:
top - 11:20:43 up 1 day, 21:43, 5 users, load average: 0.38, 0.34, 0.34
Tasks: 58 total, 2 running, 55 sleeping, 1 stopped, 0 zombie
Cpu(s): 5.3%us, 1.8%sy, 0.0%ni, 91.5%id, 0.5%wa, 0.0%hi, 0.8%si, 0.0%st
Mem: 2076988k total, 2014076k used, 62912k free, 24368k buffers
Swap: 7807536k total, 72k used, 7807464k free, 1295172k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25521 squid 15 0 686m 606m 1568 R 14 29.9 148:58.04 squid
1 root 15 0 1944 648 556 S 0 0.0 0:03.18 init
2 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0squid运行信息
复制内容到剪贴板
代码:
# /opt/modules/squid2/bin/squidclient -h localhost -p 80 mgr:info|less
HTTP/1.0 200 OK
Server: squid/2.6.STABLE13
Date: Fri, 15 Jun 2007 03:18:17 GMT
Content-Type: text/plain
Expires: Fri, 15 Jun 2007 03:18:17 GMT
Last-Modified: Fri, 15 Jun 2007 03:18:17 GMT
X-Cache: MISS from c001.ccn.eol.cn
X-Cache-Lookup: MISS from c001.ccn.eol.cn:80
Via: 1.0 c001.ccn.eol.cn:80 (squid/2.6.STABLE13)
Connection: close
Squid Object Cache: Version 2.6.STABLE13
Start Time: Thu, 14 Jun 2007 01:36:17 GMT
Current Time: Fri, 15 Jun 2007 03:18:17 GMT
Connection information for squid:
Number of clients accessing cache: 59335
Number of HTTP requests received: 27048668
Number of ICP messages received: 1
Number of ICP messages sent: 1
Number of queued ICP replies: 0
Number of HTCP messages received: 0
Number of HTCP messages sent: 0
Request failure ratio: 0.00
Average HTTP requests per minute since start: 17541.2
Average ICP messages per minute since start: 0.0
Select loop called: 122456770 times, 0.756 ms avg
Cache information for squid:
Request Hit Ratios: 5min: 56.1%, 60min: 56.1%
Byte Hit Ratios: 5min: 61.3%, 60min: 62.2%
Request Memory Hit Ratios: 5min: 33.1%, 60min: 33.6%
Request Disk Hit Ratios: 5min: 0.7%, 60min: 0.8%
Storage Swap size: 384168 KB
Storage Mem size: 470352 KB
Mean Object Size: 16.87 KB
Requests given to unlinkd: 0
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.00286 0.00286
Cache Misses: 0.01035 0.01035
Cache Hits: 0.00000 0.00000
Near Hits: 0.00919 0.00919
Not-Modified Replies: 0.00000 0.00000
DNS Lookups: 0.00094 0.00094
ICP Queries: 0.00000 0.00000
Resource usage for squid:
UP Time: 92520.526 seconds
CPU Time: 8916.321 seconds
CPU Usage: 9.64%
CPU Usage, 5 minute avg: 16.25%
CPU Usage, 60 minute avg: 16.53%
Process Data Segment Size via sbrk(): 554060 KB
Maximum Resident Size: 0 KB
Page faults with physical i/o: 2
Memory usage for squid via mallinfo():
Total space in arena: 554060 KB
Ordinary blocks: 549387 KB 13166 blks
Small blocks: 0 KB 0 blks
Holding blocks: 106172 KB 15 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 4672 KB
Total in use: 655559 KB 99%
Total free: 4672 KB 1%
Total size: 660232 KB
Memory accounted for:
Total accounted: 597356 KB
memPoolAlloc calls: 2535774457
memPoolFree calls: 2535116743
File descriptor usage for squid:
Maximum number of file descriptors: 32768
Largest file desc currently in use: 5457
Number of file desc currently in use: 5101
Files queued for open: 0
Available number of file descriptors: 27667
Reserved number of file descriptors: 100
Store Disk files open: 0
IO loop method: epoll
Internal Data Structures:
24177 StoreEntries
21096 StoreEntries with MemObjects
21041 Hot Object Cache Items
22771 on-disk objects并发连接
# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}' ;
复制内容到剪贴板
代码:
LAST_ACK 26
SYN_RECV 112
CLOSE_WAIT 5
ESTABLISHED 5902
FIN_WAIT1 237
FIN_WAIT2 111
SYN_SENT 5
CLOSING 9
TIME_WAIT 695其他
截取日志工具
* 创建截取日志脚本
复制内容到剪贴板
代码:
# vi /opt/bin/cutsquidlog.sh
#!/bin/sh
#for cut squid log by hours
#add crontab 0 * * * * cutsquidlog.sh
# change by 16hot at 2007-06-13
#
DATESTR=`date -d "1hours ago" +%Y.%m.%d_%H`
HOST=`hostname`
LOG_DIR="/home/weblogs" ;
LOG=/home/weblogs/access.log
/usr/sbin/squid -k rotate
sleep 60
cd /home/weblogs/
mv $LOG.0 access_${DATESTR}H_${HOST}_log
##
## cat site log
## 2007-06-13
catSiteLog () {
logFile="${LOG_DIR}/access_${DATESTR}H_${HOST}_log" ;
siteUrl="${1}";
outFile="${LOG_DIR}/access_${DATESTR}H_${HOST}_${siteUrl}_log"
## delete old log file
if [ -f $2 ]; then
rm -rf $2;
fi
echo "Start cat ${1}'s log ...";
/usr/local/bin/gawk '$8~/'${siteUrl}'/' ${logFile} > ${outFile}
}
catSiteLog gkcx.eol.cn
catSiteLog ask.eol.cn
#end * 设置可以执行属性
# chmod +x /opt/bin/cutsquidlog.sh
* 在crontab里设置每个小时执行一次
复制内容到剪贴板
代码:
# crontab -e
0 * * * * /opt/bin/cutsquidlog.sh[编辑] gawk分割日志效率测试
* 日志文件大小
复制内容到剪贴板
代码:
# ll -h
total 3.2G
-rw-r----- 1 squid squid 636M Jun 13 14:54 access.log
-rw-r--r-- 1 root root 1.6G Jun 13 14:55 access2.log
-rw-r--r-- 1 root root 998M Jun 13 14:57 ask.eol.cn-access.log* 分离使用时间
复制内容到剪贴板
代码:
# time `gawk '$8~/'ask.eol.cn'/' access2.log > ask.eol.cn-access.log`
real 0m57.582s
user 0m17.157s
sys 0m5.560s 从1.6G的文件里，分离出998M的日志记录，只需要59秒。因为每个小时的日志达到1G的可能性不大，因此使用gawk分割不同域名的日志，完全不会影响服务器性能的。
gcc-4.1编译squid的出错信息
复制内容到剪贴板
代码:
/usr/bin/ar cru libdiskd.a diskd/diskd.o diskd/store_dir_diskd.o diskd/store_io_diskd.o
ranlib libdiskd.a
depbase=`echo coss/store_io_coss.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`; \
if gcc -DHAVE_CONFIG_H -I. -I. -I../../include -I. -I../../include -I../../include -I../../src -m32 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Wall -g -O2 -D_REENTRANT -MT coss/store_io_coss.o -MD -MP -MF "$depbase.Tpo" -c -o coss/store_io_coss.o coss/store_io_coss.c; \
then mv -f "$depbase.Tpo" "$depbase.Po"; else rm -f "$depbase.Tpo"; exit 1; fi
coss/store_io_coss.c: In function 'storeCossMaybeFreeBuf':
coss/store_io_coss.c:770: internal compiler error: in tree_low_cst, at tree.c:4399
Please submit a full bug report,
with preprocessed source if appropriate.
See <URL:[url]http://gcc.gnu.org/bugs.html[/url]> for instructions.
For Debian GNU/Linux specific bug reporting instructions,
see <URL:file:///usr/share/doc/gcc-4.1/README.Bugs>.
Preprocessed source stored into /tmp/ccuqODbe.out file, please attach this to your bugreport.
make[3]: *** [coss/store_io_coss.o] Error 1
make[3]: Leaving directory `/opt/distfiles/squid-2.6.STABLE13/src/fs'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/opt/distfiles/squid-2.6.STABLE13/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/opt/distfiles/squid-2.6.STABLE13/src'
make: *** [all-recursive] Error 1没有初始化就运行squid的错误信息
复制内容到剪贴板
代码:
2007/06/13 11:46:12| Process ID 24894
2007/06/13 11:46:12| With 1024 file descriptors available
2007/06/13 11:46:12| Using epoll for the IO loop
2007/06/13 11:46:12| Performing DNS Tests...
2007/06/13 11:46:12| Successful DNS name lookup tests...
2007/06/13 11:46:12| DNS Socket created at 0.0.0.0, port 32799, FD 5
2007/06/13 11:46:12| Adding domain eol.cn from /etc/resolv.conf
2007/06/13 11:46:12| Adding nameserver 202.205.109.1 from /etc/resolv.conf
2007/06/13 11:46:12| Adding nameserver 202.205.11.9 from /etc/resolv.conf
2007/06/13 11:46:12| User-Agent logging is disabled.
2007/06/13 11:46:12| Referer logging is disabled.
2007/06/13 11:46:12| Unlinkd pipe opened on FD 10
2007/06/13 11:46:12| Swap maxSize 8388608 KB, estimated 645277 objects
2007/06/13 11:46:12| Target number of buckets: 32263
2007/06/13 11:46:12| Using 32768 Store buckets
2007/06/13 11:46:12| Max Mem size: 1572864 KB
2007/06/13 11:46:12| Max Swap size: 8388608 KB
2007/06/13 11:46:12| Local cache digest enabled; rebuild/rewrite every 300/3600 sec
2007/06/13 11:46:12| Store logging disabled
FATAL: stripePath: Cannot stat /opt/cache1/coss1.
Squid Cache (Version 2.6.STABLE13): Terminated abnormally.
CPU Usage: 0.004 seconds = 0.004 user + 0.000 sys
Maximum Resident Size: 0 KB
Page faults with physical i/o: 0squid初始化时，io等待非常高
# vmstat -n 5
复制内容到剪贴板
代码:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 10 72 53060 4420 1952588 0 0 6 758 137 17 1 1 96 2
1 10 72 51452 5052 1945480 0 1 5 65536 790 288 0 30 1 69
0 10 72 53676 5332 1943580 0 0 2 63533 783 229 0 17 1 82
0 11 72 53580 5596 1941956 0 0 13 62267 796 230 0 15 1 85
0 11 72 54980 5892 1939164 0 0 19 62872 810 264 0 15 0 85
0 10 72 53808 6196 1939688 0 0 9 62882 809 232 0 16 0 84
0 10 72 51940 6372 1941932 0 0 2 62658 805 246 0 16 1 83
0 10 72 52148 6356 1941680 0 0 2 65944 820 281 0 17 3 79
1 9 72 53248 6568 1940516 0 0 2 65179 809 247 0 17 0 83
0 10 72 52244 5904 1941708 0 0 2 52913 830 255 0 14 0 86
0 10 72 52876 5180 1942484 0 0 4 102203 1073 2409 0 17 3 80
0 10 72 52116 4952 1946960 0 0 2 115633 1141 2256 0 16 3 81
1 6 72 51716 4468 1958284 0 0 3 113568 1105 436 0 24 5 70
0 2 72 53228 4076 1950412 0 0 6 63030 910 571 0 40 5 55
1 9 72 52100 3792 1950292 0 0 3 71111 892 356 0 24 0 76
0 10 72 53872 3684 1945640 0 0 2 69783 875 241 0 18 0 82
0 9 72 52960 3656 1941704 0 0 2 68410 870 298 0 17 1 82
1 9 72 53748 3644 1945496 0 0 2 70035 884 292 0 18 1 80
1 11 72 53012 3712 1946304 0 0 216 70226 895 277 0 17 0 83
1 10 72 52816 4052 1942832 0 0 5 67647 865 339 0 19 1 80
0 10 72 53480 4320 1942932 0 0 4 69870 881 280 0 13 0 87
1 4 72 53556 4376 1944684 0 0 1 63812 903 247 0 9 0 91
0 10 72 53832 4664 1943416 0 0 2 63185 889 268 0 15 0 84
1 10 72 49432 4836 1947448 0 0 1 63270 908 233 0 9 0 91
1 7 72 52628 5308 1945184 0 0 5 48394 850 299 0 23 16 60
1 9 72 52816 5012 1944856 0 0 3 70085 882 288 0 27 0 73
0 9 72 52504 4996 1945008 0 0 2 69722 881 251 0 15 0 85
1 14 72 54008 4748 1940896 0 0 2 69648 863 412 0 19 1 80
2 14 72 51508 4428 1943260 0 0 2 68810 870 314 0 17 0 83
0 14 72 53812 4124 1941580 0 0 2 69089 879 310 0 17 0 83
0 14 72 52712 4172 1942852 0 0 2 70387 884 298 0 17 0 83
1 11 72 52860 4308 1941784 0 0 2 56370 870 258 0 15 1 84
1 13 72 51940 4328 1944404 0 0 2 68749 873 359 0 17 1 81
0 13 72 52444 4328 1945264 0 0 2 68814 879 306 0 10 0 89
0 2 72 58032 4340 1945240 0 0 1 68338 861 179 0 2 8 90
0 0 72 70332 4348 1945240 0 0 0 60326 820 56 0 2 28 70
0 0 72 70580 4348 1945240 0 0 0 0 258 8 0 0 100 0
0 0 72 70836 4348 1945240 0 0 0 0 261 8 0 0 100 0
0 0 72 70844 4348 1945240 0 0 0 449 369 34 0 0 92 8
