Updates from 二月, 2012 Toggle Comment Threads | 键盘快捷键

  • bixuan 16:13 on 2012 年 02 月 11 日 链接地址 | 回复
    Tags: , 性能优化   

    性能黄金法则 

    原文地址:http://www.stevesouders.com/blog/2012/02/10/the-performance-golden-rule/

    昨天我在Google Ventures为他们的一些投资公司做了个研讨会。我不知道听众会有多少关于性能优化的背景知识,因此我从2007年的第一个演示开始,回顾了几乎跟性能优化相关的所有内容,真的是很怀旧啊。话说距离我开始谈论《高性能网站建设指南》的最佳实践已经很多年了,我重新审视了这些早期的提示,比如减少HTTP请求,和添加Expires头,还有压缩组件

    不过我还需要回顾得更远一些,想到在还没有VelocityWPO之前,我或许还得澄清一下为什么我会如此关注前端性能。我找到了当时包含性能黄金法则的幻灯片:

    80-90%的最终用户响应时间都花在前端上。
    从这里开始。

    还有一些其他相关的幻灯片展示了一些流行的网站分别花在后端和前端的时间,但是数据已经很旧并且很有限了,因此我决定更新一下,下面是我的成果。

    首先是一个瀑布图,它展示了前后端的划分。这个瀑布图是LinkedIn的。这里“后端”的时间是指从服务器返回第一个字节到客户端所花费的时间。它通常包含大部分的后端处理:数据库查询、远程web服务调用、拼接HTML等等。其余的是“前端”的时间,它包含了显而易见的前端阶段,诸如执行JavaScript代码以及渲染页面等。它同时也包含了下载页面上所有相关资源的时间。我把这些划分到前端时间里是因为,有许多切实可行的办法可以减少这个时间,比如 异步加载脚本合并脚本和样式表以及域名分散(即通过多个域名实现并行下载的策略  ——译者注)等。

    Golden waterfall

    对于排名前十位的网站分析结果显示,平均在前端花费的时间占比为76%,略低于黄金法则中提出的80-90%的值。不过别忘了,这些网站的前端都经过了高度的优化,并且其中两个是载入资源非常少的搜索页面(而不是结果页面)。

    Golden top10

    对于排名10000左右的10个网站进行的分析,可以得到一个更典型的视图。平均在前端花费的时间占比为92%,高于排名前10的76%,甚至高于黄金法则中建议的80-90%。

    Golden 9990

    为了使与会者接受这个法则,我展示了他们自己网站的前后端花费时间占比,得到的结果为前端占比84%。这有助于使他们的认可我的理论,即前端的性能才是最难最有挑战的,也是最应该给予关注的。

    Golden startups

    后来我想起来我在HTTP Archive上还有关于网站耗时的信息。不过我一般不展示这些信息,因为我认为真正的用户度量应该更准确一些,不过我计算了被抓取到的50000个网站的前后端耗时占比,结果前端占比为87%。

    Top50ksite

    能够获取这些比2007年更新的信息来验证性能黄金法则真是太好了,而且它也显示了前端性能优化越来越受重视了。如果你担心可用性和可扩展性,那就关注一下后端。但是如果你担心载入网站时用户等待的时间太久,那么关注前端才是王道。

    FROM:http://44ux.com/index.php/2012/02/the-performance-golden-rule/

     
  • bixuan 13:48 on 2011 年 03 月 05 日 链接地址 | 回复
    Tags: , ,   

    安装php-handlersocket 

    至于HandlerSocket的安装请参考:MySQL+HandlerSocket安装一文。

    安装php-handlersocket一开始还遇到了点小问题,说找不到-Ihsclient的库文件,估计是我安装HandlerSocket-Plugin的时候指定了prefix,而在编译php-handlersocet的时候没加环境导致的,后来为了偷懒,就直接安装到/usr/local了。

    wegt -c http://php-handlersocket.googlecode.com/files/php-handlersocket-0.0.7.tar.gz
    tar zxvf php-handlersocket-0.0.7.tar.gz
    cd php-handlersocket
    PREFIX=/opt/app
    $PREFIX/php5-fastcgi/bin/phpize
    ./configure --with-handlersocket \
    --with-handlersocket-includedir=$PREFIX/HandlerSocket-Plugin/include/handlersocket \
    --with-php-config=$PREFIX/php5-fastcgi/bin/php-config
    make && make install
    cd ..

    然后将extension=”handlersocket.so”加到php,ini配置即可。

     
  • bixuan 14:18 on 2011 年 02 月 25 日 链接地址 | 回复
    Tags: , , 故障   

    一个mysql的小故障 

    今天机器重启后,发现一个innodb(mysql-5.0.45)的表提示如下错误:Incorrect information in file: ‘./MD/t_monitor_client.frm’,

    原因是:表结构出现错误。

    解决方法:
    1、停止mysql;
    2、在其他的机器上重该表,
    3、将t_monitor_client.frm复制到故障机器上,
    4、启动mysql,
    5、检查,没问题就成功了。

     
  • bixuan 00:31 on 2011 年 02 月 19 日 链接地址 | 回复
    Tags: ,   

     
  • bixuan 00:29 on 2011 年 02 月 19 日 链接地址 | 回复
    Tags:   

     
  • bixuan 00:28 on 2011 年 02 月 19 日 链接地址 | 回复
    Tags: ,   

     
  • bixuan 21:42 on 2011 年 02 月 17 日 链接地址 | 回复
    Tags: , geoip   

    HOWTO Implement GeoDNS using BIND & MaxMind

    This HOWTO documents an elegant Linux BASH script that can be used to help configure BIND to be geo-aware. The script utilises the information contained within the freely downloadable GeoIP CSV file, published monthly, by MaxMind. No patching of the BIND source code is required for this to work (unlike other methods that have been documented online) thus making it easier to manage GeoIP updates to BIND as and when MaxMind publish updated versions of their GeoIP CSV file or the ISC release newer versions of BIND. If you are seeking to implement geo-aware DNS with BIND on theIPv6 network, you will probably find this extremely useful.

    Licensing & Copyright

    The copyrighted material on this page is made available to anyone wishing to use, modify, copy, or redistribute it subject to the terms and conditions of the GNU General Public License. The scripts published on this page are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY expressed or implied, including the implied warranties of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. For further information, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

    BUG FIX ANNOUNCEMENT

    If you have accessed this page before the 1st of January 2010, and thus are using these scripts as they were published on this page before this date, changes have since been made to them to address a couple of discovered issues.

    1. The first is a change to the fastest recursive script. The change is nothing major but effectively reduces execution time slightly by splitting IP ranges when generating the GeoIP.acl file rather than splitting IP ranges when creating the CBE (Country,Begin,End) CSV file. The change is purely in relation to where the range splitting takes place, resulting in grep pattern matching against fewer lines, thus marginally reducing the execution time of the script.
    2. The second fix has been made to all scripts and was discovered when noticing that the recursive awk function could not correctly split extremely large IP ranges, with an order of magnitude exceeding about 231. For example, giving the script the range 0 to 2147483647 would result in it printing 0.0.0.0/0 rather than 0.0.0.0/1. I located this issue to a rounding anomaly with the printf function within awk and the solution is to simply ensure that all occurrences of the logarithmic division calculation in each script are truncated to a whole number using the int function. This bug has probably not caused people too much grief because the ranges supplied within the MaxMind GeoIP CSV file are nowhere near a magnitude of 231 (the largest IP range listed as of writing is of magnitude 226, representing the network 28.0.0.0/6 in the United States). Nevertheless, this was a bug and has now been fixed in the scripts published below.

    Overview

    I was recently asked by my employer to bring our DNS in-house from UltraDNS where we originally hosted all our domain names. Due to various requirements within the company, they were utilising UltraDNS’s geo-targetting feature to enable internet users in different areas of the world to resolve hosts on our domains to varying IP addresses, depending on the geographical (country) location of these users.

    Having already been exposed to BIND’s views feature some years ago, I googled on how it would be possible to make BIND geo-aware. There is not much documentation about this online but I found one such solution which involved patching the BIND source code. All well and good but, in all honesty, this seemed like using a sledge hammer to crack a nut. Besides, our company does not like patching (hacking) source code unless there is a real requirement to do so as it normally entails maintenance by having to refit changes into revisions of the BIND source code as and when the ISC release newer versions of BIND.

    I analysed the patching BIND method further and the solution still uses two fundamental things to achieve a geo-aware DNS setup; BIND’s views feature and the freely downloadable GeoIP data available from MaxMind. It was then I realised that to make BIND geo-aware, all that is required is to reformat the data in the MaxMind GeoIP CSV file into something which BIND likes, and will accept in its configuration file. The easiest and most manageable way to achieve this is by using the BIND Access Control List clause, but here lies the problem. The MaxMind GeoIP CSV file operates in IP ranges whereas BIND ACLs operate on IP networks, in classic net/mask notation. So, basically, I had to formulate a method to transform MaxMind IP ranges into BIND ACLs. This method is attainable by using the Linux BASH script(s) shown below.

    The result is the automatic creation of a single and maintainable GeoIP.acl include file that can be instantly added into any already running BIND DNS server, without the requirement for source code patching and recompilation, producing a geo-aware production-ready DNS server in a matter of minutes.

    Linux BASH script(s) to fetch, unzip, reformat and generate the GeoIP.acl include file for BIND

    There are two different BASH scripts documented below which will generate the GeoIP.acl include file for BIND. The second is an improvement over the first but I’ve left it documented anyway as it was my original implementation. The first uses an iterative BASH loop (slower) whereas the second uses a recursive AWK function (much faster). Both achieve exactly the same thing by employing different programming constructs. For speed and efficiency, I recommend using thesecond recursive script.

    NOTE: By default, some distributions of Linux use a non-GNU version of AWK which lacks the bitwise AND function. In this instance, GAWK must be installed (the GNU version of AWK) for the scripts below to function correctly (thanks to Ruben for pointing this out).

    Each script will attempt to download the latest MaxMind GeoIP CSV file (which is actually a ZIP file). Once downloaded, it will use this file and reprocess it each time it is executed. Removing the ZIP file and then rerunning the script will force it to perform another fetch from MaxMind. Once the ZIP file has been fetched, each script will unzip it, reformat the enclosed GeoIP CSV file (taking several passes to do this if the iterative version is used) and then generate the file GeoIP.aclwhich is the include file that can be added into BIND’s configuration to make it geo-aware.

    Iterative Version (slowest)

    #!/bin/bash
    
    [ -f GeoIPCountryCSV.zip ] || wget -T 5 -t 1 http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
    unzip GeoIPCountryCSV.zip || exit 1
    
    echo -n "Creating initial CBE (Country,Begin,End) CSV file..."
    awk -F \" '{print $10","$6","$8}' GeoIPCountryWhois.csv > cbe0.csv
    rm -f GeoIPCountryWhois.csv
    echo -ne "DONE\nSplitting CBE CSV file..."
    
    lc0=0; lc1=$(wc -l cbe0.csv | awk '{print $1}')
    
    while [ $lc0 -lt $lc1 ]
    do
      lc0=$lc1; echo -ne "\n$lc0\t"
      awk -F , '{m = 2^32-2^int(log($3-$2+1)/log(2)); n = and(m,$3); if (n == and(m,$2)) print; else printf "%s,%u,%u\n%s,%u,%u\n",$1,$2,n-1,$1,n,$3}' cbe0.csv > cbe1.csv
      mv -f cbe1.csv cbe0.csv; lc1=$(wc -l cbe0.csv | awk '{print $1}')
      echo -ne "+$[$lc1-$lc0]\t"; [ $lc0 -lt $lc1 ] && echo -n "OK"
    done
    
    echo -ne "DONE\nGenerating BIND GeoIP.acl file..."
    
    (for c in $(awk -F , '{print $1}' cbe0.csv | sort -u)
    do
      echo "acl \"$c\" {"
      grep "^$c," cbe0.csv | awk -F , '{printf "\t%u.%u.%u.%u/%u;\n",$2/2^24%256,$2/2^16%256,$2/2^8%256,$2%256,32-int(log($3-$2+1)/log(2))}'
      echo -e "};\n"
    done) > GeoIP.acl
    
    rm -f cbe0.csv
    echo "DONE"
    
    exit 0
    Here’s this script in action!
    $ ./GeoIP.sh
    --00:00:00--  http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
               => `GeoIPCountryCSV.zip'
    Resolving geolite.maxmind.com... 64.246.48.99
    Connecting to geolite.maxmind.com|64.246.48.99|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1,556,500 (1.5M) [application/zip]
    
    100%[================================================================================>] 1,556,500    820.41K/s
    
    00:00:02 (818.35 KB/s) - `GeoIPCountryCSV.zip' saved [1556500/1556500]
    
    Archive:  GeoIPCountryCSV.zip
      inflating: GeoIPCountryWhois.csv
    Creating initial CBE (Country,Begin,End) CSV file...DONE
    Splitting CBE CSV file...
    106184  +31276  OK
    137460  +23038  OK
    160498  +11755  OK
    172253  +6413   OK
    178666  +3544   OK
    182210  +1905   OK
    184115  +949    OK
    185064  +463    OK
    185527  +202    OK
    185729  +94     OK
    185823  +38     OK
    185861  +19     OK
    185880  +5      OK
    185885  +2      OK
    185887  +0      DONE
    Generating BIND GeoIP.acl file...DONE

    #!/bin/bash
    
    [ -f GeoIPCountryCSV.zip ] || wget -T 5 -t 1 http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
    unzip GeoIPCountryCSV.zip || exit 1
    
    echo -n "Creating CBE (Country,Begin,End) CSV file..."
    awk -F \" '{print $10","$6","$8}' GeoIPCountryWhois.csv > cbe.csv
    rm -f GeoIPCountryWhois.csv
    echo -ne "DONE\nGenerating BIND GeoIP.acl file..."
    
    (for c in $(awk -F , '{print $1}' cbe.csv | sort -u)
    do
      echo "acl \"$c\" {"
      grep "^$c," cbe.csv | awk -F , 'function s(b,e,l,m,n) {l = int(log(e-b+1)/log(2)); m = 2^32-2^l; n = and(m,e); if (n == and(m,b)) printf "\t%u.%u.%u.%u/%u;\n",b/2^24%256,b/2^16%256,b/2^8%256,b%256,32-l; else {s(b,n-1); s(n,e)}} s($2,$3)'
      echo -e "};\n"
    done) > GeoIP.acl
    
    rm -f cbe.csv
    echo "DONE"
    
    exit 0
    Here’s this script in action!
    $ ./GeoIP.sh
    --00:00:00--  http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
               => `GeoIPCountryCSV.zip'
    Resolving geolite.maxmind.com... 64.246.48.99
    Connecting to geolite.maxmind.com|64.246.48.99|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1,556,500 (1.5M) [application/zip]
    
    100%[================================================================================>] 1,556,500    820.41K/s
    
    00:00:02 (818.35 KB/s) - `GeoIPCountryCSV.zip' saved [1556500/1556500]
    
    Archive:  GeoIPCountryCSV.zip
      inflating: GeoIPCountryWhois.csv
    Creating CBE (Country,Begin,End) CSV file...DONE
    Generating BIND GeoIP.acl file...DONE

    Both of these scripts will generate the file GeoIP.acl in the current working directory which looks something like this:

    acl "A1" {
            64.46.32.0/23;
            64.46.35.0/24;
            64.46.40.64/26;
            64.46.42.0/23;
            64.46.47.0/24;
            66.38.243.0/24;
            67.15.183.0/25;
            69.10.130.128/26;
            69.10.139.0/25;
            69.10.140.192/26;
    
    ...
    
    acl "GB" {
            2.6.190.56/29;
            9.20.0.0/17;
            12.129.72.32/29;
            23.0.0.0/9;
            25.0.0.0/8;
            32.58.57.0/29;
            32.58.58.0/28;
            32.58.59.0/29;
            32.60.34.96/27;
            51.0.0.0/8;
    
    ...
    
            217.204.159.96/29;
            217.204.159.104/30;
            217.204.159.112/28;
            217.204.159.128/25;
            217.204.160.0/19;
            217.204.192.0/18;
            217.205.0.0/16;
            217.206.0.0/15;
            217.237.189.240/29;
            217.243.204.144/29;
    };
    
    ...
    
            217.194.132.0/24;
            217.194.145.144/29;
            217.194.146.192/26;
            217.194.147.240/28;
            217.194.149.32/28;
            217.194.149.168/29;
            217.194.156.0/26;
            217.194.157.48/28;
            217.194.157.144/29;
            217.194.157.168/29;
    };

    How do these scripts work?

    I wont go into the technicalities of how these scripts work (this is left as an exercise for the reader) but the first iterative script creates a new CSV file containing 3 fields (Country,Begin,End) and then repeatedly searches for and splits these IP ranges on network boundaries so we are left with a CSV file that has exactly the same coverage of IPs as before but has been processed so that the IP ranges reside on values that allow for each range to be expressed concisely in net/mask notation. The final part of the script then uses this CSV file to generate the GeoIP.acl include file.

    The second recursive script achieves the same result faster by creating a new CSV file as before, containing 3 fields (Country,Begin,End), and then performing recursive range splitting “on the fly” within awk itself, for each country, to generate the GeoIP.acl include file.

    Once either of these scripts have finished running, you can slot the newly created GeoIP.acl file straight into your existing BIND configuration file, by adding the line:

    include "/path/to/GeoIP.acl";

    to named.conf. It will then be possible to create custom geo-views within BIND, like this:

    view "north_america" {
      match-clients { US; CA; MX; };
      recursion no;
      zone "example555.com" {
        type master;
        file "pri/example555-north-america.db";
      };
    };
    
    view "south_america" {
      match-clients { AR; CL; BR; PY; PE; EC; CO; VE; BO; UY; };
      recursion no;
      zone "example555.com" {
        type master;
        file "pri/example555-south-america.db";
      };
    };
    
    view "other" {
      match-clients { any; };
      recursion no;
      zone "example555.com" {
        type master;
        file "pri/example555-other.db";
      };
    };

    If you decide to cron these scripts within your BIND name server(s), do remember to reload named (normally achieved by running the command service named reload on RedHat/CentOS) so the new ACL definitions within the GeoIP.acl file are loaded into BIND’s memory.

    Summary

    I hope this article proves useful for others (that’s why I have documented it). Interestingly, my original implementation of this was by using a PHP script coupled with MySQL, loading the MaxMind CSV file into a database table, and then running SELECT, UPDATE and INSERT queries to split up the IP ranges. Whilst this worked, it depended on having PHP and MySQL installed and configured. The above scripts achieve exactly the same thing but only using BASH commands and utilities, such as awk,grep and sort, which in my view, is far cleaner!

    Incidently, it is actually possible to produce the GeoIP.acl file without using grep or any intermediate CSV file (shown below). These scripts may be used instead but with markedly longer execution times and, because of this, an echo statement, outputting the current country code to standard error, has been introduced into the main loop to give an indication of progress while the script is running.

    Recursive Versions (smallest)

    #!/bin/bash
    
    [ -f GeoIPCountryCSV.zip ] || wget -T 5 -t 1 http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
    unzip GeoIPCountryCSV.zip || exit 1
    
    (for c in $(awk -F \" '{print $10}' GeoIPCountryWhois.csv | sort -u)
    do
      echo "$c" >&2
      echo "acl \"$c\" {"
      awk -F \" 'function s(b,e,l,m,n) {l = int(log(e-b+1)/log(2)); m = 2^32-2^l; n = and(m,e); if (n == and(m,b)) printf "\t%u.%u.%u.%u/%u;\n",b/2^24%256,b/2^16%256,b/2^8%256,b%256,32-l; else {s(b,n-1); s(n,e)}} c == $10 {s($6,$8)}' c=$c GeoIPCountryWhois.csv
      echo -e "};\n"
    done) > GeoIP.acl
    
    rm -f GeoIPCountryWhois.csv
    
    exit 0

    We can marginally reduce the execution time of the above script by adjusting its awk line to match the current country using a regular expression, as opposed to setting the awk variable c and then checking if c == $10, as follows:

    #!/bin/bash
    
    [ -f GeoIPCountryCSV.zip ] || wget -T 5 -t 1 http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
    unzip GeoIPCountryCSV.zip || exit 1
    
    (for c in $(awk -F \" '{print $10}' GeoIPCountryWhois.csv | sort -u)
    do
      echo "$c" >&2
      echo "acl \"$c\" {"
      awk -F \" 'function s(b,e,l,m,n) {l = int(log(e-b+1)/log(2)); m = 2^32-2^l; n = and(m,e); if (n == and(m,b)) printf "\t%u.%u.%u.%u/%u;\n",b/2^24%256,b/2^16%256,b/2^8%256,b%256,32-l; else {s(b,n-1); s(n,e)}} '"/,\"$c\",/"' {s($6,$8)}' GeoIPCountryWhois.csv
      echo -e "};\n"
    done) > GeoIP.acl
    
    rm -f GeoIPCountryWhois.csv
    
    exit 0

    Do note, however, that I personally prefer the previous grep method as it is much faster than these two scripts because it initially reformats the data within the CSV file into something that allows for fast regex pattern matching on the country field (by moving this field to the beginning of each line) allowing awk to take care of the more complicated task of IP range splitting that operates on the begin (2nd) and end (3rd) integer IP fields.

    Over the last decade, IPv6 has become more and more mainstream. As much as I have not seen any requirement for geo-aware DNS serving on the IPv6 network, I would imagine this will gradually become needed. BIND already handles IPv6 addresses within its ACLs so I have published further scripts below that allow the creation of aGeoIPv6.acl include file containing IPv6 net/mask entries, using the freely downloadable GeoIPv6 CSV file available from MaxMind.

    It was a challenge to come up with a working solution using the same principles as in the above scripts, but across a much larger address space. This is because IPv6 uses a 128 bit address space, compared to IPv4 being only 32 bits. The scripts above get away with using simple BASH utilities such as awk for doing the necessary IP range splitting with 32 bits but, as I found out, awk is unable to handle numbers which are up in the realms of 64 bits and beyond. So I’ve had to pull various different Linux utilities into play here to achieve this.

    In order to handle large numbers up to and beyond 64 bits in magnitude, one has to look at other programming languages and the libraries they offer. After evaluating today’s available languages like Python (which handles large numbers out the box) and PHP (which can only handle large numbers with an additional library installed), I decided to go with Perl. Perl has, on most standard installs, a bignum library that is available and ready to go. This library is transparent and as soon as it is included into a script, all number processing will automatically use it. It has all the necessary operations like bitwise AND that the above scripts make use of. However, when writing the Perl script below, I ran into an inconsistency with the log function whilst using the bignum library and, for anything above 64 bits,bignum also exhibits major rounding anomalies. To avoid this curveball, I decided to bring the common Linux arbitrary precision calculator bc into play to take over both of these roles. Together, Perl and bc offer the accuracy and speed required to split decimal IP ranges with magnitudes of 64 bits and beyond.

    So, here are the scripts. The first script is, as before, a standard BASH script (called GeoIPv6.sh). It is much the same as before but rather than piping the filtered grep lines to awk, it pipes them to a newly created Perl script instead. It also contains some further adjustments at the top to download the latest GeoIPv6 CSV file from MaxMind’s servers, as well as an optional pipe of the Perl script output to sed to abbreviate IPv6 addresses to their “double-colon (::) notation” equivalent.

    #!/bin/bash
    
    d=http://geolite.maxmind.com/download/geoip/database/
    f=$(wget -qT 5 -t 1 -O- $d | egrep -o 'GeoIPv6-[0-9]{8}\.csv\.gz' | head -1)
    [ -z "$f" ] && exit 1; [ -f $f ] || wget -T 5 -t 1 $d$f || exit 1
    
    echo -n "Creating CBE (Country,Begin,End) CSV file..."
    gunzip -c $f | awk -F \" '{print $10","$6","$8}' > cbe.csv
    echo -e "DONE\nGenerating BIND GeoIPv6.acl file..."
    
    (for c in $(awk -F , '{print $1}' cbe.csv | sort -u)
    do
      echo "$c" >&2
      echo "acl \"${c}v6\" {"
      grep "^$c," cbe.csv | ./GeoIPv6.pl | sed 's \(:0\)\+/ ::/ '
      echo -e "};\n"
    done) > GeoIPv6.acl
    
    rm -f cbe.csv
    echo "DONE"
    
    exit 0

    The Perl script I have called GeoIPv6.pl, with the following contents:

    #!/usr/bin/perl
    
    use strict;
    use bignum;
    use IPC::Open2; open2(*BCOUT,*BCIN,'bc -l');
    
    sub rs {
      my ($b,$e) = @_;
      print BCIN "scale=40; l($e-$b+1)/l(2)\n";
      my ($l) = split('\.',<BCOUT>);
      my $m = 2**128-2**$l;
      my $n = $m & $e;
      if ($n == ($m & $b)) {
        my @x; for (my $p = 112; $p > 0; $p -= 16) {
          print BCIN "scale=0; $b/2^$p\n";
          push(@x,<BCOUT>%65536);
        }
        printf "\t%x:%x:%x:%x:%x:%x:%x:%x/%u;\n",$x[0],$x[1],$x[2],$x[3],$x[4],$x[5],$x[6],$b%65536,128-$l;
      } else {
        rs($b,$n-1); rs($n,$e);
      }
    }
    
    while (<STDIN>) {chomp($_); my ($c,$b,$e) = split(',',$_); rs($b,$e)}

    This Perl script effectively reads from standard input in precisely the same way as the original awk script does (expecting each line to be in the format of a CBE (Country,Begin,End) CSV file) but, unlike awk, can perform IP range splitting on 128 bit decimal numbers, printing IPv6 net/mask entries to standard output. Note the use of a dual pipe to the Linux arbitrary precision calculator bc to manage the logarithmic division calculation and also to accurately truncate values before they are passed to the printf function (done by a small for loop that places these entries into an array). Most importantly, note that we must increase the default scaleof 20 within bc to at least 40 to be able to accurately cope with the logarithmic division calculation. Observe:

    $ echo 'l(2^128-1)/l(2)' | bc -l
    128.00000000000000000132
    $ echo 'scale=20; l(2^128-1)/l(2)' | bc -l
    128.00000000000000000132
    $ echo 'scale=39; l(2^128-1)/l(2)' | bc -l
    128.000000000000000000000000000000000000088
    $ echo 'scale=40; l(2^128-1)/l(2)' | bc -l
    127.9999999999999999999999999999999999999956

    The reason we also choose to open a dual pipe to bc within Perl is to stop the forking of a separate bc process each time we need to perform a division calculation (forking a new process is costly in terms of CPU time). By opening up a dual pipe to a single persistent bc process, we can simply throw and retrieve each calculation into and out off it quickly. The IPC::Open2 Perl module is required to do dual pipes and this may need to be installed on your system.

    Once these two scripts have been created, it will be possible to run ./GeoIPv6.sh to generate the GeoIPv6.acl include file for BIND. Note that the execution time here will be far greater than before, since we are using Perl with bignum support, and passing division calculations to a separate persistent bc process. As such, the BASH script has been modified to output the current country code being processed to standard error to indicate progress. Once the script has completed execution, the GeoIPv6.acl include file will have been created in the current working directory, which looks something like this:

    acl "ADv6" {
            2001:4df8::/32;
    };
    
    acl "AEv6" {
            2001:8f8::/32;
            2a00:d30::/32;
            2a00:f28::/32;
    };
    
    acl "AMv6" {
            2001:1bb0::/32;
            2001:4d00::/32;
            2a00:f38::/32;
            2a00:1290::/32;
            2a00:1500::/32;
            2a02:d18::/32;
    
    ...
    
    acl "GBv6" {
            2001:630::/32;
            2001:678:4::/47;
            2001:67c:18::/48;
            2001:67c:90::/48;
            2001:67c:b4::/48;
            2001:67c:c0::/48;
            2001:67c:d4::/48;
            2001:6f8::/32;
            2001:710::/32;
            2001:768::/32;
    
    ...
    
            2a02:ce8::/32;
            2a02:da0::/32;
            2a02:df8::/32;
            2a02:e38::/32;
            2a02:e68::/32;
            2a02:eb0::/32;
            2a02:ef8::/32;
            2a02:f70::/32;
            2a02:fb0::/32;
            2a02:fb8::/32;
    };
    
    ...
    
            2001:43d8::/32;
            2001:43f8:20::/48;
            2001:43f8:30::/48;
            2001:43f8:40::/48;
            2001:43f8:50::/48;
            2001:43f8:70::/45;
            2001:43f8:90::/48;
            2001:43f8:a0::/48;
            2001:43f8:d0::/48;
    };
    
    acl "ZWv6" {
            2001:42b0::/32;
    };

    Performance versus Maintainability (pros/cons for/against this ACL method compared to BIND source code patching)

    John ‘Warthog9′ Hawley, the chief administrator of http://www.kernel.org (a high-traffic site which implemented BIND GeoDNS on the 19th of September 2008 via patching), recently contacted me about this HOWTO with some interesting points concerning the implications of using this ACL method over BIND source code patching. I will briefly discuss this here, as it will affect which route you take when implementing GeoDNS within BIND.

    In a nutshell, patching BIND for GeoDNS support results in a DNS server that can answer queries at an extremely rapid rate compared with this ACL method (I have confirmed this; it is quite easy to test; see below). This is because the MaxMind binary database is a binary search tree data structure, and so the worst case maximum number of lookups required to determine the country location of an IPv4 address will be 32 iterations (and most times, far less than this). Similary, for their IPv6 binary database, this number changes to 128 iterations. As you can imagine, patching the MaxMind GeoIP C library directly into BIND to achieve GeoDNS will result in a server which is able to process, lookup and answer DNS queries with very few CPU cycles. As such, if your DNS servers are high-traffic servers, responding to many DNS requests per second, it would be advisable to go with the source code patching route.

    Alternatively, if maintainability is of more importance to you, the ACL method described in this HOWTO is still a viable option, but with the consequence of a substantial performance hit. According to John (who has been chatting with Paul Vixie, the primary author and architect of BIND until release 8), the ACL feature was never designed with the intention to store and hold the number of ACL entries that the above scripts generate, for GeoDNS purposes. This I can believe, as the scripts above (for IPv4) produce an ACL definition file containing over 200,000 ACL entries, which BIND has to load and subsequently store in its memory once launched. I am not fully aware of the data structures used within BIND to store ACLs, but they will be far less efficient than the simple binary search tree that MaxMind offer with their binary GeoIP databases. It is for this reason that the ACL method described in this HOWTO will result in a far slower DNS server, depending on how many views you create and the ACLs assigned to them.

    To give you an idea of just how much of a performance hit this ACL method induces, I have a small low-power server on my network running a CentaurHauls VIA Nehemiah CPU @ 1 GHz (2000 BogoMips) with a 192.168.0.0/16 IP address (see RFC 1918; all other hosts on my LAN are in this network so none of them would be a match in any of the above ACLs). When loading BIND with the GeoIP.acl include file, and creating a catch-all view that matches any client (not using any of the ACLs in the GeoIP.acl include file), the DNS response time tends to be about 2 ms. If, however, another view is created before this catch-all one in named.conf, and the clause:

    match-clients { A1; A2; AD; AE; AF; AG; AI; AL; AM; AN; ... VI; VN; VU; WF; WS; YE; YT; ZA; ZM; ZW; };

    is added to this view (forcing it to attempt a match across every single ACL definition inside the GeoIP.acl file), the response time sores to around 85 ms. In other words, the amount of work that we have now asked BIND to do, in order for it to verify if any of the ACLs are a match for a client with IP address in 192.168.0.0/16, has resulted in it slowing down by a factor of 40 (a rough guestimate figure only) which is a substantial performance hit that needs to be considered. For this reason, if using the ACL method described in this HOWTO, try and limit the number of views you create and the number of ACLs assigned to them as this will lower the amount of work BIND has to do when answering DNS queries made to it.

    In short, you should determine if speed (source code patching) or maintainability (ACL include file) is of more importance to you and be fully aware of the pros and cons of each method of GeoDNS implementation within BIND. As a systems administrator, use your head to decide which method to go with. As http://www.kernel.org is a global site, ranked around 10,000 across all sites on the internet (according to Alexa), John has done the right thing and gone with the patching method when deploying BIND GeoDNS servers for Kernel.org.

    From:http://phix.me/geodns/

     
  • bixuan 19:34 on 2011 年 01 月 16 日 链接地址 | 回复
    Tags: , Eucalyptus, 云平台   

    Installing Eucalyptus (2.0) on Centos 5.5(在Centos5.5上搭建私有云平台) 

    Eucalyptus can be installed on CentOS 5 from source or by using binary RPM packages. This document details the steps required to install Eucalyptus from RPMs. In what follows, the value of $VERSION must be set to the version of Eucalyptus you wish to install. For example, you can set the value to 2.0.2 using bash:

    export VERSION=2.0.2

    Notice: Before you begin, please ensure that you have an up-to-date CentOS installation on your target machine(s).

    Prerequisites

    If you start with a standard CentOS installation, you will satisfy all prerequisites with the following steps:

    1. Front-end, node(s), and client machine system clocks are synchronized (e.g., using NTP).
      yum install -y ntp
      ntpdate pool.ntp.org
    2. Front end needs java, command to manipulate a bridge, and the binaries for dhcp server (do not configure or run dhcp server on the CC):
      yum install -y java-1.6.0-openjdk ant ant-nodeps dhcp bridge-utils perl-Convert-ASN1.noarch scsi-target-utils httpd
    3. Node has a fully installed and configured installation of Xen that allows controlling the hypervisor via HTTP from localhost.
      yum install -y xen
      sed --in-place 's/#(xend-http-server no)/(xend-http-server yes)/' /etc/xen/xend-config.sxp
      sed --in-place 's/#(xend-address localhost)/(xend-address localhost)/' /etc/xen/xend-config.sxp
      /etc/init.d/xend restart
    4. Firewall rules must permit the Eucalyptus components to communicate with one another, and clients to communicate with Eucalyptus. On the front-end, ports 8443, 8773, 8774 and 9001 must be open; on the node, port 8775 must be open. If you are planning on using Elastic IPs and/or Security Groups, consider disabling the firewall and use Eucalyptus facilities for enabling custom firewall rules (see Network configuration for more information). To do so, on both the front-end and the nodes:
      • run system-config-securitylevel-tui
      • select Security Level: Disabled
      • select OK

    Download and Install RPMs

    Eucalyptus binary installation is broken up into several packages: one for each of the components (CLC, Walrus, CC, etc.), as well as a couple of common packages.

    There are two options for downloading and installing the packages:

    1. Yum option

    2. Packages are available from our yum repository. To use this option, create ‘/etc/yum.repos.d/euca.repo’ file with the following four lines:

      [euca]
      name=Eucalyptus
      baseurl=http://www.eucalyptussoftware.com/downloads/repo/eucalyptus/$VERSION/yum/centos/
      enabled=1

      Now install Eucalyptus on the front-end:

      yum install eucalyptus-cloud.$ARCH eucalyptus-cc.$ARCH eucalyptus-walrus.$ARCH eucalyptus-sc.$ARCH --nogpgcheck

      and install Eucalyptus on the node:

      yum install eucalyptus-nc.$ARCH --nogpgcheck

      where $ARCH is the architecture of your host (either ‘i386′ or ‘x86_64′).

    1. Tarball option

    2. The packages are available in a single tarball, wherein we also include copies of third-party CentOS packages that Eucalyptus depends on (Rampart, Axis2C, many Java libraries), at http://open.eucalyptus.com/downloads (look for a CentOS tarball of the right Eucalyptus version and architecture).

      Untar the bundle in a temporary location:

      tar zxvf eucalyptus-$VERSION-*.tar.gz
      cd eucalyptus-$VERSION-*

      In the examples below we use x86_64, which should be replaced with i386 or i586on 32-bit architectures.

      Install RPMs on the front end

      First, on the front end, install third-party dependency RPMs:

      cd eucalyptus-$VERSION*-rpm-deps-x86_64
      
      rpm -Uvh aoetools-21-1.el4.x86_64.rpm \
               euca-axis2c-1.6.0-1.x86_64.rpm \
               euca-rampartc-1.3.0-1.x86_64.rpm \
               vblade-14-1mdv2008.1.x86_64.rpm \
               vtun-3.0.2-1.el5.rf.x86_64.rpm \
               lzo2-2.02-3.el5.rf.x86_64.rpm\
             	 perl-Crypt-OpenSSL-Random-0.04-1.el5.rf.x86_64.rpm\
               perl-Crypt-OpenSSL-RSA-0.25-1.el5.rf.x86_64.rpm\
               perl-Crypt-X509-0.32-1.el5.rf.noarch.rpm\
               python25-2.5.1-bashton1.x86_64.rpm\
               python25-devel-2.5.1-bashton1.x86_64.rpm\
               python25-libs-2.5.1-bashton1.x86_64.rpm
      cd ..

      then install the -cloud, -walrus, -cc and -sc RPMs:

      rpm -Uvh eucalyptus-$VERSION-*.x86_64.rpm \
               eucalyptus-common-java-$VERSION-*.x86_64.rpm \
               eucalyptus-cloud-$VERSION-*.x86_64.rpm \
               eucalyptus-walrus-$VERSION-*.x86_64.rpm \
               eucalyptus-sc-$VERSION-*.x86_64.rpm \
               eucalyptus-cc-$VERSION-*.x86_64.rpm \
               eucalyptus-gl-$VERSION-*.x86_64.rpm

      Install RPMs on the nodes

      Next, on each node install the dependency packages:

      cd eucalyptus-$VERSION*-rpm-deps-x86_64
      rpm -Uvh aoetools-21-1.el4.x86_64.rpm \
               euca-axis2c-1.6.0-1.x86_64.rpm \
               euca-rampartc-1.3.0-1.x86_64.rpm\
               perl-Crypt-OpenSSL-Random-0.04-1.el5.rf.x86_64.rpm\
               perl-Crypt-OpenSSL-RSA-0.25-1.el5.rf.x86_64.rpm\
               perl-Crypt-X509-0.32-1.el5.rf.noarch.rpm\
               python25-2.5.1-bashton1.x86_64.rpm\
               python25-devel-2.5.1-bashton1.x86_64.rpm\
               python25-libs-2.5.1-bashton1.x86_64.rpm
      cd ..

      then install the node controller RPM with dependencies:

      rpm -Uvh eucalyptus-$VERSION-*.x86_64.rpm \
               eucalyptus-gl-$VERSION-*.x86_64.rpm \
               eucalyptus-nc-$VERSION-*.x86_64.rpm

    Post-Install Steps

    The last step in the installation is to make sure that the user ‘eucalyptus’, which is created at RPM installation time, is configured to interact with the hypervisor through libvirt on all of your compute nodes. On each node, access the libvirtd configuration file at:

    /etc/libvirt/libvirtd.conf

    Confirm that the following lines are uncommented, as shown:

    unix_sock_group = "libvirt"  =>  unix_sock_group = "libvirt"
    unix_sock_ro_perms = "0777"  =>  unix_sock_ro_perms = "0777"
    unix_sock_rw_perms = "0770"  =>  unix_sock_rw_perms = "0770"

    To check that libvirt is configured and interacting properly with the hypervisor, run the following command on each node:

    su eucalyptus -c "virsh list"

    The output of that command may include error messages (failed to connect to xend), but as long as it includes a listing of all domains (at least Domain-0), the configuration is in order.

    Now start up your Eucalyptus services. On the front-end:

    /etc/init.d/eucalyptus-cloud start
    /etc/init.d/eucalyptus-cc start

    On the node:

    /etc/init.d/eucalyptus-nc start

    At this point you should be ready to proceed with first-time configuration.

    原文:http://open.eucalyptus.com/wiki/EucalyptusInstallationCentos_v2.0

     
  • bixuan 09:58 on 2010 年 11 月 30 日 链接地址 | 回复
    Tags: ,   

    优化页面加载时间 

    It is widely accepted that fast-loading pages improve the user experience. In recent years, many sites have started using AJAX techniques to reduce latency. Rather than round-trip through the server retrieving a completely new page with every click, often the browser can either alter the layout of the page instantly or fetch a small amount of HTML, XML, or javascript from the server and alter the existing page. In either case, this significantly decreases the amount of time between a user click and the browser finishing rendering the new content.

    However, for many sites that reference dozens of external objects, the majority of the page load time is spent in separate HTTP requests for images, javascript, and stylesheets. AJAX probably could help, but speeding up or eliminating these separate HTTP requests might help more, yet there isn’t a common body of knowledge about how to do so.

    While working on optimizing page load times for a high-profile AJAX application, I had a chance to investigate how much I could reduce latency due to external objects. Specifically, I looked into how the HTTP client implementation in common browsers and characteristics of common Internet connections affect page load time for pages with many small objects.

    I found a few things to be interesting:

    • IE, Firefox, and Safari ship with HTTP pipelining disabled by default; Opera is the only browser I know of that enables it. No pipelining means each request has to be answered and its connection freed up before the next request can be sent. This incurs average extra latency of the round-trip (ping) time to the user divided by the number of connections allowed. Or if your server has HTTP keepalives disabled, doing another TCP three-way handshake adds another round trip, doubling this latency.
    • By default, IE allows only two outstanding connections per hostname when talking to HTTP/1.1 servers or eight-ish outstanding connections total. Firefox has similar limits. Using up to four hostnames instead of one will give you more connections. (IP addresses don’t matter; the hostnames can all point to the same IP.)
    • Most DSL or cable Internet connections have asymmetric bandwidth, at rates like 1.5Mbit down/128Kbit up, 6Mbit down/512Kbit up, etc. Ratios of download to upload bandwidth are commonly in the 5:1 to 20:1 range. This means that for your users, a request takes the same amount of time to send as it takes to receive an object of 5 to 20 times the request size. Requests are commonly around 500 bytes, so this should significantly impact objects that are smaller than maybe 2.5k to 10k. This means that serving small objects might mean the page load is bottlenecked on the users’ upload bandwidth, as strange as that may sound.

    Using these, I came up with a model to guesstimate the effective bandwidth of users of various flavors of network connections when loading various object sizes. It assumes that each HTTP request is 500 bytes and that the HTTP reply includes 500 bytes of headers in addition to the object requested. It is simplistic and only covers connection limits and asymmetric bandwidth, and doesn’t account for the TCP handshake of the first request of a persistent (keepalive) connection, which is amortized when requesting many objects from the same connection. Note that this is best-case effective bandwidth and doesn’t include other limitations like TCP slow-start, packet loss, etc. The results are interesting enough to suggest avenues of exploration but are no substitute for actually measuring the difference with real browsers.

    To show the effect of keepalives and multiple hostnames, I simulated a user on net offering 1.5Mbit down/384Kbit up who is 100ms away with 0% packet loss. This roughly corresponds to medium-speed ADSL on the other side of the U.S. from your servers. Shown here is the effective bandwidth while loading a page with many objects of a given size, with effective bandwidth defined as total object bytes received divided by the time to receive them:

    [1.5megabit 100ms graph]

    Interesting things to note:

    • For objects of relatively small size (the left-hand portion of the graph), you can see from the empty space above the plotted line how little of the user’s downstream bandwidth is being used, even though the browser is requesting objects as fast as it can. This user has to be requesting objects larger than 100k before he’s mostly filling his available downstream bandwidth.
    • For objects under roughly 8k in size, you can double his effective bandwidth by turning keepalives on and spreading the requests over four hostnames. This is a huge win.
    • If the user were to enable pipelining in his browser (such as setting Firefox’s network.http.pipelining in about:config), the number of hostnames we use wouldn’t matter, and he’d make even more effective use of his available bandwidth. But we can’t control that server-side.

    Perhaps more clearly, the following is a graph of how much faster pages could load for an assortment of common access speeds and latencies with many external objects spread over four hostnames and keepalives enabled. Baseline (0%) is one hostname and keepalives disabled.

    [Speedup of 4 hostnames and keepalives on]

    Interesting things from that graph:

    • If you load many objects smaller than 10k, both local users and ones on the other side of the world could see substantial improvement from enabling keepalives and spreading requests over 4 hostnames.
    • There is a much greater improvement for users further away.
    • This will matter more as access speeds increase. The user on 100meg ethernet only 20ms away from the server saw the biggest improvement.

    One more thing I examined was the effect of request size on effective bandwidth. The above graphs assumed 500 byte requests and 500 bytes of reply headers in addition to the object contents. How does changing that affect performance of our 1.5Mbit down/384Kbit up and 100ms away user, assuming we’re already using four hostnames and keepalives?

    [Effective bandwidth at various request sizes]

    This shows that at small object sizes, we’re bottlenecked on the upstream bandwidth. The browser sending larger requests (such as ones laden with lots of cookies) seems to slow the requests down by 40% worst-case for this user.

    As I’ve said, these graphs are based on a simulation and don’t account for a number of real-world factors. But I’ve unscientifically verified the results with real browsers on real net and believe them to be a useful gauge. I’d like to find the time and resources to reproduce these using real data collected from real browsers over a range of object sizes, access speeds, and latencies.

    Measuring the effective bandwidth of your users

    You can measure the effective bandwidth of your users on your site relatively easily, and if the effective bandwidth of users viewing your pages is substantially below their available downstream bandwidth, it might be worth attempting to improve this.

    Before giving the browser any external object references (<img src=”…”>, <link rel=”stylesheet” href=”…”>, <script src=”…”>, etc), record the current time. After the page load is done, subtract the time you began, and include that time in the URL of an image you reference off of your server.

    Sample javascript implementing this:

    <html>
    <head>
    <title>...</title>
    <script type="text/javascript">
    <!--
    var began_loading = (new Date()).getTime();
    
    function done_loading() {
     (new Image()).src = '/timer.gif?u=' + self.location + '&t=' +
      (((new Date()).getTime() - began_loading) / 1000);
    }
    // -->
    </script>
    <!--
    Reference any external javascript or stylesheets after the above block.
    // -->
    </head>
    <body onload="done_loading()">
    <!--
    Put your normal page content here.
    // -->
    </body>
    </html>

    This will produce web log entries of the form:

    10.1.2.3 - - [28/Oct/2006:13:47:45 -0700] "GET /timer.gif?u=http://example.com/page.html&t=0.971 HTTP/1.1" 200 49 ...

    in this case, showing that for this user, loading the rest of http://example.com/page.html took 0.971 seconds. And if you know that the combined size of everything referenced from that page is 57842 bytes, 57842 bytes * 8 bits per byte / 0.971 seconds = 476556 bits per second effective bandwidth for that page load. If this user should be getting 1.5Mbit downstream bandwidth, there is substantial room for improvement.

    Tips to reduce your page load time

    After you gather some page-load times and effective bandwidth for real users all over the world, you can experiment with changes that will improve those times. Measure the difference and keep any that offer a substantial improvement.

    Try some of the following:

    • Turn on HTTP keepalives for external objects. Otherwise you add an extra round-trip to do another TCP three-way handshake and slow-start for every HTTP request. If you are worried about hitting global server connection limits, set the keepalive timeout to something short, like 5-10 seconds. Also look into serving your static content from a different webserver than your dynamic content. Having thousands of connections open to a stripped down static file webserver can happen in like 10 megs of RAM total, whereas your main webserver might easily eat 10 megs of RAM per connection.
    • Load fewer external objects. Due to request overhead, one bigger file just loads faster than two smaller ones half its size. Figure out how to globally reference the same one or two javascript files and one or two external stylesheets instead of many; if you have more, try preprocessing them when you publish them. If your UI uses dozens of tiny GIFs all over the place, consider switching to a much cleaner CSS-based design which probably won’t need so many images. Or load all of your common UI images in one request using a technique called “CSS sprites“.
    • If your users regularly load a dozen or more uncached or uncacheable objects per page, consider evenly spreading those objects over four hostnames. This usually means your users can have 4x as many outstanding connections to you. Without HTTP pipelining, this results in their average request latency dropping to about 1/4 of what it was before.

      When you generate a page, evenly spreading your images over four hostnames is most easily done with a hash function, like MD5. Rather than having all <img> tags load objects from http://static.example.com/, create four hostnames (e.g. static0.example.com, static1.example.com, static2.example.com, static3.example.com) and use two bits from an MD5 of the image path to choose which of the four hosts you reference in the <img> tag. Make sure all pages consistently reference the same hostname for the same image URL, or you’ll end up defeating caching.

      Beware that each additional hostname adds the overhead of an extra DNS lookup and an extra TCP three-way handshake. If your users have pipelining enabled or a given page loads fewer than around a dozen objects, they will see no benefit from the increased concurrency and the site may actually load more slowly. The benefits only become apparent on pages with larger numbers of objects. Be sure to measure the difference seen by your users if you implement this.

    • Possibly the best thing you can do to speed up pages for repeat visitors is to allow static images, stylesheets, and javascript to be unconditionally cached by the browser. This won’t help the first page load for a new user, but can substantially speed up subsequent ones.

      Set an Expires header on everything you can, with a date days or even months into the future. This tells the browser it is okay to not revalidate on every request, which can add latency of at least one round-trip per object per page load for no reason.

      Instead of relying on the browser to revalidate its cache, if you change an object, change its URL. One simple way to do this for static objects if you have staged pushes is to have the push process create a new directory named by the build number, and teach your site to always reference objects out of the current build’s base URL. (Instead of <img src=”http://example.com/logo.gif”> you’d use <img src=”http://example.com/build/1234/logo.gif”>. When you do another build next week, all references change to <img src=”http://example.com/build/1235/logo.gif”>.) This also nicely solves problems with browsers sometimes caching things longer than they should — since the URL changed, they think it is a completely different object.

      If you conditionally gzip HTML, javascript, or CSS, you probably want to add a “Cache-Control: private” if you set an Expires header. This will prevent problems with caching by proxies that won’t understand that your gzipped content can’t be served to everyone. (The Vary header was designed to do this more elegantly, but you can’t use it because of IE brokenness.)

      For anything where you always serve the exact same content when given the same URL (e.g. static images), add “Cache-Control: public” to give proxies explicit permission to cache the result and serve it to different users. If a caching proxy local to the user has the content, it is likely to have much less latency than you; why not let it serve your static objects if it has them?

      Avoid the use of query params in image URLs, etc. At least the Squid cache refuses to cache any URL containing a question mark by default. I’ve heard rumors that other things won’t cache those URLs at all, but I don’t have more information.

    • On pages where your users are often sent the exact same content over and over, such as your home page or RSS feeds, implementing conditional GETs can substantially improve response time and save server load and bandwidth in cases where the page hasn’t changed.

      When serving a static files (including HTML) off of disk, most webservers will generate Last-Modified and/or ETag reply headers for you and make use of the correspondingIf-Modified-Since and/or If-None-Match mechanisms on requests. But as soon as you add server-side includes, dynamic templating, or have code generating your content as it is served, you are usually on your own to implement these.

      The idea is pretty simple: When you generate a page, you give the browser a little extra information about exactly what was on the page you sent. When the browser asks for the same page again, it gives you this information back. If it matches what you were going to send, you know that the browser already has a copy and send a much smaller 304 (Not Modified) reply instead of the contents of the page again. And if you are clever about what information you include in an ETag, you can usually skip the most expensive database queries that would’ve gone into generating the page.

    • Minimize HTTP request size. Often cookies are set domain-wide, which means they are also unnecessarily sent by the browser with every image request from within that domain. What might’ve been a 400 byte request for an image could easily turn into 1000 bytes or more once you add the cookie headers. If you have a lot of uncached or uncacheable objects per page and big, domain-wide cookies, consider using a separate domain to host static content, and be sure to never set any cookies in it.
    • Minimize HTTP response size by enabling gzip compression for HTML and XML for browsers that support it. For example, the 17k document you are reading takes 90ms of the full downstream bandwidth of a user on 1.5Mbit DSL. Or it will take 37ms when compressed to 6.8k. That’s 53ms off of the full page load time for a simple change. If your HTML is bigger and more redundant, you’ll see an even greater improvement.

      If you are brave, you could also try to figure out which set of browsers will handle compressed Javascript properly. (Hint: IE4 through IE6 asks for its javascript compressed, then breaks badly if you send it that way.) Or look into Javascript obfuscators that strip out whitespace, comments, etc and usually get it down to 1/3 to 1/2 its original size.

    • Consider locating your small objects (or a mirror or cache of them) closer to your users in terms of network latency. For larger sites with a global reach, either use a commercial Content Delivery Network, or add a colo within 50ms of 80% of your users and use one of the many available methods for routing user requests to your colo nearest them.
    • Regularly use your site from a realistic net connection. Convincing the web developers on my project to use a “slow proxy” that simulates bad DSL in New Zealand (768Kbit down, 128Kbit up, 250ms RTT, 1% packet loss) rather than the gig ethernet a few milliseconds from the servers in the U.S. was a huge win. We found and fixed a number of usability and functional problems very quickly.

      To implement the slow proxy, I used the netem and HTB kernel modules available in the Linux 2.6 kernel, both of which are set up with the tc command line tool. These offer the most accurate simulation I could find, but are definitely not for the faint of heart. I’ve not used them, but supposedly Tamper Data for Firefox, Fiddler for Windows, and Charles for OSX can all rate-limit and are probably easier to set up, but they may not simulate latency properly.

    • Use Firebug for Firefox from a realistic net connection to see a graphical timeline of what it is doing during a page load. This shows where Firefox has to wait for one HTTP request to complete before starting the next one and how page load time increases with each object loaded. YSlow extends Firebug to offer tips on how to improve your site’s performance.

      The Safari team offers a tip on a hidden feature in their browser that offers some timing data too.

      Or if you are familiar with the HTTP protocol and TCP/IP at the packet level, you can watch what is going on using tcpdumpngrep, or ethereal. These tools are indispensable for all sorts of network debugging.

    • Try benchmarking common pages on your site from a local network with ab, which comes with the Apache webserver. If your server is taking longer than 5 or 10 milliseconds to generate a page, you should make sure you have a good understanding of where it is spending its time.

      If your latencies are high and your webserver process (or CGI if you are using that) is eating a lot of CPU during this test, it is often a result of using a scripting language that needs to recompile your scripts with every request. Software like eAccelerator for PHP, mod_perl for perl, mod_python for python, etc can cache your scripts in a compiled state, dramatically speeding up your site. Beyond that, look at finding a profiler for your language that can tell you where you are spending your CPU. If you improve that, your pages will load faster and you’ll be able to handle more traffic with fewer machines.

      If your site relies on doing a lot of database work or some other time-consuming task to generate the page, consider adding server-side caching of the slow operation. Most people start with writing a cache to local memory or local disk, but that starts to fall down if you expand to more than a few web server machines. Look into usingmemcached, which essentially creates an extremely fast shared cache that’s the combined size of the spare RAM you give it off of all of your machines. It has clients available in most common languages.

    • (Optional) Petition browser vendors to turn on HTTP pipelining by default on new browsers. Doing so will remove some of the need for these tricks and make much of the web feel much faster for the average user. (Firefox has this disabled supposedly because some proxies, some load balancers, and some versions of IIS choke on pipelined requests. But Opera has found sufficient workarounds to enable pipelining by default. Why can’t other browsers do similarly?)

    The above list covers improving the speed of communication between browser and server and can be applied generally to many sites, regardless of what web server software they use or what language the code behind your site is written in. There is, unfortunately, a lot that isn’t covered.

    While the tips above are intended to improve your page load times, a side benefit of many of them is a reduction in server bandwidth and CPU needed for the average page view. Reducing your costs while improving your user experience seems it should be worth spending some time on.

    From: http://www.die.net/musings/page_load_time/

     
  • bixuan 23:15 on 2010 年 11 月 11 日 链接地址 | 回复
    Tags: , 环境变量   

    Linux下添加环境变量与GCC编译器添加INCLUDE与LIB环境变量 

    对所有用户有效在/etc/profile增加以下内容。只对当前用户有效在HOME目录下的

    .bashrc或.bash_profile(redhat)或者.profile(suse)里增加下面的内容:

    (注意:等号前面不要加空格,否则可能出现 command not found)

    1. 在PATH中找到可执行文件程 序的路径:
      export PATH =$PATH:$HOME/bin
    2. gcc找到头文件的路径:
      export C_INCLUDE_PATH=$C_INCLUDE_PATH:/opt/app/include
    3. g++找到头文件的路径:
      export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/include/libxml2:/opt/app/include
    4. 找到动态链接库的路径:
      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/app/lib
    5. 找到静态库的路径:
      export LIBRARY_PATH=$LIBRARY_PATH:/opt/app/lib
     
  • bixuan 01:46 on 2010 年 09 月 11 日 链接地址 | 回复
    Tags: , ttserver   

    搞了个ttserver和memcached信息汇总表 

    ttserver和memcached多了,就不是很好管理了,今天花了点时间,整了个统一的报表页面,这样方便查看:

    tm-stats-011

    程序是php开发的,比较简单,页面也挺丑陋的,如果有朋友想完善的,可以与我联系(gtalk/email: bixuan#gmail.com,请将#换成@),希望能对朋友有所帮助:)

     
  • bixuan 14:41 on 2010 年 05 月 11 日 链接地址 | 回复
    Tags: , , nmon,   

    非常棒的系统监控工具nmon 

    nmon for Linux 官方:http://nmon.sourceforge.net/pmwiki.php

    下面是收集的资料:nmon工具介绍 (监控优化-操作系统监控)

     
  • bixuan 23:07 on 2010 年 04 月 10 日 链接地址 | 回复
    Tags: disk, iostat, , utilization   

    iostat and disk utilization monitoring nirvana 

    Posted by Bhavin Turakhia

    In my neverending quest of performance monitoring, I have been constantly trying to find better ways to monitor disk utilization on a server. At Directi we use the usual medley of tools at our disposal viz. iostat, sar, sysstat. I made serious progress last week, when Dushyanth from my team shared this post on IO Monitoring on Linux, by the folks over at Pythian, on our internal mailing list. Here are my notes on the subject.

    Performance measurement and Capacity planning are a science. It is common practice at Directi to attempt to determine what the performance bottlenecks in any given application are. A usual generalization is to determine whether an application is cpu-bound / memory-bound / IO bound.

    IO bound applications end up wasting cpu cycles, especially incase of Disk IO, since most programming languages do not have Async Disk IO support today. Therefore in order to maximize performance and optimize resource utilization one should try and reduce iowait time of a CPU and tweak a deployment to make an application cpu-bound.

    When your CPU seems to be spending a lot of time on iowait you need to make some changes. However an iowait can occur either because there is a lot of Disk/Network IO taking place, or because the disk subsystem is saturated and cannot provide greater throughput. iostat allows you to determine which one it is. A regular iostat output consists of the following fields -

    # iostat -dkx 60

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
    dm-0 0.00 0.00 611.40 414.23 20286.60 1656.93 42.79 17.50 17.33 0.96 98.57

    Explanation of the above fields:

    * Device: the block device whose performance counters are being reported
    * r/s and w/s: number of read and write requests issued per second to the device (in this case 611 and 414)
    * rsec/s and wsec/s – number of sectors read/written per second
    * rkB/s and wkB/s – number of kilobytes read/written per second
    * avgrq-sz – average number of sectors per request (for both reads and writes). ie (rsec + wsec) / (r + w)
    * avgqu-sz – average queue length in the monitoring interval (in this case 42.79)
    * await – average time that each IO Request took to complete. This includes the time that the request was waiting in the queue and the time that the request took to be serviced by the device
    * svctim – average time each IO request took to complete during the monitoring interval
    * Note: await includes svctim. Infact await (average time taken for each IO Request to complete) = the average time that each request was in queue (lets call it queuetime) PLUS the average time each request took to process (svctim)
    * %util: This number depicts the percentage of time that the device spent in servicing requests. This can be calculated with the above values. In the above example the total number of reads and writes issued per second is 611 + 414 => 1025. Each request takes 0.96 ms to process. Therefore 1025 requests would take 1025 x 0.96 => 984 ms to process. So out of the 1 second that these requests were sent to the device in, 984 ms were taken to process the requests. This means the device utilization is 984/1000 * 100 => ~98.4%. As you can see in the above iostat output the %util does show ~ 98.5%

    Interpreting iostat values

    Lets take the above example

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
    dm-0 0.00 0.00 611.40 414.23 20286.60 1656.93 42.79 17.50 17.33 0.96 98.57

    * avg time that each request spent in queue (qtime) = await – svctime = 17.33 – 0.96 => 16.37 ms
    * avg time tha each request spent being serviced = 0.96 ms
    * so averagely each IO request spent 17.33ms to et processed of which 16.37 ms were spent just waiting in queue
    * %util can be calculated as (r/s + w/s) * svctim / 1000ms * 100 => 1025*0.96/1000 * 100 => 98.5%
    * This simple means that in a 1 second interval, 1025 requests were sent to disk, each of which took 0.96ms for the disk to process resulting in 984 ms of disk utilization time in a period of 1 s (or 1000 ms). This means the disk is greater than 98% utilized

    On this disk subsystem, it is clear that the disk cannot process more IO requests than what it is getting

    Lets take another example -

    Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
    sdb 6.33 139.07 46.30 19.53 526.27 634.40 35.26 0.54 8.17 3.30 21.74

    * avg time that each request spent in queue (qtime) = await – svctime = 8.17 – 3.30 => 4.87 ms
    * avg time tha each request spent being serviced = 3.30 ms
    * so averagely each IO request spent 8.17 ms to et processed of which 4.87 ms (a little more than half) were spent waiting in queue
    * %util can be calculated as (r/s + w/s) * svctim / 1000ms * 100 => 65.83 * 3.3/1000 * 100 => 21.72%
    * This simple means that in a 1 second interval, 65 requests were sent to disk, each of which took 3.30ms for the disk to process resulting in 217 ms of disk utilization time in a period of 1 s (or 1000 ms). This means the disk is around 21.7 % utilized

    On this disk subsystem, it is clear that the disk is not fully utilized. While due to the nature of the requests, averagely requests are spending half their time in queue, that is not so bad. This disk subsystem is capable of greater throughput.

    Notes

    On every Linux box the following should be graphed at 5 minute averages

    * %util: When this figure is consistently approaching above 80% you will need to take any of the following actions -
    o increasing RAM so dependence on disk reduces
    o increasing RAID controller cache so disk dependence decreases
    o increasing number of disks so disk throughput increases (more spindles working parallely)
    o horizontal partitioning
    * (await-svctim)/await*100: The percentage of time that IO operations spent waiting in queue in comparison to actually being serviced. If this figure goes above 50% then each IO request is spending more time waiting in queue than being processed. If this ratio skews heavily upwards (in the >75% range) you know that your disk subsystem is not being able to keep up with the IO requests and most IO requests are spending a lot of time waiting in queue. In this scenario you will again need to take any of the actions above
    * %iowait: This number shows the % of time the CPU is wasting in waiting for IO. A part of this number can result from network IO, which can be avoided by using an Async IO library. The rest of it is simply an indication of how IO-bound your application is. You can reduce this number by ensuring that disk IO operations take less time, more data is available in RAM, increasing disk throughput by increasing number of disks in a RAID array, using SSD (Check my post on Solid State drives vs Hard Drives) for portions of the data or all of the data etc

    From:http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/

     
  • bixuan 00:36 on 2010 年 03 月 10 日 链接地址 | 回复
    Tags: push, 报警,   

    使用pushme.to做手机报警 

    下个手机客户端,然后注册个pushme.to的帐号,比如:ourlinux,然后就可以通过pushme的接口进行发消息了。

    HTML代码如下:

    linux下简单的执行如下:

    curl -o /dev/null -d “nickname=ourlinux&signature=OWL&submit=%20OK%20&&message=test” http://pushme.to/ourlinux/

    记得将ourlinux改成自己的接收的账号!

    这样就可以将报警的信息通过pushme来发送了!

    其他的不说了,很简单,一看就明白了!

    感谢joe推荐使用pushme!

     
    • nonamexz 09:46 on 2010 年 03 月 10 日 链接地址

      我还是更喜欢用139邮箱的短信到达通知。

  • bixuan 09:53 on 2010 年 02 月 26 日 链接地址 | 回复
    Tags: url, 域名   

    当你输入一个网址的时候,实际会发生什么? 

    From:http://article.yeeyan.org/view/54517/91367

    翻译:litfresh | 2010-02-24 16:24:59 | 阅读2097 | 来源

    作为一个软件开发者,你一定会对网络应用如何工作有一个完整的层次化的认知,同样这里也包括这些应用所用到的技术:像浏览器,HTTP,HTML,网络服务器,需求处理等等。

    本文将更深入的研究当你输入一个网址的时候,后台到底发生了一件件什么样的事~

    1. 首先嘛,你得在浏览器里输入要网址:

    image

    2. 浏览器查找域名的IP地址

    image

    导航的第一步是通过访问的域名找出其IP地址。DNS查找过程如下:

    • 浏览器缓存 – 浏览器会缓存DNS记录一段时间。 有趣的是,操作系统没有告诉浏览器储存DNS记录的时间,这样不同浏览器会储存个自固定的一个时间(2分钟到30分钟不等)。
    • 系统缓存 – 如果在浏览器缓存里没有找到需要的记录,浏览器会做一个系统调用(windows里是gethostbyname)。这样便可获得系统缓存中的记录。
    • 路由器缓存 – 接着,前面的查询请求发向路由器,它一般会有自己的DNS缓存。
    • ISP DNS 缓存 – 接下来要check的就是ISP缓存DNS的服务器。在这一般都能找到相应的缓存记录。
    • 递归搜索 – 你的ISP的DNS服务器从跟域名服务器开始进行递归搜索,从.com顶级域名服务器到Facebook的域名服务器。一般DNS服务器的缓存中会有.com域名服务器中的域名,所以到顶级服务器的匹配过程不是那么必要了。

    DNS递归查找如下图所示:

    500px-An_example_of_theoretical_DNS_recursion_svg

    DNS有一点令人担忧,这就是像wikipedia.org 或者 facebook.com这样的整个域名看上去只是对应一个单独的IP地址。还好,有几种方法可以消除这个瓶颈:

    • 循环 DNS 是DNS查找时返回多个IP时的解决方案。举例来说,Facebook.com实际上就对应了四个IP地址。
    • 负载平衡器 是以一个特定IP地址进行侦听并将网络请求转发到集群服务器上的硬件设备。 一些大型的站点一般都会使用这种昂贵的高性能负载平衡器。
    • 地理 DNS 根据用户所处的地理位置,通过把域名映射到多个不同的IP地址提高可扩展性。这样不同的服务器不能够更新同步状态,但映射静态内容的话非常好。
    • Anycast 是一个IP地址映射多个物理主机的路由技术。 美中不足,Anycast与TCP协议适应的不是很好,所以很少应用在那些方案中。

    大多数DNS服务器使用Anycast来获得高效低延迟的DNS查找。

    3. 浏览器给web服务器发送一个HTTP请求

    image

    因为像Facebook主页这样的动态页面,打开后在浏览器缓存中很快甚至马上就会过期,毫无疑问他们不能从中读取。

    所以,浏览器将把一下请求发送到Facebook所在的服务器:

    GET http://facebook.com/ HTTP/1.1
     Accept: application/x-ms-application, image/jpeg, application/xaml+xml, [...]
     User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; [...]
     Accept-Encoding: gzip, deflate
     Connection: Keep-Alive
     Host: facebook.com
     Cookie: datr=1265876274-[...]; locale=en_US; lsd=WW[...]; c_user=2101[...]

    GET 这个请求定义了要读取的URL: “http://facebook.com/”。 浏览器自身定义 (User-Agent 头), 和它希望接受什么类型的相应 (Accept and Accept-Encoding 头). Connection头要求服务器为了后边的请求不要关闭TCP连接。

    请求中也包含浏览器存储的该域名的cookies。可能你已经知道,在不同页面请求当中,cookies是与跟踪一个网站状态相匹配的键值。这样cookies会存储登录用户名,服务器分配的密码和一些用户设置等。Cookies会以文本文档形式存储在客户机里,每次请求时发送给服务器。

    用来看原始HTTP请求及其相应的工具很多。作者比较喜欢使用fiddler,当然也有像FireBug这样其他的工具。这些软件在网站优化时会帮上很大忙。

    除了获取请求,还有一种是发送请求,它常在提交表单用到。发送请求通过URL传递其参数(e.g.: http://robozzle.com/puzzle.aspx?id=85)。发送请求在请求正文头之后发送其参数。

    像“http://facebook.com/”中的斜杠是至关重要的。这种情况下,浏览器能安全的添加斜杠。而像“http://example.com/folderOrFile”这样的地址,因为浏览器不清楚folderOrFile到底是文件夹还是文件,所以不能自动添加斜杠。这时,浏览器就不加斜杠直接访问地址,服务器会响应一个重定向,结果造成一次不必要的握手。

    4. facebook服务的永久重定向响应

    image

    图中所示为Facebook服务器发回给浏览器的响应:

    HTTP/1.1 301 Moved Permanently
     Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0,
     pre-check=0
     Expires: Sat, 01 Jan 2000 00:00:00 GMT
     Location: http://www.facebook.com/
     P3P: CP="DSP LAW"
     Pragma: no-cache
     Set-Cookie: made_write_conn=deleted; expires=Thu, 12-Feb-2009 05:09:50 GMT;
     path=/; domain=.facebook.com; httponly
     Content-Type: text/html; charset=utf-8
     X-Cnection: close
     Date: Fri, 12 Feb 2010 05:09:51 GMT
     Content-Length: 0

    服务器给浏览器响应一个301永久重定向响应,这样浏览器就会访问“http://www.facebook.com/” 而非“http://facebook.com/”。

    为什么服务器一定要重定向而不是直接发会用户想看的网页内容呢?这个问题有好多有意思的答案。

    其中一个原因跟搜索引擎排名有关。你看,如果一个页面有两个地址,就像http://www.igoro.com/ 和http://igoro.com/,搜索引擎会认为它们是两个网站,结果造成每一个的搜索链接都减少从而降低排名。而搜索引擎知道301永久重定向是什么意思,这样就会把访问带www的和不带www的地址归到同一个网站排名下。

    还有一个是用不同的地址会造成缓存友好性变差。当一个页面有好几个名字时,它可能会在缓存里出现好几次。

    5. 浏览器跟踪重定向地址

    image

    现在,浏览器知道了“http://www.facebook.com/”才是要访问的正确地址,所以它会发送另一个获取请求:

    GET http://www.facebook.com/ HTTP/1.1
     Accept: application/x-ms-application, image/jpeg, application/xaml+xml, [...]
     Accept-Language: en-US
     User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; [...]
     Accept-Encoding: gzip, deflate
     Connection: Keep-Alive
     Cookie: lsd=XW[...]; c_user=21[...]; x-referer=[...]
     Host: http://www.facebook.com

    头信息以之前请求中的意义相同。

    6. 服务器“处理”请求

    image

    服务器接收到获取请求,然后处理并返回一个响应。

    这表面上看起来是一个顺向的任务,但其实这中间发生了很多有意思的东西- 就像作者博客这样简单的网站,何况像facebook那样访问量大的网站呢!

    • Web 服务器软件
      web服务器软件(像IIS和阿帕奇)接收到HTTP请求,然后确定执行什么请求处理来处理它。请求处理就是一个能够读懂请求并且能生成HTML来进行响应的程序(像ASP.NET,PHP,RUBY…)。举个最简单的例子,需求处理可以以映射网站地址结构的文件层次存储。像http://example.com/folder1/page1.aspx这个地址会映射/httpdocs/folder1/page1.aspx这个文件。web服务器软件可以设置成为地址人工的对应请求处理,这样page1.aspx的发布地址就可以是http://example.com/folder1/page1。
    • 请求处理
      请求处理阅读请求及它的参数和cookies。它会读取也可能更新一些数据,并讲数据存储在服务器上。然后,需求处理会生成一个HTML响应。

    所有动态网站都面临一个有意思的难点 -如何存储数据。小网站一半都会有一个SQL数据库来存储数据,存储大量数据和/或访问量大的网站不得不找一些办法把数据库分配到多台机器上。解决方案有:sharding (基于主键值讲数据表分散到多个数据库中),复制,利用弱语义一致性的简化数据库。

    委托工作给批处理是一个廉价保持数据更新的技术。举例来讲,Fackbook得及时更新新闻feed,但数据支持下的“你可能认识的人”功能只需要每晚更新(作者猜测是这样的,改功能如何完善不得而知)。批处理作业更新会导致一些不太重要的数据陈旧,但能使数据更新耕作更快更简洁。

    7. 服务器发回一个HTML响应

    image

    图中为服务器生成并返回的响应:

    HTTP/1.1 200 OK
     Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0,
     pre-check=0
     Expires: Sat, 01 Jan 2000 00:00:00 GMT
     P3P: CP="DSP LAW"
     Pragma: no-cache
     Content-Encoding: gzip
     Content-Type: text/html; charset=utf-8
     X-Cnection: close
     Transfer-Encoding: chunked
     Date: Fri, 12 Feb 2010 09:05:55 GMT
    
     2b3��������T�n�@����[...]

    整个响应大小为35kB,其中大部分在整理后以blob类型传输。

    内容编码头告诉浏览器整个响应体用gzip算法进行压缩。解压blob块后,你可以看到如下期望的HTML:

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"   
     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
     <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"
     lang="en" id="facebook" class=" no_js">
     <head>
     <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
     <meta http-equiv="Content-language" content="en" />
     ...

    关于压缩,头信息说明了是否缓存这个页面,如果缓存的话如何去做,有什么cookies要去设置(前面这个响应里没有这点)和隐私信息等等。

    请注意报头中把Content-type设置为“text/html”。报头让浏览器将该响应内容以HTML形式呈现,而不是以文件形式下载它。浏览器会根据报头信息决定如何解释该响应,不过同时也会考虑像URL扩展内容等其他因素。

    8. 浏览器开始显示HTML

    在浏览器没有完整接受全部HTML文档时,它就已经开始显示这个页面了:

    image

    9. 浏览器发送获取嵌入在HTML中的对象

    image

    在浏览器显示HTML时,它会注意到需要获取其他地址内容的标签。这时,浏览器会发送一个获取请求来重新获得这些文件。

    下面是几个我们访问facebook.com时需要重获取的几个URL:

    这些地址都要经历一个和HTML读取类似的过程。所以浏览器会在DNS中查找这些域名,发送请求,重定向等等…

    但不像动态页面那样,静态文件会允许浏览器对其进行缓存。有的文件可能会不需要与服务器通讯,而从缓存中直接读取。服务器的响应中包含了静态文件保存的期限信息,所以浏览器知道要把它们缓存多长时间。还有,每个响应都可能包含像版本号一样工作的ETag头(被请求变量的实体值),如果浏览器观察到文件的版本ETag信息已经存在,就马上停止这个文件的传输。

    试着猜猜看“fbcdn.net”在地址中代表什么?聪明的答案是”Facebook内容分发网络”。Facebook利用内容分发网络(CDN)分发像图片,CSS表和JavaScript文件这些静态文件。所以,这些文件会在全球很多CDN的数据中心中留下备份。

    静态内容往往代表站点的带宽大小,也能通过CDN轻松的复制。通常网站会使用第三方的CDN。例如,Facebook的静态文件由最大的CDN提供商Akamai来托管。

    举例来讲,当你试着ping static.ak.fbcdn.net的时候,可能会从某个akamai.net服务器上获得响应。有意思的是,当你同样再ping一次的时候,响应的服务器可能就不一样,这说明幕后的负载平衡开始起作用了。

    10. 浏览器发送异步(AJAX)请求

    image

    在Web 2.0伟大精神的指引下,页面显示完成后客户端仍与服务器端保持着联系。

    以Facebook聊天功能为例,它会持续与服务器保持联系来及时更新你那些亮亮灰灰的好友状态。为了更新这些头像亮着的好友状态,在浏览器中执行的JavaScript代码会给服务器发送异步请求。这个异步请求发送给特定的地址,它是一个按照程式构造的获取或发送请求。还是在Facebook这个例子中,客户端发送给http://www.facebook.com/ajax/chat/buddy_list.php一个发布请求来获取你好友里哪个在线的状态信息。

    提起这个模式,就必须要讲讲”AJAX”– “异步JavaScript 和 XML”,虽然服务器为什么用XML格式来进行响应也没有个一清二白的原因。再举个例子吧,对于异步请求,Facebook会返回一些JavaScript的代码片段。

    除了其他,fiddler这个工具能够让你看到浏览器发送的异步请求。事实上,你不仅可以被动的做为这些请求的看客,还能主动出击修改和重新发送它们。AJAX请求这么容易被蒙,可着实让那些计分的在线游戏开发者们郁闷的了。(当然,可别那样骗人家~)

    Facebook聊天功能提供了关于AJAX一个有意思的问题案例:把数据从服务器端推送到客户端。因为HTTP是一个请求-响应协议,所以聊天服务器不能把新消息发给客户。取而代之的是客户端不得不隔几秒就轮询下服务器端看自己有没有新消息。

    这些情况发生时长轮询是个减轻服务器负载挺有趣的技术。如果当被轮询时服务器没有新消息,它就不理这个客户端。而当尚未超时的情况下收到了该客户的新消息,服务器就会找到未完成的请求,把新消息做为响应返回给客户端。

    总结一下

    希望看了本文,你能明白不同的网络模块是如何协同工作的

     
c
写新的
j
下一篇文章/下一个回复
k
前一篇文章/以前的回复
r
回复
e
编辑
o
显示/隐藏 回复
t
回到顶部
l
go to login
h
show/hide help
esc
取消