1：查看CPU負載--mpstat

mpstat -P ALL [internal [count]]

參數的含義如下：

-P ALL 表示監控所有CPU

internal 相鄰的兩次采樣的間隔時間

count 采樣的次數

mpstat命令從/proc/stat獲得數據輸出

輸出的含義如下：

CPU 處理器ID

user 在internal時間段里，用戶態的CPU時間（%） ，不包含 nice值為負 進程 ?usr/?total*100

nice 在internal時間段里，nice值為負進程的CPU時間（%） ?nice/?total*100

system 在internal時間段里，核心時間（%） ?system/?total*100

iowait 在internal時間段里，硬盤IO等待時間（%） ?iowait/?total*100

irq 在internal時間段里，軟中斷時間（%） ?irq/?total*100

soft 在internal時間段里，軟中斷時間（%） ?softirq/?total*100

idle 在internal時間段里，CPU除去等待磁盤IO操作外的因為任何原因而空閑的時間閑置時間 （%） ?idle/?total*100

intr/s 在internal時間段里，每秒CPU接收的中斷的次數 ?intr/?total*100

CPU總的工作時間total_cur=user+system+nice+idle+iowait+irq+softirq

total_pre=pre_user+ pre_system+ pre_nice+ pre_idle+ pre_iowait+ pre_irq+ pre_softirq

user=user_cur – user_pre

total=total_cur-total_pre

其中_cur 表示當前值，_pre表示interval時間前的值。上表中的所有值可取到兩位小數點。

2：查看磁盤io情況及CPU負載--vmstat

usage: vmstat [-V] [-n] [delay [count]]

              -V prints version.

              -n causes the headers not to be reprinted regularly.

              -a print inactive/active page stats.

              -d prints disk statistics

              -D prints disk table

              -p prints disk partition statistics

              -s prints vm table

              -m prints slabinfo

              -S unit size

              delay is the delay between updates in seconds. 

              unit size k:1000 K:1024 m:1000000 M:1048576 (default is K)

              count is the number of updates.

vmstat從/proc/stat獲得數據

輸出的含義如下: 

FIELD DESCRIPTION FOR VM MODE

   Procs

       r: The number of processes waiting for run time.

       b: The number of processes in uninterruptible sleep.

   Memory

       swpd: the amount of virtual memory used.

       free: the amount of idle memory.

       buff: the amount of memory used as buffers.

       cache: the amount of memory used as cache.

       inact: the amount of inactive memory. (-a option)

       active: the amount of active memory. (-a option)

   Swap

       si: Amount of memory swapped in from disk (/s).

       so: Amount of memory swapped to disk (/s).

   IO

       bi: Blocks received from a block device (blocks/s).

       bo: Blocks sent to a block device (blocks/s).

   System

       in: The number of interrupts per second, including the clock.

       cs: The number of context switches per second.

   CPU

       These are percentages of total CPU time.

       us: Time spent running non-kernel code. (user time, including nice time)

       sy: Time spent running kernel code. (system time)

       id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.

       wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

       st: Time spent in involuntary wait. Prior to Linux 2.6.11, shown as zero.

3：查看內存使用情況--free

usage: free [-b|-k|-m|-g] [-l] [-o] [-t] [-s delay] [-c count] [-V]

  -b,-k,-m,-g show output in bytes, KB, MB, or GB

  -l show detailed low and high memory statistics

  -o use old format (no -/+buffers/cache line)

  -t display total for RAM + swap

  -s update every [delay] seconds

  -c update [count] times

  -V display version information and exit

[root@Linux /tmp]# free

            total     used        free       shared    buffers   cached

Mem:       255268    238332      16936         0        85540   126384

-/+ buffers/cache:   26408       228860 

Swap:      265000      0         265000

Mem：表示物理內存統計 

-/+ buffers/cached：表示物理內存的緩存統計 

Swap：表示硬盤上交換分區的使用情況，這里我們不去關心。

系統的總物理內存：255268Kb（256M），但系統當前真正可用的內存b并不是第一行free 標記的 16936Kb，它僅代表未被分配的內存。

第1行  Mem：

total：表示物理內存總量。 

used：表示總計分配給緩存（包含buffers 與cache ）使用的數量，但其中可能部分緩存并未實際使用。 

free：未被分配的內存。 

shared：共享內存，一般系統不會用到，這里也不討論。 

buffers：系統分配但未被使用的buffers 數量。 

cached：系統分配但未被使用的cache 數量。buffer 與cache 的區別見后面。 

total = used + free    

第2行   -/+ buffers/cached：

used：也就是第一行中的used - buffers-cached   也是實際使用的內存總量。 

free：未被使用的buffers 與cache 和未被分配的內存之和，這就是系統當前實際可用內存。

free 2= buffers1 + cached1 + free1   //free2為第二行、buffers1等為第一行

buffer 與cache 的區別

A buffer is something that has yet to be "written" to disk. 

A cache is something that has been "read" from the disk and stored for later use

第3行：

對操作系統來講是Mem的參數.buffers/cached 都是屬于被使用,所以它認為free只有16936.

對應用程序來講是(-/+ buffers/cach).buffers/cached 是等同可用的，因為buffer/cached是為了提高文件讀取的性能，當應用程序需在用到內存的時候，buffer/cached會很快地被回收。

所以從應用程序的角度來說，可用內存=系統free memory+buffers+cached.

swap

swap就是LINUX下的虛擬內存分區,它的作用是在物理內存使用完之后,將磁盤空間(也就是SWAP分區)虛擬成內存來使用.

4：查看網卡情況--sar

詳細見man

4.1：查看網卡流量：sar -n DEV delay count 

服務器網卡最大能承受流量由網卡本身決定，分為10M、10/100自適應、100+以及1G網卡，一般普通服務器用的是百兆，也有用千兆的。

輸出解釋：

IFACE

       Name of the network interface for which statistics are reported.

rxpck/s

       Total number of packets received per second.

txpck/s

       Total number of packets transmitted per second.

rxbyt/s

       Total number of bytes received per second.

txbyt/s

       Total number of bytes transmitted per second.

rxcmp/s

       Number of compressed packets received per second (for cslip etc.).

txcmp/s

       Number of compressed packets transmitted per second.

rxmcst/s

       Number of multicast packets received per second.

4.2：查看網卡失敗情況：sar -n EDEV delay count 

輸出解釋：

IFACE

       Name of the network interface for which statistics are reported.

rxerr/s

       Total number of bad packets received per second.

txerr/s

       Total number of errors that happened per second while transmitting packets.

coll/s

       Number of collisions that happened per second while transmitting packets.

rxdrop/s

       Number of received packets dropped per second because of a lack of space in linux buffers.

txdrop/s

       Number of transmitted packets dropped per second because of a lack of space in linux buffers.

txcarr/s

       Number of carrier-errors that happened per second while transmitting packets.

rxfram/s

       Number of frame alignment errors that happened per second on received packets.

rxfifo/s

       Number of FIFO overrun errors that happened per second on received packets.

txfifo/s

       Number of FIFO overrun errors that happened per second on transmitted packets.

5：定位問題進程--top, ps

top -d delay，詳細見man

ps aux 查看進程詳細信息

ps axf 查看進程樹

6：查看某個進程與文件關系--losf

需要root權限才能看到全部，否則只能看到登錄用戶權限范圍內的內容

lsof -p 77//查看進程號為77的進程打開了哪些文件

lsof -d 4//顯示使用fd為4的進程 

lsof abc.txt//顯示開啟文件abc.txt的進程

lsof -i :22//顯示使用22端口的進程

lsof -i tcp//顯示使用tcp協議的進程

lsof -i tcp:22//顯示使用tcp協議的22端口的進程

lsof +d /tmp//顯示目錄/tmp下被進程打開的文件

lsof +D /tmp//同上，但是會搜索目錄下的目錄，時間較長

lsof -u username//顯示所屬user進程打開的文件

7：查看程序運行情況--strace

usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file]

              [-p pid] ... [-s strsize] [-u username] [-E var=val] ...

              [command [arg ...]]

   or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ...

              [command [arg ...]]

常用選項：

-f：除了跟蹤當前進程外，還跟蹤其子進程。

-c：統計每一系統調用的所執行的時間,次數和出錯的次數等. 

-o file：將輸出信息寫到文件file中，而不是顯示到標準錯誤輸出（stderr）。

-p pid：綁定到一個由pid對應的正在運行的進程。此參數常用來調試后臺進程。

8：查看磁盤使用情況--df

test@wolf:~$ df

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/sda1              3945128   1810428   1934292  49% /

udev                    745568        80    745488   1% /dev

/dev/sda3             12649960   1169412  10837948  10% /usr/local

/dev/sda4             63991676  23179912  37561180  39% /data

9：查看網絡連接情況--netstat

常用：netstat -lpn

選項說明：

 -p, --programs           display PID/Program name for sockets

 -l, --listening          display listening server sockets

 -n, --numeric            don't resolve names

 -a, --all, --listening   display all sockets (default: connected)

Notes:
*. Time, Clocks and the Ordering of Events in a Distributed System" (1978)
    1. The issue is that in a distributed system you cannot tell if event A happened before event B, unless A caused B in some way. Each observer can see events happen in a different order, except for events that cause each other, ie there is only a partial ordering of events in a distributed system.
    2. Lamport defines the "happens before" relationship and operator, and goes on to give an algorithm that provides a total ordering of events in a distributed system, so that each process sees events in the same order as every other process.
    3. Lamport also introduces the concept of a distributed state machine: start a set of deterministic state machines in the same state and then make sure they process the same messages in the same order.
    4. Each machine is now a replica of the others. The key problem is making each replica agree what is the next message to process: a consensus problem.
    5. However, the system is not fault tolerant; if one process fails that others have to wait for it to recover.

*.  "Notes on Database Operating Systems" (1979).
    1. 2PC problem: Unfortunately 2PC would block if the TM (Transaction Manager) fails at the wrong time.

*.  "NonBlocking Commit Protocols" (1981)
    1. 3PC problem: The problem was coming up with a nice 3PC algorithm, this would only take nearly 25 years!

*. "Impossibility of distributed consensus with one faulty process" (1985)
    1. this famous result is known as the "FLP" result
    2. By this time "consensus" was the name given to the problem of getting a bunch of processors to agree a value.
    3. The kernel of the problem is that you cannot tell the difference between a process that has stopped and one that is running very slowly, making dealing with faults in an asynchronous system almost impossible.
    4. a distributed algorithm has two properties: safety and liveness. 2PC is safe: no bad data is ever written to the databases, but its liveness properties aren't great: if the TM fails at the wrong point the system will block.
    5. The asynchronous case is more general than the synchronous case: an algorithm that works for an asynchronous system will also work for a synchronous system, but not vice versa.

*.  "The Byzantine Generals Problem" (1982)
    1. In this form of the consensus problem the processes can lie, and they can actively try to deceive other processes.

*.  "A Comparison of the Byzantine Agreement Problem and the Transaction Commit Problem." (1987) .
    1. At the time the best consensus algorithm was the Byzantine Generals, but this was too expensive to use for transactions.

*.  "Uniform consensus is harder than consensus" (2000)
    1. With uniform consensus all processes must agree on a value, even the faulty ones - a transaction should only commit if all RMs are prepared to commit.
   
*.  "The Part-Time Parliament" (submitted in 1990, published 1998)
    1. Paxos consensus algorithm
   
*.  "How to Build a Highly Availability System using Consensus" (1996).
    1. This paper provides a good introduction to building fault tolerant systems and Paxos.

*.  "Paxos Made Simple (2001)
    1. The kernel of Paxos is that given a fixed number of processes, any majority of them must have at least one process in common. For example given three processes A, B and C the possible majorities are: AB, AC, or BC. If a decision is made when one majority is present eg AB, then at any time in the future when another majority is available at least one of the processes can remember what the previous majority decided. If the majority is AB then both processes will remember, if AC is present then A will remember and if BC is present then B will remember.
    2. Paxos can tolerate lost messages, delayed messages, repeated messages, and messages delivered out of order.
    3. It will reach consensus if there is a single leader for long enough that the leader can talk to a majority of processes twice. Any process, including leaders, can fail and restart; in fact all processes can fail at the same time, the algorithm is still safe. There can be more than one leader at a time.
    4. Paxos is an asynchronous algorithm; there are no explicit timeouts. However, it only reaches consensus when the system is behaving in a synchronous way, ie messages are delivered in a bounded period of time; otherwise it is safe. There is a pathological case where Paxos will not reach consensus, in accordance to FLP, but this scenario is relatively easy to avoid in practice.

*.   "Consensus in the presence of partial synchrony" (1988)
    1. There are two versions of partial synchronous system: in one processes run at speeds within a known range and messages are delivered in bounded time but the actual values are not known a priori; in the other version the range of speeds of the processes and the upper bound for message deliver are known a priori, but they will only start holding at some unknown time in the future.
    2. The partial synchronous model is a better model for the real world than either the synchronous or asynchronous model; networks function in a predicatable way most of the time, but occasionally go crazy.
   
*.   "Consensus on Transaction Commit" (2005).
    1. A third phase is only required if there is a fault, in accordance to the Skeen result. Given 2n+1 TM replicas Paxos Commit will complete with up to n faulty replicas.
    2. Paxos Commit does not use Paxos to solve the transaction commit problem directly, ie it is not used to solve uniform consensus, rather it is used to make the system fault tolerant.
    3.  Recently there has been some discussion of the CAP conjecture: Consistency, Availability and Partition. The conjecture asserts that you cannot have all three in a distributed system: a system that is consistent, that can have faulty processes and that can handle a network partition.
    4. Now take a Paxos system with three nodes: A, B and C. We can reach consensus if two nodes are working, ie we can have consistency and availability. Now if C becomes partitioned and C is queried, it cannot respond because it cannot communicate with the other nodes; it doesn't know whether it has been partitioned, or if the other two nodes are down, or if the network is being very slow. The other two nodes can carry on, because they can talk to each other and they form a majority. So for the CAP conjecture, Paxos does not handle a partition because C cannot respond to queries. However, we could engineer our way around this. If we are inside a data center we can use two independent networks (Paxos doesn't mind if messages are repeated). If we are on the internet, then we could have our client query all nodes A, B and C, and if C is partitioned the client can query A or B unless it is partitioned in a similar way to C.
    5. a synchronous network, if C is partitioned it can learn that it is partitioned if it does not receive messages in a fixed period of time, and thus can declare itself down to the client.

*.   "Co-Allocation, Fault Tolerance and Grid Computing" (2006).


[REF] http://betathoughts.blogspot.com/2007/06/brief-history-of-consensus-2pc-and.html

1.      Interactive games are write-heavy. Typical web apps read more than they write so many common architectures may not be sufficient. Read heavy apps can often get by with a caching layer in front of a single database. Write heavy apps will need to partition so writes are spread out and/or use an in-memory architecture.

2.    Design every component as a degradable service. Isolate components so increased latencies in one area won't ruin another. Throttle usage to help alleviate problems. Turn off features when necessary.

3.    Cache Facebook data. When you are deeply dependent on an external component consider caching that component's data to improve latency.

4.    Plan ahead for new release related usage spikes.

5.      Sample. When analyzing large streams of data, looking for problems for example, not every piece of data needs to be processed. Sampling data can yield the same results for much less work.

The key ideas are to isolate troubled and highly latent services from causing latency and performance issues elsewhere through use of error and timeout throttling, and if needed, disable functionality in the application using on/off switches and functionality based throttles.

1.如果是非引用賦值，用于賦值的變量指向的zval的is_ref=0，則直接指向，refcount++；若zval的is_ref=1，則copy on write，原zval refcount不變, 新變量指向一個新的zval，is_ref=0, refcount=1;

2.如果是引用賦值，用于復制的變量指向的zval的is_ref=0，則copy on write，原zval refcount--，新變量和引用變量同時指向新的zval，is_ref=1,refcount=2; 若zval的is_ref=1，則直接指向,refcount++;