《Linux System and Performance Monitoring》CPU篇

前言: 原文《Linux System and Performance Monitoring》,本文尝试翻译文章中的CPU篇,并且省略了一些地方,也在一些地方加了自己的理解。

转载请注明出处: http://www.firefoxbug.com/?p=2760

Introducing the CPU

The utilization of a CPU is largely dependent on what resource is attempting to access it. The kernel has a scheduler that is responsible for scheduling two kinds of resources: threads (single or multi) and interrupts. The scheduler gives different priorities to the different resources. The following list outlines the priorities from highest to lowest:

  • Interrupts – Devices tell the kernel that they are done processing. For example, a NIC delivers a packet or a hard drive provides an IO request
  • Kernel (System) Processes – All kernel processing is handled at this level of priority.
  • User Processes – This space is often referred to as “userland”. All software applications run in the user space. This space has the lowest priority in the kernel scheduling mechanism.

In order to understand how the kernel manages these different resources, a few key concepts need to be introduced. The following sections introduce context switches, run queues, and utilization.

CPU的利用率主要依赖于想访问CPU的资源。内核的调度器负责调度两种资源: 线程和中断,调度器会给这两种资源不同的优先级,下面列出内核调度各个资源从高到底的优先级。

  • 中断: 设备告诉内核正在处理的事情。举例,网卡收到了一个数据包,放到网卡的缓冲区,然后需要告知内核
  • 内核进程: 在内核空间跑的线程或者进程
  • 用户进程: 也就是所谓的用户空间,在这个内存区域跑的所有应用都算,在整个内核调度优先级级别里是最低的。

为了理解内核是如何管理这些不同的资源,需要引入一些概念,接下来就会介绍上下文切换,运行队列和utilization

Context Switches

Most modern processors can only run one process (single threaded) or thread at time. The n-way Hyper threaded processors have the ability to run n threads at a time. Still, the Linux kernel views each processor core on a dual core chip as an independent processor. For example, a system with one dual core processor is reported as two individual processors by the Linux kernel.

大多数现代的处理器可以在某一时刻执行一个进程或者线程,N路处理器能同时处理N个线程。对于Linux内核而言,每个带有双核芯片被作为是一个独立的处理器。举个例子,一个带有双处理器的系统被当做是两个独立的处理器。

ps: CPU多核技术,一个物理的CPU上可以有多个芯片从而实现多个核,也就是说有物理CPU数和逻辑CPU数

物理CPU核数:
cat /proc/cpuinfo |grep "physical id"|sort |uniq|wc -l

逻辑CPU核数
cat /proc/cpuinfo |grep "processor"|wc -l  

A standard Linux kernel can run anywhere from 50 to 50,000 process threads at once. With only one CPU, the kernel has to schedule and balance these process threads. Each thread has an allotted time quantum to spend on the processor. Once a thread has either passed the time quantum or has been preempted by something with a higher priority (a hardware interrupt, for example), that thread is place back into a queue while the higher priority thread is placed on the processor. This switching of threads is referred to as a context switch.

一个标准的linux内核能同时处理50到50000个线程,在只有一个cpu的情况下,内核必须不断调度和均衡这些要处理的线程。cpu会分配给每个线程一个时间片,一旦线程执行时间超过了分配的时间片,或者执行过程中被更高优先级的线程打断(硬件中断之类的),那么这个被打断的线程就会放入一个队列,同时cpu会被更高优先级的线程抢占,这个线程之间的切换就叫做上下文切换。

Every time the kernel conducts a context switch, resources are devoted to moving that thread off of the CPU registers and into a queue. The higher the volume of context switches on a system, the more work the kernel has to do in order to manage the scheduling of processes.

内核每次做上下文切换,把线程执行状态从CPU寄存器里保存以及把线程放入队列是需要资源的。在一个系统里,上下文切换数量越高,内核为了管理调度所有线程所做的工作就越多。

The Run Queue

Each CPU maintains a run queue of threads. Ideally, the scheduler should be constantly running and executing threads. Process threads are either in a sleep state (blocked and waiting on IO) or they are runnable. If the CPU sub-system is heavily utilized, then it is possible that the kernel scheduler can’t keep up with the demand of the system. As a result, runnable processes start to fill up a run queue. The larger the run queue, the longer it will take for process threads to execute.

A very popular term called “load” is often used to describe the state of the run queue. The system load is a combination of the amount of process threads currently executing along with the amount of threads in the CPU run queue. If two threads were executing on a dual core system and 4 were in the run queue, then the load would be 6. Utilities such as top report load averages over the course of 1, 5, and 15 minutes.

每个CPU为所有线程维持着一个运行队列,调度器从队列里不断运行和处理线程。处理的线程要么处于sleep(阻塞或者等待IO)状态,要么就是处于运行状态。如果CPU被高度利用,内核调度器可能会跟不上系统的需求,结果就可能导致可运行的线程开始在运行队列里堆积,运行队列越长,处理这些堆积线程的时间越长。

一个流行的术语就是叫做"load",用它来描述运行队列的状态。一个系统的load是当前正在处理的线程数量和在CPU运行队列里的线程数量的综合反应,如果一个双核系统上两个线程在处理,并且有4个线程在运行队列里等待,那么load就是6。top命令出来的就是对于系统近1,5,15分钟系统load的描述。

更多查看理解Linux系统负荷

CPU Utilization

CPU utilization is defined as the percentage of usage of a CPU. How a CPU is utilized is an important metric for measuring system. Most performance monitoring tools categorize CPU utilization into the following categories:

  • User Time – The percentage of time a CPU spends executing process threads in the user space.
  • System Time – The percentage of time the CPU spends executing kernel threads and interrupts.
  • Wait IO – The percentage of time a CPU spends idle because ALL process threads are blocked waiting for IO requests to complete.
  • Idle – The percentage of time a processor spends in a completely idle state.

CPU利用率被定义为CPU的使用百分比,是一项测试系统的指标。大多数性能监控工具把CPU利用率分成下面几类:

  • 用户时间: CPU处理一个线程在用户空间占用时间的百分比
  • 系统时间: CPU处理内核线程以及中断所占时间百分比
  • 等待IO: CPU由于所有线程都阻塞等待IO完成从而空闲的占用时间百分比
  • Idle: CPU完全空闲的时间百分比

ps : WaitIO和Idle的区别大概是这样,前者CPU是有正在处理的线程,但是所有的线程都在等待IO从而阻塞,导致CPU处于空闲的时间。后者是系统完全没有线程要调度。

CPU Performance Monitoring

Understanding how well a CPU is performing is a matter of interpreting run queue, utilization, and context switching performance. As mentioned earlier, performance is all relative to baseline statistics. There are, however, some general performance expectations on any system. These expectations include:

  • Run Queues – A run queue should have no more than 1-3 threads queued per processor. For example, a dual processor system should not have more than 6 threads in the run queue.
  • CPU Utilization – If a CPU is fully utilized, then the following balance of utilization should be achieved.
  • 65% – 70% User Time
  • 30% - 35% System Time
  • 0% - 5% Idle Time
  • Context Switches – The amount of context switches is directly relevant to CPU utilization. A high amount of context switching is acceptable if CPU utilization stays within the previously mentioned balance

要理解CPU性能要涉及到运行队列,利用率以及上下文切换。下面是一些性能预测指标:

  • 运行队列: 每个处理器的运行队列不应该不应该超过1-3个等待运行线程。举个例子,一个双核处理器系统的运行队列不应该超过6个线程。
  • CPU使用率: 如果一个CPU是完全运作,那么使用率的各项时间百分比应该是这样:
  • 65%-70% 用户时间
  • 30%-35% 系统时间
  • 0%-5% Idle空闲时间
  • 上下文切换: 上下文切换直接会影响CPU使用率,高数量的上下文切换是可以接受的如果CPU利用率各项指标符合上面的负载分配

Using the vmstat Utility

vmstat是一项负载消耗很低的监控系统性能的工具,后面跟参数,几秒钟刷新一次。

vmstat

指标 含义
r CPU运行队列中等待执行的线程数量,CPU还没执行
b 阻塞等待IO的线程数量
in 正在被处理的中断数量
cs 系统正在上下文切换的数量
us CPU在用户空间执行所占时间百分比
sy CPU执行系统线程或者中断所占时间百分比
id CPU完全空闲的所占时间百分比
wa 由于所有线程都等待IO从而阻塞,CPU处于空闲状态的时间百分比

If your system has multiple processor cores, you can use the mpstat command to monitor each individual core. The Linux kernel treats a dual core processor as 2 CPU’s. So, a dual processor system with dual cores will report 4 CPUs available. The mpstat command provides the same CPU utilization statistics as vmstat, but mpstat breaks the statistics out on a per processor basis.

如果系统有多核处理器,可以使用mpstat来监控每个独立的核,Linux内核把一个双核处理器认为有两个2芯片,因此一个双处理器的系统会显示4个cpu核。mpstat命令提供实现针对每一个核的cpu利用率监控。

$ mpstat -P ALL 1
05:51:06 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
05:51:08 PM  all   14.80    0.00    3.05    3.73    0.03    0.65    0.00    0.00   77.74
05:51:08 PM    0    7.96    0.00    1.99    1.99    0.00    1.00    0.00    0.00   87.06
05:51:08 PM    1   30.39    0.00    4.90    1.96    0.00    1.47    0.00    0.00   61.27
05:51:08 PM    2   13.64    0.00    1.52    0.00    0.00    0.51    0.00    0.00   84.34
M

Conclusion

Monitoring CPU performance consists of the following actions:

  • Check the system run queue and make sure there are no more than 3 runnable threads per processor
  • Make sure the CPU utilization is split between 70/30 between user and system
  • When the CPU spends more time in system mode, it is more than likely overloaded and trying to reschedule priorities
  • Running CPU bound process always get penalized while I/O process are rewarded

监控CPU性能包含下面这些方法:

  • 检查系统运行队列,确保每个处理器没有超过3个等待运行的线程
  • 确保CPU利用率用户百分比和系统百分比分别是70%和30%
  • 当CPU在系统空间执行的时间更加多,很有可能是超负载同时尝试重新调度高优先级的线程
  • 运行CPU密集型的进程总是比I/O密集型的更占优势

标签:Linux, OS

评论已关闭