Skip to main content

A positioning analysis of a high server CPU usage

Through performance monitoring, it is found that the CPU occupancy rate of a certain core of the online server has reached 100%, and it is caused by one of our core services. Fortunately, because our service process is borne by multiple identical workers (threads), apart from the high CPU usage, there is no impact on the service. With the last time we found the criminal who eats IO, this time we are going to hunt down the secret agents lurking in the group, which is even more thrilling!
System environment



System environment
It is easy to locate who is consuming the highest CPU with the top command.


The top command locates the highest CPU process
Take our business process (imDevServer) as an example, why is it said that this product is a lurker? Because this is a multi-threaded process, we need to know that the smallest unit that actually occupies the cpu is the thread, so it must be caused by one or several of the threads that occupy the CPU too high. Then use the top -H -p pid command to view the percentage of CPU occupied by each thread in the process.


top -H -p pid command to view the percentage of CPU occupied by threads in the process
As shown in the figure above, we can see that the thread with id 8863 has the highest CPU occupancy rate. Well, we only need to find the cpu he stole now. Although this kid has a strict mouth, we have a complete interrogation process and we are not afraid that he will not recruit. The first thing to do is the strace -T -r -c -p pid command


strace -T -r -c -p pid command
Its function is to check the system call and the time spent. Although epoll_wait takes up a lot of call time, it is a normal blocking call. We then let pstack pid go out


pstack pid
You can see the call stack of each thread, find the thread that occupies the highest CPU, and then look at his call stack, it is easy to see which step logically caused the busy loop, and then use trace -p tid to see Look at the calling process of the thread and then locate the code, fix the bug, and retrieve the stolen cpu.

Comments

Popular posts from this blog

AMD's GPU technology enters the mobile phone chip market for the first time

In addition to the release of the Exynos2100 processor, Samsung also confirmed a major event at this Exynos event, that is, the custom GPU that they have worked with AMD for many years will soon appear and will be used on the next flagship machine. The current Exynos2100 processor uses ARM’s Mali-G78GPU core with a total of 14 cores, so the GPU architecture developed by Samsung will be the next Exynos processor, and the GPU will be the focus. This is probably the meaning of Exynos2100’s GPU stacking. The key reason. Dr. InyupKang, president of Samsung’s LSI business, confirmed that the next-generation mobile GPU in cooperation with AMD will be used in the next flagship product, but he did not specify which product. Samsung is not talking about the next-generation flagship but the next one, so it is very likely that a new Exynos processor will be available this year, either for the GalaxyNote21 series or the new generation of folding screen GalaxyZFold3. In 2019, AMD and Samsung reached

Apple and Intel want to join the game, what happened to the GPU market?

Intel recently announced that it will launch Xe-LP GPU at the end of this year, officially entering the independent GPU market, and will hand over to TSMC for foundry. At the 2020 WWDC held not long ago, Apple also revealed that it is possible to abandon AMD's GPU and use a self-developed solution based on the ARM architecture. It will launch a self-developed GPU next year. What happened to the GPU market? Why are the giants entering the game?    Massive data calls for high-performance GPU    Why has the demand for GPUs increased so rapidly in recent years? Because we are entering an era where everything needs to be visualized. Dai Shuyu, a partner of Aiwa (Beijing) Technology Co., Ltd., told a reporter from China Electronics News that visualization requires a large amount of graphics and image computing capabilities, and a large amount of high-performance image processing capabilities are required for both the cloud and the edge.    Aiwa (Beijing) Technology Co., Ltd. is an enterp

NVIDIA officially launches RTX 30 series mobile graphics cards

In the early morning of January 13, NVIDIA officially launched the RTX30 series of mobile graphics cards at the CES2021 exhibition. Ampere-based GPUs have also reached the mobile terminal, mainly including RTX3080, RTX3070 and RTX3060 models. In addition to improving game performance, the RTX30 series of mobile graphics cards have twice the energy efficiency of the previous generation, and support the third-generation Max-Q technology, mainly supporting DynamicBoost2.0 dynamic acceleration technology, WisperMode2.0 noise control, ResizableBAR (similar to AMD’s SAM technology) and DLSS. The third-generation Max-Q technology uses AI and new system optimization to make high-performance gaming laptops faster and more powerful than ever. These technologies include: ·DynamicBoost2.0: The CPU and GPU powers of traditional gaming notebooks are fixed, while games and creative applications are dynamic, and the requirements for the system will vary with the number of frames. With DynamicBoost2.0,