Skip to main content

A positioning analysis of a high server CPU usage

Through performance monitoring, it is found that the CPU occupancy rate of a certain core of the online server has reached 100%, and it is caused by one of our core services. Fortunately, because our service process is borne by multiple identical workers (threads), apart from the high CPU usage, there is no impact on the service. With the last time we found the criminal who eats IO, this time we are going to hunt down the secret agents lurking in the group, which is even more thrilling!
System environment



System environment
It is easy to locate who is consuming the highest CPU with the top command.


The top command locates the highest CPU process
Take our business process (imDevServer) as an example, why is it said that this product is a lurker? Because this is a multi-threaded process, we need to know that the smallest unit that actually occupies the cpu is the thread, so it must be caused by one or several of the threads that occupy the CPU too high. Then use the top -H -p pid command to view the percentage of CPU occupied by each thread in the process.


top -H -p pid command to view the percentage of CPU occupied by threads in the process
As shown in the figure above, we can see that the thread with id 8863 has the highest CPU occupancy rate. Well, we only need to find the cpu he stole now. Although this kid has a strict mouth, we have a complete interrogation process and we are not afraid that he will not recruit. The first thing to do is the strace -T -r -c -p pid command


strace -T -r -c -p pid command
Its function is to check the system call and the time spent. Although epoll_wait takes up a lot of call time, it is a normal blocking call. We then let pstack pid go out


pstack pid
You can see the call stack of each thread, find the thread that occupies the highest CPU, and then look at his call stack, it is easy to see which step logically caused the busy loop, and then use trace -p tid to see Look at the calling process of the thread and then locate the code, fix the bug, and retrieve the stolen cpu.

Comments

Popular posts from this blog

AMD's GPU technology enters the mobile phone chip market for the first time

In addition to the release of the Exynos2100 processor, Samsung also confirmed a major event at this Exynos event, that is, the custom GPU that they have worked with AMD for many years will soon appear and will be used on the next flagship machine. The current Exynos2100 processor uses ARM’s Mali-G78GPU core with a total of 14 cores, so the GPU architecture developed by Samsung will be the next Exynos processor, and the GPU will be the focus. This is probably the meaning of Exynos2100’s GPU stacking. The key reason. Dr. InyupKang, president of Samsung’s LSI business, confirmed that the next-generation mobile GPU in cooperation with AMD will be used in the next flagship product, but he did not specify which product. Samsung is not talking about the next-generation flagship but the next one, so it is very likely that a new Exynos processor will be available this year, either for the GalaxyNote21 series or the new generation of folding screen GalaxyZFold3. In 2019, AMD and Samsung reached

Apple and Intel want to join the game, what happened to the GPU market?

Intel recently announced that it will launch Xe-LP GPU at the end of this year, officially entering the independent GPU market, and will hand over to TSMC for foundry. At the 2020 WWDC held not long ago, Apple also revealed that it is possible to abandon AMD's GPU and use a self-developed solution based on the ARM architecture. It will launch a self-developed GPU next year. What happened to the GPU market? Why are the giants entering the game?    Massive data calls for high-performance GPU    Why has the demand for GPUs increased so rapidly in recent years? Because we are entering an era where everything needs to be visualized. Dai Shuyu, a partner of Aiwa (Beijing) Technology Co., Ltd., told a reporter from China Electronics News that visualization requires a large amount of graphics and image computing capabilities, and a large amount of high-performance image processing capabilities are required for both the cloud and the edge.    Aiwa (Beijing) Technology Co., Ltd. is an enterp

SSD vs. HDD: that is nice for You?

Deciding on the proper garage isn’t just about comparing ability and value. The type of garage your pc uses topics for overall performance, inclusive of electricity utilization and reliability. Stable country drives (SSDs) and difficult disk drives (HDDs) are the 2 major storage options to consider and it’s critical to realize the satisfactory use for each and how they compare facet by using aspect. What is an HDD? An HDD is a statistics garage tool that lives within the pc. It has spinning disks inside where statistics is saved magnetically. The HDD has an arm with several "heads" (transducers) that study and write statistics at the disk. It is much like how a turntable file participant works, with an LP record (tough disk) and a needle on an arm (transducers). The arm moves the heads throughout the surface of the disk to get admission to distinct information. HDDs are considered a legacy generation, that means they’ve been round longer than SSDs. In popular, they may be dec