öffnen Tags

öffnen Kategorien

Vom WebReader vorlesen lassen

Memory Leak in Ciscos IOS 12.1(13)

17. Nov 2009, 21:43

For about two moths I'm monitoring our Cisco Catalysts with own Nagios-plugins, written in Perl (perhaps I'll provide them them afterwards). Since one week Nagios is alerting, the memory of three of the switches are running out of memory.

These three switches (Type: Catalysts WS-C2950G-48-EI) are running an IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(13)EA1, they are older ones.
So it's getting strange, three different switches, same model, same image, are getting memory problems independet of each other!?
Due to other more importent problems I couldn't care about this curiosity till yesterday. This issue became necessary, because one more week and the switches ran out of mem, so I had to solve this challange.

Connecting to one of them I was searching for a process that is absorbing all the memory, and the following process was grabbing my attention:

1
2
3
4
5
#show processes memory
PID TTY Allocated Freed Holding Getbufs Retbufs Process
[ ... ]
69 0 744289376 892704748 1445732 0 0 SNMP ENGINE
[ ... ]

Such a high holding number isn't normal for a process like this!? And it continues requesting memory! The next command will verify my first opinion:

1
2
3
4
5
6
7
8
9
10
11
12
#show memory allocating-process totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 80B9EAE0 4834592 4381012 453580 231252 438956
I/O A08B4CE0 3055072 1344220 1710852 1670976 1710672

Allocator PC Summary for: Processor

PC Total Count Name
0x8003506C 1500584 17074 SNMP ENGINE
0x8018EF84 942408 571 *Packet Data*
0x80202830 504272 74 Interrupt Stack
[ ... ]

It's the most allocating process! Some more informations about this process:

1
2
3
4
5
6
7
8
9
10
11
#show processes memory 69

Proc Memory Summary for pid = 69
Holding = 1446260

pc = 0x8003506C, size = 001427712, count = 16144
pc = 0x80202830, size = 000012044, count = 0001
pc = 0x802C7518, size = 000005184, count = 0009
pc = 0x801FD368, size = 000000580, count = 0001
pc = 0x800353BC, size = 000000556, count = 0004
pc = 0x801EAE2C, size = 000000184, count = 0001

Ok, it looks weird, so I was searching for such problems and found this article at Ciscos website:


This is a Cisco Internet Operating System (IOS) bug. To get rid of it, upgrade to the latest version in your release train
[...]
To identify the process that is consuming more memory, compare the Holding column of the show processes memory command over the time interval. Sometimes you can very clearly see that one process is holding several megabytes of memory.

So upgrading the IOS seems to be the solution, in 12.1(19) it should have been fixed.
For now I disabled the Nagios-checks for these switches, so the memory level stays constant. It's a low level, but has to be enough, untill I'll receive the new IOS image.

Tags: Bugs Cisco IOS Nagios Netzwerk Security Software

Kategorien: Betriebssystem Bugs Netzwerk Sicherheit Software

© 2009-2018 by Martin Scharm