OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#5175 — backbone update
Scheduled Maintenance Report for Network & Infrastructure
Completed
We are going to update the routers of the backbone
on the last version of the IOS. The aim of this operation is :
to fix all the bugs tied to the IPV6 and then reformat the TCAM CEF
with a specific configuration.

#sh mls cef maximum-routes
FIB TCAM maximum routes :
=======================
Current :-
-------
IPv4 - 911k
MPLS - 1k
IPv6 - 50k
IP multicast - 1k

Then add it on all the router.
no spanning-tree vlan 1-4094

Update(s):

Date: 2011-06-03 02:55:52 UTC
it's done. ouff ! :)

Date: 2011-06-03 02:55:28 UTC
still ams-5-6k

Date: 2011-06-02 23:34:57 UTC
bor-1-6k en in progress.

Date: 2011-06-02 23:34:35 UTC
fra-5-6k stabilized.

ams-1-6k in progress.

Date: 2011-06-02 23:31:29 UTC
Tonight we have plenty of problems.
The second update is not doing good with the g1 ,wa have 3 problems to deal with within less than 1 hours.

The router here will not reboot without the intervention of sysadmin. Long live serie cables !

Jun 2 23:32:30 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
Jun 2 23:32:30 GMT: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
Jun 2 23:32:33 GMT: %SYS-SP-5-RELOAD: Reload requested by Delayed Reload. Reload Reason: Reload Command.
Jun 2 23:32:33 GMT: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
monitor: command \"boot\" aborted due to user interrupt
Exit at the end of BOOT string
rommon 1 > boot
Please reset before booting
rommon 2 > reset
System Bootstrap, Version 8.5(2)
Copyright (c) 1994-2007 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 1048576 Kbytes of main memory

[...]

Date: 2011-06-02 23:25:33 UTC
fra-5-6k in progress.

Date: 2011-06-02 22:29:59 UTC
Jun 2 23:04:05 GMT: %EARL-DFC1-2-SWITCH_BUS_IDLE: Switching bus is idle for 5 seconds. The card grant is 0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x46] JA_DR_RI_0_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x85] JA_DR_RI_1_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x1F
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [0:0x5F7] JA_TM_LO_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x46] JA_DR_RI_0_STA_FCO = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x85] JA_DR_RI_1_STA_FCO = 0x3
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x28A] JA_FI_FT_RCV_RATE_SEL = 0xC
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x289] JA_FI_FT_XMIT_SHAPE = 0xFFF
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CB] JA_AG_PM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CC] JA_AG_PM_RAM_FULLNESS_1 = 0x101
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CD] JA_AG_PM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4CE] JA_AG_PM_RAM_FULLNESS_3 = 0x102
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F1] JA_AG_RM_RAM_FULLNESS_0 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F2] JA_AG_RM_RAM_FULLNESS_1 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F3] JA_AG_RM_RAM_FULLNESS_2 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x4F4] JA_AG_RM_RAM_FULLNESS_3 = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F6] JA_TM_HI_FULLNESS = 0x0
Jun 2 23:04:05 GMT: %PF_ASIC-DFC1-3-ASIC_DUMP: [1:0x5F7] JA_TM_LO_FULLNESS = 0x0


Card #1 is dead.

Date: 2011-06-02 22:29:10 UTC
p19-2-6k

Date: 2011-06-01 12:24:54 UTC
Done.

There is still
fra-5
ams-1
ams-5
p19-2

then it will be finished

Date: 2011-06-01 12:24:12 UTC
We will update fra-1-6k

Date: 2011-05-24 15:04:25 UTC
lyo-1-6k is up and put again in the routing loop.

Date: 2011-05-24 15:03:38 UTC
We will update lyo-1-6k

Date: 2011-05-24 15:03:11 UTC
mad-1-6k up and put again in the routing loop.
var-1-6k up and put again in the routing loop.

Date: 2011-05-24 15:02:38 UTC
mil-1-6k up and put again in the routing loop.

Date: 2011-05-24 15:01:01 UTC
We will update var-1-6k.

Date: 2011-05-24 15:00:37 UTC
We will update mad-1-6k

Date: 2011-05-24 15:00:07 UTC
We will update mil-1-6k

Date: 2011-05-23 20:56:31 UTC
done

Date: 2011-05-23 20:56:05 UTC
We are going to update zur-1-6k

Date: 2011-05-23 20:53:13 UTC
We are going to update pra-1-6k

Date: 2011-05-20 10:08:34 UTC
We are going to update vie-1-6k .


Date: 2011-05-11 09:59:05 UTC
The updating of lyo-1 was reported.

Date: 2011-05-11 09:40:21 UTC
We will redirect the traffic routed by lyo-1-6k in order to update the router.

Date: 2011-05-06 01:19:02 UTC
gsw-1 is updated.

Date: 2011-05-06 00:03:32 UTC
We are starting the intervention on th1-1-6k.

Date: 2011-05-05 22:59:10 UTC
gsw-1/th1-1/lyo-1/mil-1 are remaining. We are going to deal with these routers tonight.



Date: 2011-04-22 05:11:21 UTC
vss-1/2/3/4 are updated.

Date: 2011-04-09 06:22:46 UTC
Done. the HSRP problem is currently fixed.
We shouldn't face it for a long time.
If everything is fine,we will retry GLBP.

Date: 2011-04-09 06:20:40 UTC
We have a tiny boot problem.We must take off and set back the disk in the additional card.

Initializing ATA monitor library...
monlib.open(): Open Error = -35
loadprog: error - on file open
Exit at the end of BOOT string

rommon 1 > dir disk0:
Command error complete on disk0:
open: read error...requested 0x4 bytes, got 0xffffffff
trouble reading device magic number

rommon 2 > dir disk0:
Initializing ATA monitor library...
Directory of disk0:
[...]
rommon 3 > reset
Initializing ATA monitor library...
Self extracting the image... [OK]

Date: 2011-04-09 06:17:13 UTC
rbx-s3 under way.

Date: 2011-04-09 06:15:44 UTC
rbx-s4 done.

Date: 2011-04-09 06:15:20 UTC
We have a problem of HSRP on HG routers rbx-s3/s4. We'll update IOS on these 2 routers to see if it fixes the problem.

Date: 2011-04-09 03:48:09 UTC
rbx-1-6k done

Date: 2011-04-09 03:47:55 UTC
rbx-2-6k done

Date: 2011-04-09 03:23:56 UTC
After updating the IOS rbx-1-6k et rbx-2-6k are not going well.

We have problems of version mismatch hardware of some cards with the software version. We are going to resolve this problem with TAC cisco.

Meanwhile,a new version of IOS was released and since the router rbx-2-6k planted
http://status.ovh.co.uk/?do=details&id=1327
with a software error,we will certainly update rbx-2-6k and rbx-1-6k with SXI6.
This may allow us to fix another problem that has appeared following the update:every night at 4:00 o'clock rbx-1-6k is full of CPU for a few minutes. This may be related to hardware problem or maybe software.

Date: 2011-03-12 06:53:07 UTC
Done

Date: 2011-03-12 06:52:59 UTC
We will update p19-7.

Date: 2011-03-07 04:22:08 UTC
p19-57 done

the p19-7 will be done another night. we have to securise better another router via p19-2 before starting with p19-7.

Date: 2011-03-07 04:20:00 UTC
p19-52 done,
Now to p19-57.

Date: 2011-03-07 04:19:37 UTC
Started
p19-52

Date: 2011-03-07 04:19:16 UTC
Tonight,we updated 3 routers of p19:
- p19-7-6k one of the 2 routeurs of p19
- p19-52-6k one of the 3 routeurs of shared hosting and OVH
- p19-57-6k one of the 3 routeurs of shared hosting and OVH

During the reboot and updating,the rooting will be ensured with the other routers.

We can not avoid small cuts that are due to the time needed to converge network (to remove the broken roads and return to the roads
that still work). about 30 seconds here and there.

Date: 2011-02-25 03:54:03 UTC
rbx-s6-6k one of the 2 routers of pCC.

Date: 2011-02-25 03:53:25 UTC
One of the cards 10G on mar-1-6k died.

Feb 25 03:02:10 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:02:10 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:02:10 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)
Feb 25 03:03:25 GMT: %PM_SCP-SP-1-LCP_FW_ERR: System resetting module 3 to recover from error: Linecard received system exception. Errcode =
2509000001
Feb 25 03:03:25 GMT: %OIR-SP-3-PWRCYCLE: Card in module 3, is being power-cycled 'Off (Module Reset due to exception or user request)'
Feb 25 03:03:25 GMT: %C6KPWR-SP-4-DISABLED: power to module in slot 3 set Off (Module Reset due to exception or user request)

Date: 2011-02-25 03:51:27 UTC
mar-1-6k one of the routers ISP in Marseille.

Date: 2011-02-25 03:50:44 UTC
rbx-s5-6k one of the 2 rooters of HG.

Date: 2011-02-24 21:50:12 UTC
rbx-s4-6k one of the 2 rooters of HG.

Date: 2011-02-24 13:54:29 UTC
We are switching the London<>New York link.

Date: 2011-02-24 12:41:20 UTC
We switch the LINX 224.

Date: 2011-02-24 12:39:03 UTC
nyk-1-6k
Posted Feb 24, 2011 - 12:38 UTC