H852小型机故障记录.docx

小型机故障记录故障机器：主机名H852BootIP1OJ0.20.30ServiceIP10.10.20.40StandbyIP192.168.0.20故障时间：2007年9月3日下午2点左右故障现象：同一网段的小型机与该主机通信正常，但跨网段的机器无法与该主机通信初步判断：该主机的网络设置被改动，或者是网络设备物理或逻辑上的变动所致处理过程：1询问网络管理员，他们最近没有对网络设备作物理或逻辑上的调整2在本机查网络设置和日志通过对本机的配置进行查询，发现存在如下现象，现详细记录如下：1 #hostnameH8522 #ifconfig-aen:flags=4e080863,80<URBROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>inet10.10.20.40netmaskOxffffffOObroadcast10.10.20.255enl:flags=4e080863,80<URBROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>inet10.0.0.1netmaskOxffDOOOOObroadcast10.255.255.255lo0:flags=e08084b<URBROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT>inet127.0.0.1netmaskOxffOOOOOObroadcast127.255.255.255inet6:1/0tcp_sendspace131072tcp_recvspace131072rfcl32313#netstat-rRoutingtablesDestinationGatewayFlagsRefsUseIfPMTUExpGroupsRouteTreeforProtocolFamily2(Internet):default10.10.20.1UG01312547en-10.0.0.010.0.0.1UHSb00enl-=>10/810.0.0.1U311028enl-10.0.0.1loopbackUGHS02IoO-10.10.20.0loopbackUHSb00IoO-=>10.10.20.0h852_serviceUHSb00en-=>10.10.20/24h852_serviceU5104741en-h852_servicelpbackUGHS016668IoO-10.10.20.255loopbackUHSb0269IoO-=>10.10.20.255h852_serviceUHSb02en-10.255.255.25510.0.0.1UHSb00enl-127/8loopbackU2114059043IoORouteTreeforProtocolFamily24(Internetv6):1:1UH00IoO4#errpt#errptIDENTIFIERTIMESTAMPTCRESOURCE_NAMEDESCRIPTIONAFA899050903163807IOgrpsvcsGroupServicesdaemonstarted92E96A0D0903163807PUtopsvcsLocaladaptermisconfigurationdetected97419D600903163807IOtopsvcsTopologyServicesdaemonstarted5294E5FF0903163807IOtopsvcsMissingorincorrectadapteraddress6D19271E0903143807IOtopsvcsTopologyServicesdaemonstopped28854E810903143807IOgrpsvcsGroupSen,icesdaemonstopped99FA80C70903143807UShaemdSOFTWARE9D30B78E0826234007TSttylRECEIVEROVER-RUNONINPUT9D30B78E0825124307TSttylRECEIVEROVER-RUNONINPUT9D30B78E0825050007TSttylRECEIVEROVER-RUNONINPUT9D30B78E0822205307TSttylRECEIVEROVER-RUNONINPUT9D30B78E0821200307TSttylRECEIVEROVER-RUNONINPUT9D30B78E0818012907TSttylRECEIVEROVER-RUNONINPUT9D30B78E0812174207TSttylRECEIVEROVER-RUNONINPUT3C81E43F0720000007PUtopsvcsLateinsendingheartbeat92E96A0D0716171507PUtopsvcsLocaladaptermisconfigurationdetected5#errpt-aj5294E5FFLABEL:IDENTIFIER:TS_FDJNVAL_ADDR_ST5294E5FFDate/Time:MonSep316:38:36BEIST2007SequenceNumber:95368MachineId:000C4ADF4C00NodeId:h852Class:Type:ResourceName:OINFOtopsvcsDescriptionMissingorincorrectadapteraddressProbableCausesAdapterisnotconfiguredAdapterhasnoaddressAdapterhasanaddressthatisoutsidetheconfigurationUserCausesAdapterwasincorrectlyconfiguredRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesFailureCausesAproblemoccurredwhenconfiguringtheadapterRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesDetailDataDETECTINGMODULErsct,force_down.C,1.11,501ERRORID6zLCZGAUwq4w8P.SEV08REFERENCECODEInterfacenameenlActualaddress10.0.0.1Expectedboot-timeaddress192.168.0.206#errpt-aj92E96A0DLABEL:IDENTIFIER:TS_MISCFG_EM92E96A0DDate/Time:MonSep316:38:37BEIST2007SequenceNumber:95370MachineId:000C4ADF4C00NodeId:h852Class:Type:ResourceName:ResourceClass:UPENDtopsvcsNONEResourceType:NONE1.ocation:VPD:Description1.ocaladaptermisconfigurationdetectedProbableCauses1.ocaladapterassignedincorrectIPaddressInstallCauses1.ocaladapterassignedincorrectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0BUwq42DCSEV08REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddress192.168.0.20LABEL:IDENTIFIER:TS_MISCFG_EM92E96A0DDate/Time:MonJul1617:15:35BEIST2007SequenceNumber:95344MachineId:000C4ADF4C00NodeId:h852Class:Type:ResourceName:ResourceClass:ResourceType:UPENDtopsvcsNONENONELocation:VPD:Description1.ocaladaptermisconfigurationdetectedProbableCauses1.ocaladapterassignedincorrectIPaddressInstallCauses1.ocaladapterassignedincorrectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0rQna4yPDlSEV08REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddress192.168.0.207部分登录信息#lastrootpts/2RAC_P5711Sep0409:31-09:44(00:13)rootpts/1RAC_P5711Sep0409:17stillloggedin.rootpts/3h851_serviceSep0316:51-17:04(00:13)rootpts/2RAC_P5711Sep0316:47-16:58(00:11)rootpts/1h851_serviceSep0316:42-18:57(02:15)rootIftOSep0313:55-16:40(02:44)rootpts/110.20.222.28Aug2415:24-16:53(01:29)rootpts/110.10.30.101Aug2410:14-12:24(02:10)rootpts/110.20.222.29Aug2120:05-20:11(00:05)rootpts/110.20.222.29Aug2119:30-19:31(00:01)rootpts/1h851_serviceAug2115:20-16:57(01:37)rootIftOAug1416:27-16:55(00:27)rootdtremote10.20.10.158:0Aug1409:03-13:38(04:35)rootpts/110.10.30.100Aug0911:08-15:41(04:33)rootdtremote10.20.10.158:0Aug0910:34-10:43(00:09)rootpts/O10.20.10.174Aug0308:59-13:16(04:17)rootpts/1h851_serviceJul1816:34-16:36(00:01)rootpts/110.20.10.174Jul1716:47-23:18(06:31)rootpts/110.20.10.174Jul1708:41-14:18(05:37)rootpts/110.20.10.174Jul1618:00-18:02(00:02)rootdtremote10.20.10.41:1Jul1616:58-17:19(00:20)rootpts/110.20.10.41Jul1616:48-16:55(00:07)rootrootIftOpts010.10.20.65Jul1616:28-17:03(11+00:34)Jul1616:25-16:26(00:00)rootpts010.20.10.24May2209:28-09:29(00:00)rootpts010.20.10.77May1615:43-15:43(00:00)rootpts010.10.10.33Apr2623:26-23:28(00:02)rootpts010.20.10.33Apr2610:04-10:08(00:04)rootpts010.20.10.33Apr2509:51-11:31(01:39)rootrebootshutdownrootrootIftOpts0IftOpts0h851_stbyApr0610:14-16:19(101+06:05)Apr0610:09Apr0610:00Apr0609:45-10:00(00:14)Aug2511:05-11:05(00:00)rootpts0h851_stbyAug2115:07-15:24(00:17)rootpts0h851_stbyAug2110:53-11:00(00:06)rootrebootshutdownrootrebootrebootshutdownrootrebootshutdownoracleIftOIftOIftOIftOIftOIftOftp10.20.10.28Jul2519:41-10:15(203+14:33)JuI2519:37Jul2514:56Jun2211:42-14:55(33+03:13)Jun2112:29Apr2810:54Apr2810:28Apr2810:27-10:28(00:01)Apr2810:23Apr2809:56Apr2708:58-08:58(00:00)"last"1432lines,108241charactersshutdownIftOoracleftp10.20.10.28Apr2809:56Apr2708:58-08:58(00:00)rootpts/210.10.20.15Apr2610:48-10:53(00:05)oracleftp10.20.10.28Apr2609:45-09:45(00:00)oracle即10.20.10.28Apr2609:32-09:32(00:00)oracleftp10.20.10.28Apr2608:58-08:58(00:00)oracleftp10.20.10.28Apr2509:01-09:02(00:01)oracleftp10.20.10.28Apr2408:59-09:00(00:01)oraclepts/110.20.10.60Apr2115:36-15:48(00:11)rootpts/110.0.1.199Apr2110:01-10:01(00:00)rootpts/110.0.1.199Apr2109:51-09:51(00:00)rootpts/110.0.1.199Apr2109:40-09:41(00:00)rootpts/110.0.1.199Apr2109:30-09:30(00:00)rootpts/110.0.1.199Apr2109:20-09:20(00:00)(oo：oo)££：io££：10IZJdV6600ls)dJOOJ(OO-OO)£0：ZO-£0：Z0IZrdV6600l/s】dJOOJ(OOW)£【：10£【领IZJdy6600l/s】dJOOJ(oo：oo)£ZNO-£Z：Zo13Jdv6600II/SJdIoOJ(oo：oo)日:ZO-。£：ZoIZJdV6600lI/SJdJOOJ（00：00）WZO-WZoIZJdV6600II/sJdIoal(OOW)WZo-N：Z0IZMV6600lIs)d100J(00：00)too-tooIZMV6600Isdjj(00：00)W：£0W：£0IZJdV6600l【/sidJOOJ（00：00）怙£0-怙£0IZdy6600ll/s】dJOOJ（00：00）S£：£0-S£：£013JdV6600l/s】dIOal（00：00）处£0巾£0IZMy6600lIadIoaI(OOW)£S：£0-4：£012JdV6600llsidIoaI(OOW)9炉£0Ta£0IZJdV6600ll/s】dIoaI(00：00)，$：£0-4：£0IZJdV6600lsdIOal(OO-OO)£0：W)-G0：H)IZMy6600l/s】dIOal(OOW)SIW-SI：tOIJJdy6600ll/s】dJOOJ（00：00）SZ：W)-SrtOISdV6600II/sjdJOOJ（00：00）9£：M)9£：H)IZJdV6600lIGdJOOJ(OOW)处:如一处：WIZJdV6600l，虱IOal(OOW)9筝血99：如IZJdy6600IIOal(00：00)90S0-90013Jdv6600lsdJOOJ(OOW)91WO-91SOIZJdV6600IIJd100J（00：00）9z：so-%：SOIZmV6600ll/s】dIOal（00：00）LVSQ-LVQISdV6600lIs)d)001（00：00）夕SO一夕SOUJdV6600lI/sjdJOOJ(OOW)夕£0夕SoIZW6600IIGdIOal(00：00)£0：90-£0：90IZndV6600IlsdIOal(00：00)ZJ：90-ZJ：90%dv6600lI/sJdJOOJ(00：00)LV90-LV90IsdV6600ll/sjdJOOJ(OOW)8£：90"£：90IsdV6600ll/sjdJOOJ（00：00）8t：90-8t90IsdV6600Il/sjdIOal（00：00）8G：90-8S：90IZJdV6600IlsdJOOJ（00：00）80Z0-80Z0【ZJdy6600l【/s】dIoal(OO-OO)80-8UZ.0IZJdV6600IlsdIOoJ(00：00)跖£0-8ZZoQdW6600lI/sjdjj(00：00)6£Z0-8£Z0IZIdV6600lIJdJOOJ（00：00）6，：Zx)-6,ZoIZJdV6600Il/s】dJOOJ（00：00）6SZ0-69Z0IZMV6600llJdIoal（00：00）6080-60：80UJdy6600lIadIOal(OOW)680-680IZJdV6600l【/sidIOal(OOW)63：80-6Z：80kJdV6600I，idIoaI(00：00)0t80-6£：80IZMV6600lIs)dJOOJ(OO-OO)0580-0S：80IZMV6600ll)dJOOJ（00：00）00：6000：60IZJdV6600lI/sjdJOOJ(00：00)0l：60-0l：60IZJdV6600ll/sidJOOJ(oo：oo)981-98103Jdv6600l/s)dJOOJ(oo：oo)9£：81-9£：8【OZJdV6600Il/s】dJOOJ(OO-OO)99：8l99：81OZJdV6600lIGdJOOJ(0000)99：8L99：8lOZJdV6600lIJdJOOJ(OOW)90：6l-90：6lOZJdV6600lIs)dJOOJ(OOW)Z.6I-96IOZJdV6600IIGdIOaI(OOW)LZ6l-LZ6OZMV6600IIGdIOal(0000)LV6l-一£：6lOZMV6600lIJd100J(00：00)£61-LP'6OZJdV6600l，）dJOOJ(OO-OO)二：6【一夕61OZJdV6600Il/sjdJOOJ（00：00）L0'0Z-LO-OZQZJdV6600llsdJOOJ(OO-OO)3U0Z-LUQZOZJdV6600lIJdJOOJ(OOW)LOOZ-LO-OZOZJdV6600lI/sjdJOOJ(OO-OO)SUQZ-L'QZOZIdV6600IIGdIOaI(OOW)跖OZ-WQZQZJdV6600IlJdJOOJ(OO-OO)8£：0Z-8£：0ZOZJdV6600II/sJd100l(00：00)眇：0Z-眇：0ZOZJdV6600I【/s）dJOOJ(OO-OO)89：OC-89：OZQZIdV6600II/sjdJOOJ（00：00）80：%-80：%OZJdy6600ll/s】dJOOJ(OO-OO)6I：【L81：IZoZJdV6600lIJdJOOJ（00：00）6Z"Z-61：IZOZMV6600lI/sidJOOJ(OO-OO)6£：l16£：lZ(EMV6600Il/s】dJOOJ(OOW)6R：IZ-6R：IZOZJdV6600IIJdJOOJ(OO-OO)6'Z-6Z0ZJdV6600II/sJdIoQI(OOW)60：ZZ-60：ZCOZMV6600I【/s）d)J(00：00)OVZZ-6-ZZOZJdV6600II/sjdJOOJ（00：00）QVZZ-QVZZOZJdV6600lIJdJOOJ(OO-OO)Ot-ZZ-OP-ZZOZJdV6600l【/s】dIoaI(OOW)OS-ZZ-OS-ZZOZJdV6600lI/sjdJOOJ(OOW)00：£Z-00：£ZOOMV6600lIs)dJJ(OOW)0【：£0l：£ZOZIdV66!TOO!IEdIOaI(OO-OO)0V£Z-0Z'£ZOZJdW6600lIJdIoaI(OOW)l£：£Zl£：£ZOZJdV6600I【/s】dJOOJ(00：00)【，：£，：£Z0"dv6600Ils)dJOOJ(OO-OO)【9：£Z-IS：£Z0ZldV6600ll/s】dJOOJ(OO-OO)IOW-IOWIZJdV6600lIJdJOOJ(OOW)IOO-IOOIZJdv6600IlsdJOOJ(OOW)ZVOO-IZ-OO13Jdv6600IIs)dJOOJ（00：00）Z£：00-Z£：0013Jdy6600IIGdJOOJ(OO-OO)WOO-AOOIZJdV6600IIJdIOal(OOW)Z'00-ZS-OO12Jdv6600lIJdIOaI(00：00)30：IO-ZOHOISdV661100Ils)dJOOJ(OO-OO)Zl-lO-ZlOIJJdy6600ll/s】dJOOJ(OO-OO)ZZ-IO-ZVlOIJdv6600lIJdJOOJ(OOW)££：IO££：IOIZJdV6600lIJdJOOJ（00：00）EtHO-CtnOIZJdW6600II/sjdJOOJ(oo：oo)6011I-6011IOZMv6600lIGdjj(oo：oo)6I11I-611OZJdV6600ll/s】dIooJ(OOTO)6Z：【L6Z：【lOZJdV6600Ils)dIOal(IOW)W：n9£：HOZJdv09OrOmI/s】dspujo(00：00)6£：n6£：【OsdV6600lZsdJOOJ(OOW)6tUl-6t11lOZJ6600ll/s】dJOOJ（00：00）00i-611IOZJdV6600ll/sidJOOJ(OOW)QUZl-QVZlOZJdV6600IlsdJOOJ(OOW)oz：Zl-OZmOZJdV6600lsdJOOJ(OOW)QVZl-QVZlOZJdV6600llsidIOoJ(00：00)ot-otOZMV6600lIs)dJOOJ(OO-OO)0£：ZLo£：【OSdV6600lsdIoOJ（00：00）0。m00：ZlOZJdV6600lIJdJOOJ（00：00）Ogm-OSaOoMV6600lIJdJOOJ（00：00）【0：£1-00：£IOZJdy6600lls)dJOOJ(00：00)lii-liiOZJdV6600l,idJOOJ(OOW)1乙£【乙£【OZMV6600l/s】dIoaI(00：00)【£：£【-【£：£【OIMV6600II/sJdJOOJ(OO-OO)【，：£【-，£【OSdV6600lI/MdJOOJ（00：00）"：£【-【£：£【OZJdV6600Il/s】dJOOJ(OOW)【0：。LlOMOZJdV6600IIs)dJOOJ(OO-OO)ZVPl-ZPtlOZJdV6600lIs)dJOOJ(OOW)ZVPl-ZVtlOZJdV6600lIGdIOal(OOW)ZVPl-在MOZJdV6600l，虱IOal(00：00)Z力力LZttlOZJdV6600l/s)djj(OO-OO)ZSWTS：WOZJdV6600I/ndJOOJ（00：00）ZO-Sl-ZO-SIOZJdV6600Il/s】dJOOJ(OOW)£l：Sl£【：SlOZJdV6600llsdJOOJ（00：00）口:GL口:SlOZMv6600lI/sidJOOJ(OOW)££：外££：G【OZJdV6600IIGdJOOJ（00：00）£，：£【-£，：SlOZldV6600Il/s】d100J(00：00)£S：SL£G：SlOSdV6600lsdIOal(00：00)£0：9l-£0：9IOZJdV6600l/s)dJOOJ(00：00)W：9lW：9lOZJdV6600llsdJOOJ(OOW)怙9怙91OZMV6600ll/sjdJOOJ（00：00）沱9)，£：91OZJdV6600II/sjdIOal（00：00）处：9【F：9【OZJdV6600llsdJOOJ(OO-OO)K：9LB：91OZJdV6600lsd100J(OOW)WLl-WLIOZJdV6600IsdIOal(00：00)l'Ll-LlOZMW6600lIs)dJOOJ(00：00)SVLl-Z'LlOzJdV6600llsdJOOJ（00：00）G£：ZJ-S£：ZjOZJdV6600ll/s】dJOOJ（00：00）p：Ll-t'LlOZJdV6600lsdJOOJ（00：00）-uOZJdV6600lI/sidIOal（00：00）S0：8l-Go：81OZJdV6600llsdJOOJ（00：00）9【：8【S【：8lOZJdV6600lsd100Jrootpts/110.0.1.199Apr2010:58-10:59(00:00)rootpts/110.0.1.199Apr2010:48-10:48(00:00)rootpts/110.0.1.199Apr2010:38-10:38(00:00)rootpts/110.0.1.199Apr2010:28-10:28(00:00)rootpts/110.0.1.199Apr2010:18-10:18(00:00)rootpts/110.0.1.199Apr2010:08-10:08(00:00)rootpts/110.0.1.199Apr2009:58-09:58(00:00)rootpts/110.0.1.199Apr2009:47-09:47(00:00)rootpts/110.0.1.199Apr2009:37-09:37(00:00)rootpts/110.0.1.199Apr2009:27-09:27(00:00)rootpts/110.0.1.199Apr2009:17-09:17(00:00)rootpts/110.0.1.199Apr2009:07-09:07(00:00)rootpts/110.0.1.199Apr2008:57-08:57(00:00)rootpts/110.0.1.199Apr2008:46-08:46(00:00)rootpts/110.0.1.199Apr2008:36-08:36(00:00)rootpts/110.0.1.199Apr2008:26-08:26(00:00)rootpts/110.0.1.199Apr2008:16-08:16(00:00)rootpts/110.0.1.199Apr2008:06-08:06(00:00)rootpts/110.0.1.199Apr2007:56-07:56(00:00)rootpts/110.0.1