Wissensdatenbank
Buehl, Kevin
28 April 2016
3.652

Wissensdatenbank

HP ProLiant DL180 G6 Kernel Panic - general protection fault

Inhaltsverzeichnis

Situation

Ich hatte über mehrere Monate regelmässig bei meinem Hewlett-Packard Server eine Kernel Panic. Zu beginn habe ich diese alle paar Tage erhalten, meistens um 1 Uhr nachts wenn mein Rsync Skript meine Daten auf einen externen Server kopiert hat. Nachdem ich dieses Skript nicht mehr ausgeführt habe, traten die Kernel Panics nicht mehr auf. Ich habe dann ein Skript erstellt, was meine Dateien in gleich grosse Archive packt. Dieser Vorgang erhörte die Systemauslastung extrem, was für einen Server im Normalfall zu keinen Problem führen sollte. Auf meinem System sah das allerdings ganz anders aus, ich habe dann jede Nacht zwischen 1 und 3 Uhr eine Kernel Panic erhalten.
Da ich in der ILO Konsole nur eine Auflösung von 640x480 Pixel habe und die Kernel Panic Logs sehr lang werden können, musste ich eine Lösung suchen diese auslesen zu können. Da bei meinen Kernel Panic's nicht mehr auf die Festplatten und Netzwerkschnittstellen zugegriffen werden konnte, habe ich die komplette Ausgabe des Systems über die RS232 (ttyS0) Schnittstelle des Servers ausgegeben. Mit einem Nullmodem-Kabel habe ich diese an einem anderen Server wieder eingelesen (
Dieser Link wurde entfernt. Links sind nur für Mitglieder sichtbar. Bitte melden Sie sich mit Ihrem Benutzernamen und Ihrem Passwort an und versuchen Sie es erneut.
). Die Ausgabe des kompletten Kernel Panics ist wie folgt:
[43950.168164] general protection fault: 0000 [#1] SMP
[43950.173777] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[43950.250561] CPU: 5 PID: 2 Comm: kthreadd Tainted: G I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43950.261228] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43950.268564] task: ffff8802374da960 ti: ffff8802374f8000 task.ti: ffff8802374f8000
[43950.276994] RIP: 0010:[<ffffffff815144fb>] [<ffffffff815144fb>] _raw_spin_lock_irqsave+0x1b/0x50
[43950.286999] RSP: 0018:ffff8802374fba50 EFLAGS: 00010046
[43950.292973] RAX: 0000000000000046 RBX: ffffea0007e0e578 RCX: 0000000000000001
[43950.300979] RDX: 0000000000010000 RSI: 0000000000000046 RDI: 02ffff8000000098
[43950.308994] RBP: 02ffff8000000080 R08: 00000000ffffffee R09: 0000000000001202
[43950.318276] R10: 000000000000374c R11: ffffffffffffffc8 R12: 02ffff8000000098
[43950.326290] R13: 0000000000240600 R14: ffff88043ffeee00 R15: 0000000000000001
[43950.334305] FS: 0000000000000000(0000) GS:ffff88043fc20000(0000) knlGS:0000000000000000
[43950.343416] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43950.349875] CR2: 00007ff2926031bc CR3: 0000000001813000 CR4: 00000000000007e0
[43950.357890] Stack:
[43950.360168] ffffffff81161007 0000000000240419 ffffea0007e0e578 000000000000001a
[43950.368550] ffffffff8116032e ffffea0007e0e000 ffff8802374fbb60 0000000000001202
[43950.376932] ffff8802374fbfd8 0000000000000000 0000000000000017 0000000000000016
[43950.385304] Call Trace:
[43950.388073] [<ffffffff81161007>] ? balloon_page_isolate+0x97/0x110
[43950.395119] [<ffffffff8116032e>] ? isolate_migratepages_range+0x89e/0x8c0
[43950.402845] [<ffffffff8116061d>] ? compact_zone+0x22d/0x4a0
[43950.409208] [<ffffffff8116091e>] ? compact_zone_order+0x8e/0xd0
[43950.415978] [<ffffffff81160bc5>] ? try_to_compact_pages+0xc5/0xf0
[43950.422929] [<ffffffff8150c854>] ? __alloc_pages_direct_compact+0xab/0x1a3
[43950.430750] [<ffffffff81159892>] ? next_online_pgdat+0x22/0x50
[43950.437406] [<ffffffff8115be6b>] ? wait_iff_congested+0xab/0x150
[43950.444254] [<ffffffff81146d65>] ? __alloc_pages_nodemask+0x645/0xb30
[43950.451592] [<ffffffff81064cc6>] ? copy_process.part.25+0x116/0x1c30
[43950.458834] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43950.466173] [<ffffffff8101155e>] ? __switch_to+0xde/0x570
[43950.472340] [<ffffffff810669b1>] ? do_fork+0xe1/0x390
[43950.478120] [<ffffffff81510d61>] ? __schedule+0x2b1/0x700
[43950.484280] [<ffffffff81066c82>] ? kernel_thread+0x22/0x30
[43950.490547] [<ffffffff810888dd>] ? kthreadd+0x13d/0x190
[43950.496521] [<ffffffff810887a0>] ? kthread_create_on_cpu+0x60/0x60
[43950.503564] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43950.509830] [<ffffffff810887a0>] ? kthread_create_on_cpu+0x60/0x60
[43950.516871] Code: eb f2 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 9c 58 66 66 90 66 90 48 89 c6 fa 66 66 90 66 66 90 ba 00 00 01 00 <f0> 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f
[43950.538937] RIP [<ffffffff815144fb>] _raw_spin_lock_irqsave+0x1b/0x50
[43950.546285] RSP <ffff8802374fba50>
[43950.550700] ---[ end trace 15b5c8255c358929 ]---
[43959.251470] ------------[ cut here ]------------
[43959.256740] WARNING: CPU: 7 PID: 2545 at /build/linux-zbOcpU/linux-3.16.7-ckt25/kernel/watchdog.c:265 watchd og_overflow_callback+0x98/0xc0()
[43959.270959] Watchdog detected hard LOCKUP on cpu 7
[43959.276188] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[43959.358999] CPU: 7 PID: 2545 Comm: systemd Tainted: G D I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43959.369922] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43959.377318] 0000000000000000 ffffffff8150e835 ffff88043fc65c98 0000000000000009
[43959.386005] ffffffff810677f7 ffff88043eb58000 ffff88043fc65ce8 ffff88043fc65d88
[43959.394688] 0000000000000000 ffff88043fc65ef8 ffffffff8106785c ffffffff8171d648
[43959.403374] Call Trace:
[43959.406197] <NMI> [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
[43959.413066] [<ffffffff810677f7>] ? warn_slowpath_common+0x77/0x90
[43959.420073] [<ffffffff8106785c>] ? warn_slowpath_fmt+0x4c/0x50
[43959.426857] [<ffffffff810fd1f8>] ? watchdog_overflow_callback+0x98/0xc0
[43959.434451] [<ffffffff81134ae7>] ? __perf_event_overflow+0x87/0x230
[43959.441654] [<ffffffff8102f89a>] ? intel_pmu_handle_irq+0x1ea/0x3d0
[43959.448859] [<ffffffff8134ba43>] ? ghes_copy_tofrom_phys+0x113/0x210
[43959.456160] [<ffffffff810274a6>] ? perf_event_nmi_handler+0x26/0x40
[43959.463363] [<ffffffff8101691d>] ? nmi_handle+0x8d/0x120
[43959.469494] [<ffffffff81016e43>] ? default_do_nmi+0x43/0x110
[43959.476006] [<ffffffff81017004>] ? do_nmi+0xf4/0x170
[43959.481749] [<ffffffff81516e0f>] ? end_repeat_nmi+0x1a/0x1e
[43959.488173] [<ffffffff8151451c>] ? _raw_spin_lock_irqsave+0x3c/0x50
[43959.495373] [<ffffffff8151451c>] ? _raw_spin_lock_irqsave+0x3c/0x50
[43959.502574] [<ffffffff8151451c>] ? _raw_spin_lock_irqsave+0x3c/0x50
[43959.509774] <<EOE>> [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43959.517793] [<ffffffff8114b0f0>] ? __activate_page+0x1d0/0x1d0
[43959.524508] [<ffffffff8114b4b1>] ? __lru_cache_add+0x51/0x80
[43959.531031] [<ffffffff81167608>] ? do_wp_page+0x288/0x7c0
[43959.537259] [<ffffffff81169d9d>] ? handle_mm_fault+0x84d/0x11c0
[43959.544066] [<ffffffff810572b7>] ? __do_page_fault+0x177/0x4f0
[43959.550784] [<ffffffff811be3df>] ? dput+0x1f/0x170
[43959.556332] [<ffffffff81516a28>] ? page_fault+0x28/0x30
[43959.562364] ---[ end trace 15b5c8255c35892a ]---
[43963.090443] ------------[ cut here ]------------
[43963.095704] WARNING: CPU: 4 PID: 28599 at /build/linux-zbOcpU/linux-3.16.7-ckt25/kernel/watchdog.c:265 watch dog_overflow_callback+0x98/0xc0()
[43963.110030] Watchdog detected hard LOCKUP on cpu 4
[43963.115260] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[43963.197929] CPU: 4 PID: 28599 Comm: kworker/u48:2 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ck t25-1
[43963.209531] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43963.216930] Workqueue: khelper __call_usermodehelper
[43963.222648] 0000000000000000 ffffffff8150e835 ffff88043fc05c98 0000000000000009
[43963.231337] ffffffff810677f7 ffff88043ea41400 ffff88043fc05ce8 ffff88043fc05d88
[43963.240027] 0000000000000000 ffff88043fc05ef8 ffffffff8106785c ffffffff8171d648
[43963.248718] Call Trace:
[43963.251533] <NMI> [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
[43963.258397] [<ffffffff810677f7>] ? warn_slowpath_common+0x77/0x90
[43963.265403] [<ffffffff8106785c>] ? warn_slowpath_fmt+0x4c/0x50
[43963.272119] [<ffffffff810fd1f8>] ? watchdog_overflow_callback+0x98/0xc0
[43963.279711] [<ffffffff81134ae7>] ? __perf_event_overflow+0x87/0x230
[43963.286911] [<ffffffff8102f89a>] ? intel_pmu_handle_irq+0x1ea/0x3d0
[43963.294113] [<ffffffff8134ba43>] ? ghes_copy_tofrom_phys+0x113/0x210
[43963.301411] [<ffffffff810274a6>] ? perf_event_nmi_handler+0x26/0x40
[43963.308611] [<ffffffff8101691d>] ? nmi_handle+0x8d/0x120
[43963.314741] [<ffffffff81016e43>] ? default_do_nmi+0x43/0x110
[43963.321260] [<ffffffff81017004>] ? do_nmi+0xf4/0x170
[43963.326994] [<ffffffff81516e0f>] ? end_repeat_nmi+0x1a/0x1e
[43963.333418] [<ffffffff815142ca>] ? _raw_spin_lock_irq+0x2a/0x40
[43963.340230] [<ffffffff815142ca>] ? _raw_spin_lock_irq+0x2a/0x40
[43963.347098] [<ffffffff815142ca>] ? _raw_spin_lock_irq+0x2a/0x40
[43963.353909] <<EOE>> [<ffffffff8115077a>] ? shrink_inactive_list+0xca/0x500
[43963.362028] [<ffffffff81151451>] ? shrink_lruvec+0x511/0x6a0
[43963.368540] [<ffffffff81151654>] ? shrink_zone+0x74/0x1b0
[43963.374768] [<ffffffff81151b8d>] ? do_try_to_free_pages+0x12d/0x520
[43963.381969] [<ffffffff8115203d>] ? try_to_free_pages+0xbd/0x150
[43963.388782] [<ffffffff81146e1e>] ? __alloc_pages_nodemask+0x6fe/0xb30
[43963.396178] [<ffffffff81064ca5>] ? copy_process.part.25+0xf5/0x1c30
[43963.403381] [<ffffffff81064cc6>] ? copy_process.part.25+0x116/0x1c30
[43963.410679] [<ffffffff8107da90>] ? ____call_usermodehelper+0x130/0x130
[43963.418171] [<ffffffff810669b1>] ? do_fork+0xe1/0x390
[43963.424014] [<ffffffff810a29dc>] ? pick_next_task_fair+0x19c/0x820
[43963.431118] [<ffffffff8101155e>] ? __switch_to+0xde/0x570
[43963.437347] [<ffffffff81066c82>] ? kernel_thread+0x22/0x30
[43963.443674] [<ffffffff8107db09>] ? __call_usermodehelper+0x29/0x90
[43963.450778] [<ffffffff810817c2>] ? process_one_work+0x172/0x420
[43963.457590] [<ffffffff81081e53>] ? worker_thread+0x113/0x4f0
[43963.464109] [<ffffffff81510d61>] ? __schedule+0x2b1/0x700
[43963.470337] [<ffffffff81081d40>] ? rescuer_thread+0x2d0/0x2d0
[43963.476954] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43963.482697] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43963.490091] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43963.496418] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43963.503813] ---[ end trace 15b5c8255c35892b ]---
[43965.682760] ------------[ cut here ]------------
[43965.688021] WARNING: CPU: 6 PID: 68 at /build/linux-zbOcpU/linux-3.16.7-ckt25/kernel/watchdog.c:265 watchdog _overflow_callback+0x98/0xc0()
[43965.702055] Watchdog detected hard LOCKUP on cpu 6
[43965.707283] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[43965.789967] CPU: 6 PID: 68 Comm: kswapd1 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43965.800693] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43965.808087] 0000000000000000 ffffffff8150e835 ffff88043fc45c98 0000000000000009
[43965.816775] ffffffff810677f7 ffff88043f464400 ffff88043fc45ce8 ffff88043fc45d88
[43965.825465] 0000000000000000 ffff88043fc45ef8 ffffffff8106785c ffffffff8171d648
[43965.834156] Call Trace:
[43965.836980] <NMI> [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
[43965.843835] [<ffffffff810677f7>] ? warn_slowpath_common+0x77/0x90
[43965.850843] [<ffffffff8106785c>] ? warn_slowpath_fmt+0x4c/0x50
[43965.857558] [<ffffffff810fd1f8>] ? watchdog_overflow_callback+0x98/0xc0
[43965.865149] [<ffffffff81134ae7>] ? __perf_event_overflow+0x87/0x230
[43965.872351] [<ffffffff8102f89a>] ? intel_pmu_handle_irq+0x1ea/0x3d0
[43965.880816] [<ffffffff8134ba43>] ? ghes_copy_tofrom_phys+0x113/0x210
[43965.888115] [<ffffffff810274a6>] ? perf_event_nmi_handler+0x26/0x40
[43965.895317] [<ffffffff8101691d>] ? nmi_handle+0x8d/0x120
[43965.901448] [<ffffffff81016e43>] ? default_do_nmi+0x43/0x110
[43965.907960] [<ffffffff81017004>] ? do_nmi+0xf4/0x170
[43965.913704] [<ffffffff81516e0f>] ? end_repeat_nmi+0x1a/0x1e
[43965.920126] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43965.927329] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43965.934530] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43965.941729] <<EOE>> [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43965.949760] [<ffffffff8114b0f0>] ? __activate_page+0x1d0/0x1d0
[43965.956474] [<ffffffff8114b9de>] ? lru_add_drain_cpu+0xce/0xe0
[43965.963188] [<ffffffff8114baf2>] ? lru_add_drain+0x12/0x20
[43965.969514] [<ffffffff81150747>] ? shrink_inactive_list+0x97/0x500
[43965.976617] [<ffffffff81151451>] ? shrink_lruvec+0x511/0x6a0
[43965.983138] [<ffffffff81151654>] ? shrink_zone+0x74/0x1b0
[43965.989368] [<ffffffff8115276e>] ? balance_pgdat+0x38e/0x5c0
[43965.995888] [<ffffffff81152aff>] ? kswapd+0x15f/0x3f0
[43966.001730] [<ffffffff810a7e60>] ? prepare_to_wait_event+0xf0/0xf0
[43966.008833] [<ffffffff811529a0>] ? balance_pgdat+0x5c0/0x5c0
[43966.015354] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43966.021096] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43966.028490] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43966.034815] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43966.042210] ---[ end trace 15b5c8255c35892c ]---
[43967.976293] ------------[ cut here ]------------
[43967.981566] WARNING: CPU: 3 PID: 1668 at /build/linux-zbOcpU/linux-3.16.7-ckt25/kernel/watchdog.c:265 watchd og_overflow_callback+0x98/0xc0()
[43967.995794] Watchdog detected hard LOCKUP on cpu 3
[43968.001025] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[43968.083708] CPU: 3 PID: 1668 Comm: nfsd Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43968.094338] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43968.101735] 0000000000000000 ffffffff8150e835 ffff88023fc65c98 0000000000000009
[43968.110424] ffffffff810677f7 ffff880237018c00 ffff88023fc65ce8 ffff88023fc65d88
[43968.119114] 0000000000000000 ffff88023fc65ef8 ffffffff8106785c ffffffff8171d648
[43968.127796] Call Trace:
[43968.130613] <NMI> [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
[43968.137484] [<ffffffff810677f7>] ? warn_slowpath_common+0x77/0x90
[43968.144493] [<ffffffff8106785c>] ? warn_slowpath_fmt+0x4c/0x50
[43968.151211] [<ffffffff810fd1f8>] ? watchdog_overflow_callback+0x98/0xc0
[43968.158805] [<ffffffff81134ae7>] ? __perf_event_overflow+0x87/0x230
[43968.166000] [<ffffffff8102f89a>] ? intel_pmu_handle_irq+0x1ea/0x3d0
[43968.173204] [<ffffffff8134ba43>] ? ghes_copy_tofrom_phys+0x113/0x210
[43968.180504] [<ffffffff810274a6>] ? perf_event_nmi_handler+0x26/0x40
[43968.187707] [<ffffffff8101691d>] ? nmi_handle+0x8d/0x120
[43968.193839] [<ffffffff81016e43>] ? default_do_nmi+0x43/0x110
[43968.200361] [<ffffffff81017004>] ? do_nmi+0xf4/0x170
[43968.206106] [<ffffffff81516e0f>] ? end_repeat_nmi+0x1a/0x1e
[43968.212534] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43968.219737] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43968.226940] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[43968.234143] <<EOE>> [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43968.242176] [<ffffffff8114af20>] ? __pagevec_lru_add+0x20/0x20
[43968.248892] [<ffffffff8114ba26>] ? __pagevec_release+0x16/0x40
[43968.255619] [<ffffffffa01e2cdd>] ? mpage_prepare_extent_to_map+0x24d/0x270 [ext4]
[43968.264214] [<ffffffffa018c8b4>] ? jbd2__journal_start+0xf4/0x1e0 [jbd2]
[43968.271909] [<ffffffffa01e6b2f>] ? ext4_writepages+0x3df/0xd00 [ext4]
[43968.279308] [<ffffffff811e382c>] ? fsnotify+0x23c/0x300
[43968.285343] [<ffffffff811a9811>] ? do_readv_writev+0x171/0x240
[43968.292060] [<ffffffff8113e9ce>] ? __filemap_fdatawrite_range+0x4e/0x60
[43968.299652] [<ffffffff8113eaa5>] ? filemap_write_and_wait_range+0x25/0x60
[43968.307443] [<ffffffffa01de5a1>] ? ext4_sync_file+0xb1/0x310 [ext4]
[43968.314653] [<ffffffffa05a4a80>] ? nfsd_vfs_write.isra.12+0x260/0x360 [nfsd]
[43968.322735] [<ffffffffa05a7fb1>] ? nfsd_write+0xf1/0x110 [nfsd]
[43968.329554] [<ffffffffa05ade6c>] ? nfsd3_proc_write+0xac/0x130 [nfsd]
[43968.336946] [<ffffffffa05a0d32>] ? nfsd_dispatch+0xb2/0x200 [nfsd]
[43968.344052] [<ffffffffa04e3b1b>] ? svc_process_common+0x41b/0x670 [sunrpc]
[43968.351940] [<ffffffffa05a0630>] ? nfsd_destroy+0x70/0x70 [nfsd]
[43968.358855] [<ffffffffa04e3e7c>] ? svc_process+0x10c/0x160 [sunrpc]
[43968.366062] [<ffffffffa05a06ef>] ? nfsd+0xbf/0x130 [nfsd]
[43968.372292] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43968.378036] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43968.385432] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43968.391760] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43968.399156] ---[ end trace 15b5c8255c35892d ]---
[43971.187265] INFO: rcu_sched detected stalls on CPUs/tasks: { 6} (detected by 0, t=5255 jiffies, g=970677, c= 970676, q=9937)
[43971.200154] sending NMI to all CPUs:
[43971.204249] NMI backtrace for cpu 0
[43971.208244] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43971.219070] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43971.226466] task: ffffffff8181a460 ti: ffffffff81800000 task.ti: ffffffff81800000
[43971.234956] RIP: 0010:[<ffffffff812b6f9d>] [<ffffffff812b6f9d>] delay_tsc+0x3d/0x70
[43971.243818] RSP: 0018:ffff88023fc03cf0 EFLAGS: 00000046
[43971.249854] RAX: 0000000000000000 RBX: 00000000ed3e0d8c RCX: 0000000000f42190
[43971.257931] RDX: 00000000ed3e0e36 RSI: 0000000000000200 RDI: 00000000000306fa
[43971.266006] RBP: 00000000000306fa R08: ffffffff818e28e0 R09: 00000000000009d0
[43971.274081] R10: 0000000000000000 R11: ffff88023fc03aa6 R12: 0000000000000000
[43971.282158] R13: 0000000000000096 R14: 0000000000000002 R15: 0000000000000002
[43971.290236] FS: 0000000000000000(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
[43971.299409] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43971.305930] CR2: 00007f9b2f9bf000 CR3: 0000000001813000 CR4: 00000000000007f0
[43971.314007] Stack:
[43971.316345] 00000000000003e9 0000000000001000 0000000000000400 ffffffff81044d52
[43971.325028] 000000000000a0ea ffffffff818e28e0 ffffffff81046769 0000000000000001
[43971.333716] ffff88023fc0d660 ffffffff81853780 0000000000000000 ffffffff818e2e40
[43971.342406] Call Trace:
[43971.345222] <IRQ>
[43971.347439] [<ffffffff81044d52>] ? native_safe_apic_wait_icr_idle+0x22/0x50
[43971.355792] [<ffffffff81046769>] ? default_send_IPI_mask_sequence_phys+0xd9/0xe0
[43971.364285] [<ffffffff810469c3>] ? arch_trigger_all_cpu_backtrace+0xc3/0x140
[43971.372366] [<ffffffff810c5add>] ? rcu_check_callbacks+0x60d/0x630
[43971.379474] [<ffffffff810d02d0>] ? tick_sched_handle.isra.16+0x60/0x60
[43971.386972] [<ffffffff81074ba0>] ? update_process_times+0x40/0x70
[43971.393981] [<ffffffff810d0290>] ? tick_sched_handle.isra.16+0x20/0x60
[43971.401474] [<ffffffff810d030c>] ? tick_sched_timer+0x3c/0x60
[43971.408092] [<ffffffff8108b187>] ? __run_hrtimer+0x67/0x1c0
[43971.414516] [<ffffffff8108b539>] ? hrtimer_interrupt+0xe9/0x220
[43971.421330] [<ffffffff810cee13>] ? tick_do_broadcast.constprop.7+0x93/0xa0
[43971.429214] [<ffffffff810ceff1>] ? tick_handle_oneshot_broadcast+0x141/0x1c0
[43971.437294] [<ffffffff81015741>] ? timer_interrupt+0x11/0x20
[43971.443817] [<ffffffff810bb375>] ? handle_irq_event_percpu+0x35/0x190
[43971.451213] [<ffffffff810bb508>] ? handle_irq_event+0x38/0x60
[43971.457831] [<ffffffff810bdfd5>] ? handle_edge_irq+0x85/0x150
[43971.464450] [<ffffffff810150cd>] ? handle_irq+0x1d/0x30
[43971.470488] [<ffffffff81517819>] ? do_IRQ+0x49/0xe0
[43971.476137] [<ffffffff8151566d>] ? common_interrupt+0x6d/0x6d
[43971.482754] <EOI>
[43971.484970] [<ffffffff813e011f>] ? cpuidle_enter_state+0x4f/0xc0
[43971.492253] [<ffffffff813e0118>] ? cpuidle_enter_state+0x48/0xc0
[43971.499165] [<ffffffff810a8398>] ? cpu_startup_entry+0x2f8/0x400
[43971.506076] [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
[43971.512500] [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
[43971.518731] [<ffffffff81902120>] ? early_idt_handler_array+0x120/0x120
[43971.526225] [<ffffffff8190271f>] ? x86_64_start_kernel+0x14d/0x15c
[43971.533329] Code: 00 00 66 66 90 0f ae e8 e8 11 44 d6 ff 66 90 89 c3 eb 12 0f 1f 00 f3 90 65 8b 04 25 e4 a0 00 00 41 39 c4 75 1a 66 66 90 0f ae e8 <e8> ee 43 d6 ff 66 90 89 c2 29 d8 39 e8 72 dc 5b 5d 41 5c c3 29
[43971.559257] NMI backtrace for cpu 1
[43971.563252] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43971.574076] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43971.581471] task: ffff88023756c0d0 ti: ffff880237598000 task.ti: ffff880237598000
[43971.589961] RIP: 0010:[<ffffffff8131e492>] [<ffffffff8131e492>] intel_idle+0xc2/0x140
[43971.599014] RSP: 0018:ffff88023759be78 EFLAGS: 00000046
[43971.605048] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[43971.613124] RDX: 0000000000000000 RSI: ffff88023759bfd8 RDI: 0000000000000046
[43971.621198] RBP: 0000000000000004 R08: 0000000000005c5e R09: 0000000000000018
[43971.629274] R10: 0000000000013aa0 R11: 0000000000000006 R12: 0000000000000020
[43971.637347] R13: 0000000000000003 R14: 0000000000000004 R15: ffffffff8189e9f8
[43971.645423] FS: 0000000000000000(0000) GS:ffff88023fc20000(0000) knlGS:0000000000000000
[43971.654595] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43971.661106] CR2: 00007f9b2f9bf000 CR3: 0000000001813000 CR4: 00000000000007e0
[43971.669171] Stack:
[43971.671510] 000000018101b3d5 ffff88023fc39300 ffff88023759bed0 ffffffff8189e880
[43971.680197] 000027fd61460c86 ffffffff813e0110 ffffffff818e2e00 ffff880237598000
[43971.688884] ffffffff8189e880 0000000000000004 0000000000000000 ffff88023fc39300
[43971.697571] Call Trace:
[43971.700397] [<ffffffff813e0110>] ? cpuidle_enter_state+0x40/0xc0
[43971.707308] [<ffffffff810a8398>] ? cpu_startup_entry+0x2f8/0x400
[43971.714216] [<ffffffff81042cbf>] ? start_secondary+0x20f/0x2d0
[43971.720929] Code: 31 d2 65 48 8b 34 25 88 b8 00 00 48 89 d1 48 8d 86 38 c0 ff ff 0f 01 c8 48 8b 86 38 c0 ff ff a8 08 75 08 b1 01 4c 89 e0 0f 01 c9 <65> 48 8b 0c 25 88 b8 00 00 f0 80 a1 3a c0 ff ff df 0f ae f0 48
[43971.748212] NMI backtrace for cpu 2
[43971.752208] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43971.763032] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43971.770428] task: ffff88023759d3b0 ti: ffff8802375a0000 task.ti: ffff8802375a0000
[43971.778919] RIP: 0010:[<ffffffff81514519>] [<ffffffff81514519>] _raw_spin_lock_irqsave+0x39/0x50
[43971.789044] RSP: 0018:ffff8802375a3e18 EFLAGS: 00000002
[43971.795076] RAX: 0000000000002237 RBX: 0000000000000002 RCX: 0000000000002237
[43971.803150] RDX: 0000000000002236 RSI: 0000000000000002 RDI: ffffffff81a8fd44
[43971.811226] RBP: ffff88023fc4cd80 R08: 0000000000003dc9 R09: 0000000000000018
[43971.819299] R10: 0000000000013545 R11: 0000000000000006 R12: 0000000000000005
[43971.827374] R13: ffff8802375a3e7c R14: ffffffff8182c340 R15: ffffffff8189e9f8
[43971.835449] FS: 0000000000000000(0000) GS:ffff88023fc40000(0000) knlGS:0000000000000000
[43971.844618] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43971.851139] CR2: 00007fa9ad391000 CR3: 0000000001813000 CR4: 00000000000007e0
[43971.859212] Stack:
[43971.861550] ffffffff810cf4c6 0000000000000005 0000000000000004 0000000000000046
[43971.870237] ffff8802375a3e7c 0000000000000004 ffffffff810cdf20 0000000000000008
[43971.878924] 0000000000000004 0000000000000020 0000000000000003 ffffffff8131e4d4
[43971.887612] Call Trace:
[43971.890438] [<ffffffff810cf4c6>] ? tick_broadcast_oneshot_control+0x66/0x200
[43971.898515] [<ffffffff810cdf20>] ? clockevents_notify+0x1d0/0x220
[43971.905521] [<ffffffff8131e4d4>] ? intel_idle+0x104/0x140
[43971.911751] [<ffffffff813e0110>] ? cpuidle_enter_state+0x40/0xc0
[43971.918660] [<ffffffff810a8398>] ? cpu_startup_entry+0x2f8/0x400
[43971.925568] [<ffffffff81042cbf>] ? start_secondary+0x20f/0x2d0
[43971.932282] Code: c6 fa 66 66 90 66 66 90 ba 00 00 01 00 f0 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f b7 17 66 39 d1 74 f4 f3 90 <0f> b7 17 66 39 d0 75 f6 eb e8 66 66 66 66 2e 0f 1f 84 00 00 00
[43971.958323] NMI backtrace for cpu 3
[43971.962315] CPU: 3 PID: 1668 Comm: nfsd Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43971.972943] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43971.980337] task: ffff8804351815f0 ti: ffff880435184000 task.ti: ffff880435184000
[43971.988826] RIP: 0010:[<ffffffff81514519>] [<ffffffff81514519>] _raw_spin_lock_irqsave+0x39/0x50
[43971.998949] RSP: 0018:ffff880435187a48 EFLAGS: 00000012
[43972.004981] RAX: 000000000000f512 RBX: 0000000000000000 RCX: 000000000000f512
[43972.013055] RDX: 000000000000f50e RSI: 0000000000000246 RDI: ffff88043ffef280
[43972.021131] RBP: ffffea00085ea3d8 R08: 0000000000000000 R09: a8001e852d000000
[43972.029206] R10: 57ffcb7adaa14b40 R11: ffff880237334608 R12: ffff88023fc6ef20
[43972.037283] R13: 0000000000000000 R14: ffff88043ffeee00 R15: ffff88043ffeee00
[43972.045357] FS: 0000000000000000(0000) GS:ffff88023fc60000(0000) knlGS:0000000000000000
[43972.054530] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43972.061048] CR2: 00007f61d9d88000 CR3: 0000000001813000 CR4: 00000000000007e0
[43972.069124] Stack:
[43972.071460] ffffffff8114ae43 ffffffff8114af20 0000000000000000 ffff880435187ac8
[43972.080146] 7fffffffffffffff 0000000000000000 ffff880435187ae0 ffff880435187c10
[43972.088834] ffffea0007a14b40 ffffffff8114ba26 ffff880435187ae0 ffffffffa01e2cdd
[43972.097521] Call Trace:
[43972.100347] [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43972.107344] [<ffffffff8114af20>] ? __pagevec_lru_add+0x20/0x20
[43972.114057] [<ffffffff8114ba26>] ? __pagevec_release+0x16/0x40
[43972.120787] [<ffffffffa01e2cdd>] ? mpage_prepare_extent_to_map+0x24d/0x270 [ext4]
[43972.129378] [<ffffffffa018c8b4>] ? jbd2__journal_start+0xf4/0x1e0 [jbd2]
[43972.137071] [<ffffffffa01e6b2f>] ? ext4_writepages+0x3df/0xd00 [ext4]
[43972.144469] [<ffffffff811e382c>] ? fsnotify+0x23c/0x300
[43972.150503] [<ffffffff811a9811>] ? do_readv_writev+0x171/0x240
[43972.157220] [<ffffffff8113e9ce>] ? __filemap_fdatawrite_range+0x4e/0x60
[43972.164812] [<ffffffff8113eaa5>] ? filemap_write_and_wait_range+0x25/0x60
[43972.172602] [<ffffffffa01de5a1>] ? ext4_sync_file+0xb1/0x310 [ext4]
[43972.179809] [<ffffffffa05a4a80>] ? nfsd_vfs_write.isra.12+0x260/0x360 [nfsd]
[43972.187880] [<ffffffffa05a7fb1>] ? nfsd_write+0xf1/0x110 [nfsd]
[43972.194695] [<ffffffffa05ade6c>] ? nfsd3_proc_write+0xac/0x130 [nfsd]
[43972.202092] [<ffffffffa05a0d32>] ? nfsd_dispatch+0xb2/0x200 [nfsd]
[43972.209202] [<ffffffffa04e3b1b>] ? svc_process_common+0x41b/0x670 [sunrpc]
[43972.217086] [<ffffffffa05a0630>] ? nfsd_destroy+0x70/0x70 [nfsd]
[43972.223999] [<ffffffffa04e3e7c>] ? svc_process+0x10c/0x160 [sunrpc]
[43972.231203] [<ffffffffa05a06ef>] ? nfsd+0xbf/0x130 [nfsd]
[43972.237433] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43972.243174] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43972.250571] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43972.256898] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43972.264292] Code: c6 fa 66 66 90 66 66 90 ba 00 00 01 00 f0 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f b7 17 66 39 d1 74 f4 f3 90 <0f> b7 17 66 39 d0 75 f6 eb e8 66 66 66 66 2e 0f 1f 84 00 00 00
[43972.290331] NMI backtrace for cpu 4
[43972.294324] CPU: 4 PID: 28599 Comm: kworker/u48:2 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ck t25-1
[43972.305927] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43972.313324] Workqueue: khelper __call_usermodehelper
[43972.319042] task: ffff8802358009a0 ti: ffff88005d284000 task.ti: ffff88005d284000
[43972.327533] RIP: 0010:[<ffffffff815142ca>] [<ffffffff815142ca>] _raw_spin_lock_irq+0x2a/0x40
[43972.337266] RSP: 0018:ffff88005d287890 EFLAGS: 00000097
[43972.343298] RAX: 000000000000f50e RBX: ffff88043ffeee00 RCX: 000000000000f50f
[43972.351374] RDX: 000000000000f50f RSI: 0000000000000000 RDI: ffff88043ffef280
[43972.359450] RBP: ffff88005d287b18 R08: ffff8802358009a0 R09: 0000000000000000
[43972.367526] R10: 0000000000000000 R11: 0000000000000006 R12: ffff88005d287908
[43972.375603] R13: 0000000000000020 R14: ffff88043ffef288 R15: 0000000000000002
[43972.383679] FS: 0000000000000000(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
[43972.392850] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43972.399371] CR2: 00007f9b2f9bf000 CR3: 0000000001813000 CR4: 00000000000007e0
[43972.407447] Stack:
[43972.409783] ffffffff8115077a ffff88005d2878e8 ffff88005d2878f8 ffff88005d287900
[43972.418474] 0000000000000000 ffffea0000000000 ffffea0000000017 0100000000000001
[43972.427164] ffff88043ffef280 0000000000000020 0000000000000000 0000000000000000
[43972.435853] Call Trace:
[43972.438679] [<ffffffff8115077a>] ? shrink_inactive_list+0xca/0x500
[43972.445774] [<ffffffff81151451>] ? shrink_lruvec+0x511/0x6a0
[43972.452294] [<ffffffff81151654>] ? shrink_zone+0x74/0x1b0
[43972.458523] [<ffffffff81151b8d>] ? do_try_to_free_pages+0x12d/0x520
[43972.465725] [<ffffffff8115203d>] ? try_to_free_pages+0xbd/0x150
[43972.472536] [<ffffffff81146e1e>] ? __alloc_pages_nodemask+0x6fe/0xb30
[43972.479933] [<ffffffff81064ca5>] ? copy_process.part.25+0xf5/0x1c30
[43972.487133] [<ffffffff81064cc6>] ? copy_process.part.25+0x116/0x1c30
[43972.494431] [<ffffffff8107da90>] ? ____call_usermodehelper+0x130/0x130
[43972.501923] [<ffffffff810669b1>] ? do_fork+0xe1/0x390
[43972.507762] [<ffffffff810a29dc>] ? pick_next_task_fair+0x19c/0x820
[43972.514866] [<ffffffff8101155e>] ? __switch_to+0xde/0x570
[43972.521093] [<ffffffff81066c82>] ? kernel_thread+0x22/0x30
[43972.527421] [<ffffffff8107db09>] ? __call_usermodehelper+0x29/0x90
[43972.534525] [<ffffffff810817c2>] ? process_one_work+0x172/0x420
[43972.541336] [<ffffffff81081e53>] ? worker_thread+0x113/0x4f0
[43972.547855] [<ffffffff81510d61>] ? __schedule+0x2b1/0x700
[43972.554085] [<ffffffff81081d40>] ? rescuer_thread+0x2d0/0x2d0
[43972.560703] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43972.566446] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43972.573842] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43972.580169] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43972.587564] Code: 00 66 66 66 66 90 fa 66 66 90 66 66 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 89 d1 75 01 c3 0f b7 07 66 39 d0 74 f7 <f3> 90 0f b7 07 66 39 c8 75 f6 c3 66 66 2e 0f 1f 84 00 00 00 00
[43972.613623] NMI backtrace for cpu 5
[43972.617619] CPU: 5 PID: 2131 Comm: smbd Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43972.628249] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43972.635643] task: ffff880434d78210 ti: ffff880435520000 task.ti: ffff880435520000
[43972.644134] RIP: 0010:[<ffffffff81514519>] [<ffffffff81514519>] _raw_spin_lock_irqsave+0x39/0x50
[43972.654256] RSP: 0000:ffff880435523d10 EFLAGS: 00000016
[43972.660289] RAX: 000000000000f513 RBX: 0000000000000000 RCX: 000000000000f513
[43972.668363] RDX: 000000000000f50e RSI: 0000000000000246 RDI: ffff88043ffef280
[43972.676438] RBP: ffffea0009fa64e8 R08: 0000000000000000 R09: ffffea000997f7c0
[43972.684514] R10: 0000000000003748 R11: 0000000000000000 R12: ffff88043fc2f0a0
[43972.692590] R13: 0000000000000000 R14: ffff88043ffeee00 R15: ffff88043ffeee00
[43972.700666] FS: 00007ff71022b740(0000) GS:ffff88043fc20000(0000) knlGS:0000000000000000
[43972.709837] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43972.716355] CR2: 00007ff71022ba14 CR3: 000000043663b000 CR4: 00000000000007e0
[43972.724430] Stack:
[43972.726766] ffffffff8114ae43 ffffffff8114b0f0 0000000000000000 ffff88043fc2f0a0
[43972.735458] ffffea000997f788 00007ff71022ba14 ffffea000997f788 ffff8804366bb158
[43972.744146] ffff880434c01480 ffffffff8114b4b1 ffffea00072e7378 ffff88043632c918
[43972.752836] Call Trace:
[43972.755663] [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43972.762669] [<ffffffff8114b0f0>] ? __activate_page+0x1d0/0x1d0
[43972.769383] [<ffffffff8114b4b1>] ? __lru_cache_add+0x51/0x80
[43972.775905] [<ffffffff81167608>] ? do_wp_page+0x288/0x7c0
[43972.782133] [<ffffffff81169d9d>] ? handle_mm_fault+0x84d/0x11c0
[43972.788937] [<ffffffff8109bc0f>] ? select_task_rq_fair+0x26f/0x700
[43972.796040] [<ffffffff810572b7>] ? __do_page_fault+0x177/0x4f0
[43972.802756] [<ffffffff8101b975>] ? sched_clock+0x5/0x10
[43972.808790] [<ffffffff810945f5>] ? check_preempt_curr+0x75/0xa0
[43972.815604] [<ffffffff81096a8a>] ? wake_up_new_task+0xda/0x160
[43972.822318] [<ffffffff81066a22>] ? do_fork+0x152/0x390
[43972.828254] [<ffffffff81516a28>] ? page_fault+0x28/0x30
[43972.835551] Code: c6 fa 66 66 90 66 66 90 ba 00 00 01 00 f0 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f b7 17 66 39 d1 74 f4 f3 90 <0f> b7 17 66 39 d0 75 f6 eb e8 66 66 66 66 2e 0f 1f 84 00 00 00
[43972.861624] NMI backtrace for cpu 6
[43972.865608] CPU: 6 PID: 68 Comm: kswapd1 Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43972.876336] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43972.883730] task: ffff880436e20290 ti: ffff880435250000 task.ti: ffff880435250000
[43972.892220] RIP: 0010:[<ffffffff81514519>] [<ffffffff81514519>] _raw_spin_lock_irqsave+0x39/0x50
[43972.902341] RSP: 0018:ffff880435253b08 EFLAGS: 00000012
[43972.908376] RAX: 000000000000f510 RBX: 0000000000000000 RCX: 000000000000f510
[43972.916443] RDX: 000000000000f50e RSI: 0000000000000246 RDI: ffff88043ffef280
[43972.924517] RBP: ffffea000af8d500 R08: ffff880436e20290 R09: 0000000000000020
[43972.932592] R10: 000000000000799f R11: 0000000000000400 R12: ffff88043fc4f0a0
[43972.940667] R13: 0000000000000000 R14: ffff88043ffeee00 R15: ffff88043ffeee00
[43972.948742] FS: 0000000000000000(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
[43972.957913] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[43972.964431] CR2: 00007f4d11d75000 CR3: 0000000001813000 CR4: 00000000000007e0
[43972.972506] Stack:
[43972.974843] ffffffff8114ae43 ffffffff8114b0f0 0000000000000000 0000000000000006
[43972.983533] ffff880435253dc0 ffff880435253be8 0000000000000020 ffff88043ffef288
[43972.992219] 0000000000000002 ffffffff8114b9de ffff88043ffeee00 ffff880435253dc0
[43973.000908] Call Trace:
[43973.003734] [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43973.010732] [<ffffffff8114b0f0>] ? __activate_page+0x1d0/0x1d0
[43973.017448] [<ffffffff8114b9de>] ? lru_add_drain_cpu+0xce/0xe0
[43973.024162] [<ffffffff8114baf2>] ? lru_add_drain+0x12/0x20
[43973.030487] [<ffffffff81150747>] ? shrink_inactive_list+0x97/0x500
[43973.037583] [<ffffffff81151451>] ? shrink_lruvec+0x511/0x6a0
[43973.044104] [<ffffffff81151654>] ? shrink_zone+0x74/0x1b0
[43973.050332] [<ffffffff8115276e>] ? balance_pgdat+0x38e/0x5c0
[43973.056853] [<ffffffff81152aff>] ? kswapd+0x15f/0x3f0
[43973.062693] [<ffffffff810a7e60>] ? prepare_to_wait_event+0xf0/0xf0
[43973.069796] [<ffffffff811529a0>] ? balance_pgdat+0x5c0/0x5c0
[43973.076316] [<ffffffff8108809d>] ? kthread+0xbd/0xe0
[43973.082060] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43973.089447] [<ffffffff81514958>] ? ret_from_fork+0x58/0x90
[43973.095773] [<ffffffff81087fe0>] ? kthread_create_on_node+0x180/0x180
[43973.103167] Code: c6 fa 66 66 90 66 66 90 ba 00 00 01 00 f0 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f b7 17 66 39 d1 74 f4 f3 90 <0f> b7 17 66 39 d0 75 f6 eb e8 66 66 66 66 2e 0f 1f 84 00 00 00
[43973.129238] NMI backtrace for cpu 7
[43973.133220] CPU: 7 PID: 2545 Comm: systemd Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[43973.144140] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[43973.151534] task: ffff880230dd8b60 ti: ffff8800ba59c000 task.ti: ffff8800ba59c000
[43973.160023] RIP: 0010:[<ffffffff8151451c>] [<ffffffff8151451c>] _raw_spin_lock_irqsave+0x3c/0x50
[43973.170146] RSP: 0000:ffff8800ba59fd10 EFLAGS: 00000016
[43973.176179] RAX: 000000000000f511 RBX: 0000000000000008 RCX: 000000000000f511
[43973.184254] RDX: 000000000000f50e RSI: 0000000000000246 RDI: ffff88043ffef280
[43973.192329] RBP: ffffea00087ce360 R08: 0000000000000000 R09: ffffea000ac91ee0
[43973.200406] R10: 0000000000003748 R11: 0000000000000000 R12: ffff88043fc6f0a0
[43973.208480] R13: 0000000000000246 R14: ffff88043ffeee00 R15: ffff88043ffeee00
[43973.216554] FS: 00007f4d11d6d880(0000) GS:ffff88043fc60000(0000) knlGS:0000000000000000
[43973.225725] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[43973.232246] CR2: 00007f4d12391898 CR3: 0000000436079000 CR4: 00000000000007e0
[43973.240321] Stack:
[43973.242657] ffffffff8114ae43 ffffffff8114b0f0 0000000000000000 ffff88043fc6f0a0
[43973.251344] ffffea000ac91ea8 00007f4d12391898 ffffea000ac91ea8 ffff88043668bc88
[43973.260029] ffff8804365e6480 ffffffff8114b4b1 ffffea0005735300 ffff88043556f3d8
[43973.268716] Call Trace:
[43973.271539] [<ffffffff8114ae43>] ? pagevec_lru_move_fn+0x93/0x110
[43973.278545] [<ffffffff8114b0f0>] ? __activate_page+0x1d0/0x1d0
[43973.285259] [<ffffffff8114b4b1>] ? __lru_cache_add+0x51/0x80
[43973.291779] [<ffffffff81167608>] ? do_wp_page+0x288/0x7c0
[43973.298007] [<ffffffff81169d9d>] ? handle_mm_fault+0x84d/0x11c0
[43973.304812] [<ffffffff810572b7>] ? __do_page_fault+0x177/0x4f0
[43973.311527] [<ffffffff811be3df>] ? dput+0x1f/0x170
[43973.317073] [<ffffffff81516a28>] ? page_fault+0x28/0x30
[43973.323107] Code: 66 90 66 66 90 ba 00 00 01 00 f0 0f c1 17 89 d1 c1 e9 10 66 39 d1 89 c8 75 04 48 89 f0 c3 0f b7 17 66 39 d1 74 f4 f3 90 0f b7 17 <66> 39 d0 75 f6 eb e8 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66
[43973.349327] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1789.529 msecs
[44028.699502] ------------[ cut here ]------------
[44028.704776] WARNING: CPU: 2 PID: 2484 at /build/linux-zbOcpU/linux-3.16.7-ckt25/kernel/watchdog.c:265 watchd og_overflow_callback+0x98/0xc0()
[44028.719004] Watchdog detected hard LOCKUP on cpu 2
[44028.724234] Modules linked in: xt_multiport iptable_filter ip_tables x_tables binfmt_misc veth nfsd auth_rpc gss oid_registry nfs_acl nfs lockd fscache sunrpc bridge ttm coretemp drm_kms_helper kvm_intel iTCO_wdt ch341 i pmi_si iTCO_vendor_support kvm pcspkr serio_raw drm tpm_infineon evdev lpc_ich usbserial ipmi_msghandler button tpm_tis mfd_core tpm i7core_edac edac_core acpi_cpufreq shpchp processor thermal_sys 8021q garp stp mrp llc bo nding autofs4 ext4 crc16 mbcache jbd2 xts gf128mul algif_skcipher af_alg dm_crypt dm_mod raid1 md_mod hid_gener ic usbhid hid sd_mod crc_t10dif crct10dif_generic sg crct10dif_common uhci_hcd ahci ehci_pci libahci ehci_hcd l ibata hpsa igb i2c_algo_bit crc32c_intel i2c_core dca usbcore r8169 ptp scsi_mod mii usb_common niu pps_core
[44028.807244] CPU: 2 PID: 2484 Comm: sshd Tainted: G D W I 3.16.0-4-amd64 #1 Debian 3.16.7-ckt25-1
[44028.817874] Hardware name: HP ProLiant DL180 G6 , BIOS O20 05/01/2012
[44028.825320] 0000000000000000 ffffffff8150e835 ffff88023fc45c98 0000000000000009
[44028.834013] ffffffff810677f7 ffff8802377aac00 ffff88023fc45ce8 ffff88023fc45d88
[44028.842707] 0000000000000000 ffff88023fc45ef8 ffffffff8106785c ffffffff8171d648
[44028.851406] Call Trace:
[44028.854222] <NMI> [<ffffffff8150e835>] ? dump_stack+0x5d/0x78
[44028.861096] [<ffffffff810677f7>] ? warn_slowpath_common+0x77/0x90
[44028.868104] [<ffffffff8106785c>] ? warn_slowpath_fmt+0x4c/0x50
[44028.874821] [<ffffffff8108c256>] ? atomic_notifier_call_chain+0x16/0x20
[44028.882416] [<ffffffff8137b5ac>] ? vt_console_print+0x23c/0x3f0
[44028.889304] [<ffffffff810fd1f8>] ? watchdog_overflow_callback+0x98/0xc0
[44028.896888] [<ffffffff81134ae7>] ? __perf_event_overflow+0x87/0x230
[44028.904092] [<ffffffff8102f89a>] ? intel_pmu_handle_irq+0x1ea/0x3d0
[44028.911298] [<ffffffff8134ba43>] ? ghes_copy_tofrom_phys+0x113/0x210
[44028.918598] [<ffffffff810274a6>] ? perf_event_nmi_handler+0x26/0x40
[44028.925800] [<ffffffff8101691d>] ? nmi_handle+0x8d/0x120
[44028.931934] [<ffffffff81016ebd>] ? default_do_nmi+0xbd/0x110
[44028.938455] [<ffffffff81017004>] ? do_nmi+0xf4/0x170
[44028.944199] [<ffffffff81516e0f>] ? end_repeat_nmi+0x1a/0x1e
[44028.950626] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[44028.957828] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[44028.965032] [<ffffffff81514519>] ? _raw_spin_lock_irqsave+0x39/0x50
[44028.972233] <<EOE>> [<ffffffff8114ac86>] ? release_pages+0xf6/0x220
[44028.979684] [<ffffffff8117abb5>] ? free_pages_and_swap_cache+0x95/0xb0
[44028.987179] [<ffffffff811653ac>] ? tlb_flush_mmu_free+0x2c/0x50
[44028.993992] [<ffffffff81165def>] ? tlb_finish_mmu+0xf/0x40
[44029.000320] [<ffffffff8117095a>] ? exit_mmap+0xba/0x150
[44029.006355] [<ffffffff810648bc>] ? mmput+0x5c/0x120
[44029.012003] [<ffffffff811aef2d>] ? flush_old_exec+0x3fd/0x7d0
[44029.018623] [<ffffffff811f849c>] ? load_elf_binary+0x32c/0x1750
[44029.025437] [<ffffffffa046705e>] ? load_misc_binary+0x5e/0x460 [binfmt_misc]
[44029.033515] [<ffffffff811ae3af>] ? copy_strings.isra.17+0x27f/0x2f0
[44029.040716] [<ffffffff811ae701>] ? search_binary_handler+0x91/0x1a0
[44029.047920] [<ffffffff811afa47>] ? do_execve_common.isra.23+0x4d7/0x630
[44029.055513] [<ffffffff811afdd4>] ? SyS_execve+0x24/0x30
[44029.061548] [<ffffffff81514fc9>] ? stub_execve+0x69/0xa0
[44029.067679] ---[ end trace 15b5c8255c35892e ]---

Analyse

Ich war nun sehr zuversichtlich endlich eine Lösung zu diesem Problem zu finden. Nach kurzer Internetrecherche stellte sich allerdings heraus, das die Meldung in der ersten Zeile general protection fault auf ein allgemeines Problem hinweist, was so gut wie alles sein kann.
Da der Server mehrere Jahre tadellos in einem Rechenzentrum in Betrieb war, habe ich analysiert, was sich an dem System geändert hat. Ich habe die Arbeitsspeicher meiner Hewlett-Packard Server getauscht um den Hypervisor mit dem Maximum an Arbeitsspeicher aufzurüsten. Somit habe ich alle Arbeitsspeicher aus dem Dateiserver (DL180) entfernt bis auf einen je Prozessor. Leider ist in der folgenden Nacht das System erneut mit einem Kernel Panic eingefroren. Zudem haben sich die Festplatten geändert, früher 1 TB heute sind es acht 4 TB Festplatten in einem RAID6. Ich habe nun die
Dieser Link wurde entfernt. Links sind nur für Mitglieder sichtbar. Bitte melden Sie sich mit Ihrem Benutzernamen und Ihrem Passwort an und versuchen Sie es erneut.
, auch hier scheint alles in Ordnung zu sein. Ich habe dann geprüft ob Hewlett-Packard eine neue
Dieser Link wurde entfernt. Links sind nur für Mitglieder sichtbar. Bitte melden Sie sich mit Ihrem Benutzernamen und Ihrem Passwort an und versuchen Sie es erneut.
veröffentlicht hat. HP hat Version 6.64 veröffentlicht, ich habe bei mir Version 6.00 installiert. Ein interessanter Hinweis ist in den Fixes zu finden:
Running SMARTCTL (smartmontools) on HP Proliant G6/G7 (Px1x) Smart Array controllers that have firmware version 5.70 to 6.62 installed with SATA drives attached may result in system not responding or reboot. When reboot occurred, a reboot 1719 POST error message with lockup 0x15 displayed.

Ich habe zwar die SmartMonTools nicht auf meinem Server installiert, hatte allerdings die vermuting das der Kernel evtl. Daten versucht abzurufen und deshalb in einen Kernel Panic endet.

Leider löste auch das Firmwareupdate des RAID-Controllers das Problem nicht. In zusammenarbiet mit einem Kollegen, sind wir auf ein BIOS Update für
Dieser Link wurde entfernt. Links sind nur für Mitglieder sichtbar. Bitte melden Sie sich mit Ihrem Benutzernamen und Ihrem Passwort an und versuchen Sie es erneut.
Server aufmerksam geworden. Dieses Update wurde von HP als Critical veröffentlich und behebt folgendes Problem:
Addressed a processor issue under which a rare and complex sequence of internal processor microarchitecture events that occur in specific operating environments could cause a server system to experience unexpected page faults, general protection faults, or machine check exceptions or other unpredictable system behavior.

Dieses BIOS Update behebt general protection faults, welche auch im Kernel Panic Log ersichtlich sind. Also habe ich dieses Update eingespielt und gehofft das ich nun keine Kernel Panic mehr erhalte. Am nächsten Morgen lief der Server ohne Kernel Panic, allerdings ist dieser im Laufe des Tages erneut eingefroren und hatte wieder eine Kernel Panic.

Lösung

Ich war mit meinem Latein am Ende und wusste nicht mehr weiter, so habe ich kurzer Hand alle "unnötigen" Hardware Komponenten (Netzwerkkarten) aus dem System entfernt. Ohne diese Netzwerkkarten lief der Server 48 Stunden ohne Kernel Panic. Da ich für die Realtek Netzwerkkarte die non-free Treiber installieren musste, lag der verdacht nahe das diese die Kernel Panics verursacht und habe die SUN Quad Port NIC ATLS1QGE 501-7606-06 wieder in den Server eingebaut. Als ich am nächsten Morgen aufgewacht bin hatte ich erneut eine Kernel Panic. Somit war ich mir sicher das diese von der Sun Netzwerkkarte ausgelöst wurden. Somit habe ich diese ausgebaut und die Realtek NIC wieder eingebaut.
Seit dem hatte ich keine Kernel Panic mehr.