mysql - mysqld out of memory with plenty (?) of memory -

i have busy server running percona 5.6.34-79.1-1.xenial on ubuntu 16.04. works great, however, every few weeks, mysqld shot out-of-memory killer , i'm not able find out why.

root@master02:~# grep out /var/log/syslog apr  6 13:37:03 master02 kernel: [17420955.874564] out of memory: kill process 36138 (mysqld) score 659 or sacrifice child 

so killed at 13:37:03.

however, 2 seconds before, using around 110 gb ram (the system has ~160 gb ram), 55 gb free:

root@master02:~# cat /root/logs/free/free-2017-04-06-13\:36\:01.log               total        used        free      shared  buff/cache   available mem:      165050752   109560372      593508      189240    54896872    54434632 swap:             0           0           0  root@master02:~# cat /root/logs/free/free-2017-04-06-13\:37\:01.log               total        used        free      shared  buff/cache   available mem:      165050752   109582416      602704      189624    54865632    54412072 swap:             0           0           0  root@master02:~# cat /root/logs/free/free-2017-04-06-13\:38\:01.log               total        used        free      shared  buff/cache   available mem:      165050752    17982728    92226488      189200    54841536   146007904 swap:             0           0           0 

my.cnf has "innodb-buffer-pool-size        = 130g" set. don't find mysqld allocated 50 gb in 2 secs , killed (though may wrong of course).

here dmesg showing full oom - memory allocation issue tunable? i'd appreciate hints here.

[17420955.874279] mysqld invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0 [17420955.874282] mysqld cpuset=/ mems_allowed=0-1 [17420955.874287] cpu: 4 pid: 36138 comm: mysqld not tainted 4.4.0-59-generic #80-ubuntu [17420955.874288] hardware name: xen hvm domu, bios 4.2.amazon 11/11/2016 [17420955.874290]  0000000000000286 00000000dba8c85b ffff88142f4bbaf0 ffffffff813f7583 [17420955.874292]  ffff88142f4bbcc8 ffff8827ac8db800 ffff88142f4bbb60 ffffffff8120ad5e [17420955.874295]  ffffffff81cd2dc7 0000000000000000 ffffffff81e67760 0000000000000206 [17420955.874297] call trace: [17420955.874304]  [<ffffffff813f7583>] dump_stack+0x63/0x90 [17420955.874307]  [<ffffffff8120ad5e>] dump_header+0x5a/0x1c5 [17420955.874311]  [<ffffffff81192722>] oom_kill_process+0x202/0x3c0 [17420955.874312]  [<ffffffff81192b49>] out_of_memory+0x219/0x460 [17420955.874315]  [<ffffffff81198abd>] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70 [17420955.874317]  [<ffffffff81198eb6>] __alloc_pages_nodemask+0x286/0x2a0 [17420955.874319]  [<ffffffff81198f6b>] alloc_kmem_pages_node+0x4b/0xc0 [17420955.874323]  [<ffffffff8107ea5e>] copy_process+0x1be/0x1b70 [17420955.874326]  [<ffffffff811c164d>] ? handle_mm_fault+0xcbd/0x1820 [17420955.874328]  [<ffffffff810805a0>] _do_fork+0x80/0x360 [17420955.874329]  [<ffffffff81080929>] sys_clone+0x19/0x20 [17420955.874333]  [<ffffffff818384f2>] entry_syscall_64_fastpath+0x16/0x71 [17420955.874343] mem-info: [17420955.874354] active_anon:27160197 inactive_anon:28926 isolated_anon:0                    active_file:5497699 inactive_file:7563747 isolated_file:0                    unevictable:914 dirty:2486 writeback:0 unstable:0                    slab_reclaimable:556865 slab_unreclaimable:45056                    mapped:20876 shmem:47414 pagetables:71927 bounce:0                    free:154548 free_pcp:64 free_cma:0 [17420955.874357] node 0 dma free:15904kb min:8kb low:8kb high:12kb active_anon:0kb inactive_anon:0kb active_file:0kb inactive_file:0kb unevictable:0kb isolated(anon):0kb isolated(file):0kb present:15988kb managed:15904kb mlocked:0kb dirty:0kb writeback:0kb mapped:0kb shmem:0kb slab_reclaimable:0kb slab_unreclaimable:0kb kernel_stack:0kb pagetables:0kb unstable:0kb bounce:0kb free_pcp:0kb local_pcp:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0 all_unreclaimable? yes [17420955.874361] lowmem_reserve[]: 0 3706 80497 80497 80497 [17420955.874370] node 0 dma32 free:311248kb min:2072kb low:2588kb high:3108kb active_anon:3411180kb inactive_anon:1348kb active_file:4kb inactive_file:8kb unevictable:280kb isolated(anon):0kb isolated(file):0kb present:3915776kb managed:3835152kb mlocked:280kb dirty:0kb writeback:0kb mapped:524kb shmem:2308kb slab_reclaimable:65844kb slab_unreclaimable:14292kb kernel_stack:1760kb pagetables:13784kb unstable:0kb bounce:0kb free_pcp:0kb local_pcp:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0 all_unreclaimable? no [17420955.874373] lowmem_reserve[]: 0 0 76791 76791 76791 [17420955.874376] node 0 normal free:143060kb min:42940kb low:53672kb high:64408kb active_anon:62267420kb inactive_anon:31360kb active_file:7207656kb inactive_file:7550284kb unevictable:3296kb isolated(anon):0kb isolated(file):0kb present:79953920kb managed:78634400kb mlocked:3296kb dirty:3116kb writeback:0kb mapped:18940kb shmem:49312kb slab_reclaimable:838024kb slab_unreclaimable:78600kb kernel_stack:51840kb pagetables:159192kb unstable:0kb bounce:0kb free_pcp:136kb local_pcp:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:76 all_unreclaimable? no [17420955.874380] lowmem_reserve[]: 0 0 0 0 0 [17420955.874383] node 1 normal free:147980kb min:45088kb low:56360kb high:67632kb active_anon:42962188kb inactive_anon:82996kb active_file:14783136kb inactive_file:22704696kb unevictable:80kb isolated(anon):0kb isolated(file):0kb present:83886080kb managed:82565296kb mlocked:80kb dirty:6828kb writeback:0kb mapped:64040kb shmem:138036kb slab_reclaimable:1323592kb slab_unreclaimable:87332kb kernel_stack:48624kb pagetables:114732kb unstable:0kb bounce:0kb free_pcp:120kb local_pcp:0kb free_cma:0kb writeback_tmp:0kb pages_scanned:0 all_unreclaimable? no [17420955.874387] lowmem_reserve[]: 0 0 0 0 0 [17420955.874390] node 0 dma: 0*4kb 0*8kb 0*16kb 1*32kb (u) 2*64kb (u) 1*128kb (u) 1*256kb (u) 0*512kb 1*1024kb (u) 1*2048kb (m) 3*4096kb (m) = 15904kb [17420955.874398] node 0 dma32: 13480*4kb (ume) 6994*8kb (ume) 2710*16kb (ume) 714*32kb (umeh) 354*64kb (umeh) 225*128kb (umeh) 107*256kb (umeh) 52*512kb (ume) 29*1024kb (umeh) 0*2048kb 0*4096kb = 311248kb [17420955.874408] node 0 normal: 29953*4kb (umeh) 2985*8kb (umeh) 1*16kb (h) 0*32kb 2*64kb (h) 2*128kb (h) 2*256kb (h) 0*512kb 1*1024kb (h) 0*2048kb 0*4096kb = 145628kb [17420955.874416] node 1 normal: 36438*4kb (ume) 412*8kb (umeh) 2*16kb (h) 2*32kb (h) 0*64kb 1*128kb (h) 1*256kb (h) 2*512kb (h) 0*1024kb 0*2048kb 0*4096kb = 150552kb [17420955.874425] node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kb [17420955.874427] node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kb [17420955.874428] node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kb [17420955.874429] node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kb [17420955.874430] 13109543 total pagecache pages [17420955.874431] 0 pages in swap cache [17420955.874440] swap cache stats: add 0, delete 0, find 0/0 [17420955.874441] free swap  = 0kb [17420955.874442] total swap = 0kb [17420955.874443] 41942941 pages ram [17420955.874444] 0 pages highmem/movableonly [17420955.874444] 680253 pages reserved [17420955.874445] 0 pages cma reserved [17420955.874446] 0 pages hwpoisoned [17420955.874447] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name [17420955.874457] [  677]     0   677    38915    23334      80       3        0             0 systemd-journal [17420955.874461] [  698]     0   698    25742       47      17       3        0             0 lvmetad [17420955.874469] [  729]     0   729    11545     1219      23       3        0         -1000 systemd-udevd [17420955.874472] [ 1112]   109  1112    25081      492      19       3        0             0 systemd-timesyn [17420955.874474] [ 1333]     0  1333     4030      637      12       3        0             0 dhclient [17420955.874475] [ 1541]     0  1541     6932      547      19       3        0             0 cron [17420955.874477] [ 1552]     0  1552   365572     5403      79       7        0             0 snapd [17420955.874479] [ 1559]   102  1559    10725      667      26       3        0          -900 dbus-daemon [17420955.874480] [ 1570]     0  1570    77473      728      21       3        0             0 lxcfs [17420955.874482] [ 1576]   101  1576    64099     1008      27       3        0             0 rsyslogd [17420955.874484] [ 1578]   106  1578     1884      403       9       3        0             0 vnstatd [17420955.874486] [ 1580]     0  1580     1100      322       8       3        0             0 acpid [17420955.874488] [ 1582]     0  1582    68674     1100      37       3        0             0 accounts-daemon [17420955.874489] [ 1584]     0  1584     6511      378      18       3        0             0 atd [17420955.874491] [ 1593]     0  1593    16380      700      36       3        0         -1000 sshd [17420955.874493] [ 1595]     0  1595     7157      635      18       3        0             0 systemd-logind [17420955.874495] [ 1599]     0  1599     7470      139      19       3        0             0 cgmanager [17420955.874497] [ 1618]     0  1618     3344      116      11       3        0             0 mdadm [17420955.874498] [ 1622]     0  1622     1306      441       8       3        0             0 iscsid [17420955.874500] [ 1623]     0  1623     1431      916       8       3        0           -17 iscsid [17420955.874511] [ 1717]     0  1717     3619      388      12       3        0             0 agetty [17420955.874513] [ 1718]     0  1718     3665      343      12       3        0             0 agetty [17420955.874515] [ 1731]     0  1731     5025      653      15       3        0             0 irqbalance [17420955.874517] [ 1744]     0  1744    69272      719      39       3        0             0 polkitd [17420955.874519] [ 2484]  1001  2484    11312      218      27       3        0             0 systemd [17420955.874520] [ 2488]  1001  2488    15805      475      34       3        0             0 (sd-pam) [17420955.874523] [ 6289]     0  6289     7718     1364      19       3        0             0 tmux [17420955.874531] [ 6290]     0  6290     5381      924      15       3        0             0 bash [17420955.874533] [ 6306]     0  6306     2158      395      10       3        0             0 mysqld_safe [17420955.874537] [36138]   107 36138 43942177 27121187   70971     157        0             0 mysqld [17420955.874539] [78610]   108 78610     5992      577      16       3        0             0 nrpe [17420955.874542] [19441]     0 19441    24876     1740      54       3        0             0 sshd [17420955.874543] [19447]  1008 19447    11312     1147      27       3        0             0 systemd [17420955.874555] [19449]  1008 19449    15817      487      34       3        0             0 (sd-pam) [17420955.874557] [19575]  1008 19575    24876      834      51       3        0             0 sshd [17420955.874558] [19576]     0 19576    14970      933      34       3        0             0 sudo [17420955.874560] [19577]     0 19577     5388     1343      16       3        0             0 bash [17420955.874561] [26883]     0 26883     8430     5113      22       5        0             0 mysqld_exporter [17420955.874564] out of memory: kill process 36138 (mysqld) score 659 or sacrifice child [17420955.890336] killed process 36138 (mysqld) total-vm:175768708kb, anon-rss:108470860kb, file-rss:13888kb 

here my.cnf file:

[mysql]  # client # port                           = 3306 socket                         = /var/run/mysqld/mysqld.sock  [mysqld]  # general # user                           = mysql default-storage-engine         = innodb socket                         = /var/run/mysqld/mysqld.sock pid-file                       = /var/run/mysqld/mysqld.pid  # myisam # key-buffer-size                = 32m myisam-recover                 = force,backup  # safety # max-allowed-packet             = 16m max-connect-errors             = 1000000  # data storage # datadir                        = /var/lib/mysql/  # binary logging # log-bin                        = /var/lib/mysql-binlogs/mysql-bin expire-logs-days               = 7 sync-binlog                    = 1  binlog-format                  = mixed  # replication # server-id                      = 1 auto_increment_offset          = 1  # total number of master servers auto_increment_increment       = 2  log-slave-updates              = 1 relay-log                      = /var/lib/mysql-binlogs/relay-bin slave-net-timeout              = 60  # caches , limits # tmp-table-size                 = 32m max-heap-table-size            = 32m query-cache-type               = 0 query-cache-size               = 0 thread-cache-size              = 50 open-files-limit               = 65535 table-definition-cache         = 4096 table-open-cache               = 4096  # innodb # innodb-flush-method            = o_direct innodb-log-files-in-group      = 2 innodb-log-file-size           = 512m innodb-flush-log-at-trx-commit = 2 innodb-file-per-table          = 1 innodb-buffer-pool-size        = 125g innodb_large_prefix            = 1 innodb_file_format             = barracuda  # logging # log-error                      = /var/log/mysql/mysql-error.log log-queries-not-using-indexes  = 0 slow-query-log                 = 0 slow-query-log-file            = /var/log/mysql/mysql-slow.log  # other setup # character_set_server           = utf8mb4 collation-server               = utf8mb4_unicode_ci init-connect                   = 'set names utf8mb4'  skip-name-resolve              = 1  max_connections                = 12288 wait_timeout                   = 120 connect_timeout                = 30 interactive_timeout            = 120 

i'll assume using along php , apache. find causing issue in 1 of application's log files.

note time of failure in mysql log , in other log files during time period , should find clues lead answer.

i should note oom chooses service shut down when system runs out of memory , though shut down may not have been service created issue.