Bug#873945: apache2-bin: apache2 with http2 segfault

[Hosted Power Sales]

Package: apache2-bin
Version: 2.4.25-3+deb9u2
Severity: important
 Dear Maintainer,
 *** Reporter, please consider answering these questions, where appropriate ***
    * What led up to the situation?
 SEGFAULT Apache2 with http2, just by having users visiting the site.
 gdb /usr/sbin/apache2 core-apache2-11-33-33-16474-1504205745
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...Reading symbols from /usr/lib/debug/.build-id/1b/81c3ec196acae651912c3a4025bc1c6a07c375.debug...done.
done.
[New LWP 16532]
[New LWP 16477]
[New LWP 16474]
[New LWP 16475]
[New LWP 16479]
[New LWP 16476]
[New LWP 16480]
[New LWP 16481]
[New LWP 16511]
[New LWP 16478]
[New LWP 16487]
[New LWP 16482]
[New LWP 16512]
[New LWP 16489]
[New LWP 16483]
[New LWP 16491]
[New LWP 16494]
[New LWP 16513]
[New LWP 16484]
[New LWP 16499]
[New LWP 16485]
[New LWP 16500]
[New LWP 16516]
[New LWP 16502]
[New LWP 16486]
[New LWP 16504]
[New LWP 16517]
[New LWP 16488]
[New LWP 16506]
[New LWP 16507]
[New LWP 16490]
[New LWP 16518]
[New LWP 16492]
[New LWP 16509]
[New LWP 16514]
[New LWP 16519]
[New LWP 16522]
[New LWP 16527]
[New LWP 16520]
[New LWP 16523]
[New LWP 16524]
[New LWP 16525]
[New LWP 16529]
[New LWP 16495]
[New LWP 16533]
[New LWP 16496]
[New LWP 16497]
[New LWP 16535]
[New LWP 16498]
[New LWP 16501]
[New LWP 16536]
[New LWP 16503]
[New LWP 16537]
[New LWP 16505]
[New LWP 16510]
[New LWP 16515]
[New LWP 16521]
[New LWP 16526]
[New LWP 16528]
[New LWP 16530]
[New LWP 16531]
[New LWP 16534]
[New LWP 16538]
[New LWP 16539]
[New LWP 16540]
[New LWP 16541]
[New LWP 16493]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  h2_stream_out_prepare (stream=***@entry=0x7ff7772330a0, plen=***@entry=0x7ff77cfe89e0, peos=***@entry=0x7ff77cfe89dc, presponse=***@entry=0x7ff77cfe89e8) at h2_stream.c:604
604     h2_stream.c: No such file or directory.
[Current thread is 1 (Thread 0x7ff77cfe9700 (LWP 16532))]
(gdb) bt
#0  h2_stream_out_prepare (stream=***@entry=0x7ff7772330a0, plen=***@entry=0x7ff77cfe89e0, peos=***@entry=0x7ff77cfe89dc, presponse=***@entry=0x7ff77cfe89e8) at h2_stream.c:604
#1  0x00007ff7ae7bb6cb in on_stream_resume (ctx=0x7ff7768410a0, stream=0x7ff7772330a0) at h2_session.c:1576
#2  0x00007ff7ae7b2e3b in h2_mplx_dispatch_master_events (m=0x7ff7768412d0, on_resume=***@entry=0x7ff7ae7bb500 <on_stream_resume>, on_ctx=***@entry=0x7ff7768410a0) at h2_mplx.c:1379
#3  0x00007ff7ae7bc7ab in h2_session_process (session=0x7ff7768410a0, async=***@entry=1) at h2_session.c:2210
#4  0x00007ff7ae7a8b2a in h2_conn_run (ctx=***@entry=0x7ff78c144360, c=***@entry=0x7ff78c14b348) at h2_conn.c:212
#5  0x00007ff7ae7aea5b in h2_h2_process_conn (c=0x7ff78c14b348) at h2_h2.c:658
#6  0x0000558b46d1e6f0 in ap_run_process_connection (c=***@entry=0x7ff78c14b348) at connection.c:42
#7  0x00007ff7ad8976e8 in process_socket (my_thread_num=<optimized out>, my_child_num=<optimized out>, cs=0x7ff78c14b2b8, sock=<optimized out>, p=0x7ff78c14b028, thd=<optimized out>) at event.c:1099
#8  worker_thread (thd=<optimized out>, dummy=<optimized out>) at event.c:2003
#9  0x00007ff7b1862494 in start_thread (arg=0x7ff77cfe9700) at pthread_create.c:333
#10 0x00007ff7b15a4aff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
    * What exactly did you do (or not do) that was effective (or ineffective)?
 http2 enabled caused it, disable module fixes it immediately
   -- Package-specific info:
 -- System Information:
Debian Release: 9.1
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
 Kernel: Linux 4.9.0-3-amd64 (SMP w/3 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
 Versions of packages apache2-bin depends on:
ii  libapr1                  1.5.2-5
ii  libaprutil1              1.5.4-3
ii  libaprutil1-dbd-sqlite3  1.5.4-3
ii  libaprutil1-ldap         1.5.4-3
ii  libc6                    2.24-11+deb9u1
ii  libldap-2.4-2            2.4.44+dfsg-5
ii  liblua5.2-0              5.2.4-1.1+b2
ii  libnghttp2-14            1.18.1-1
ii  libpcre3                 2:8.39-3
ii  libssl1.0.2              1.0.2l-2
ii  libxml2                  2.9.4+dfsg1-2.2+deb9u1
ii  perl                     5.24.1-3+deb9u1
ii  zlib1g                   1:1.2.8.dfsg-5
 apache2-bin recommends no packages.
 Versions of packages apache2-bin suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
ii  w3m [www-browser]                                0.5.3-34
 Versions of packages apache2 depends on:
ii  apache2-data         2.4.25-3+deb9u2
ii  apache2-utils        2.4.25-3+deb9u2
ii  dpkg                 1.18.24
ii  init-system-helpers  1.48
ii  lsb-base             9.20161125
ii  mime-support         3.60
ii  perl                 5.24.1-3+deb9u1
ii  procps               2:3.3.12-3
 Versions of packages apache2 recommends:
ii  ssl-cert  1.0.39
 Versions of packages apache2 suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
ii  w3m [www-browser]                                0.5.3-34
 Versions of packages apache2-bin is related to:
ii  apache2      2.4.25-3+deb9u2
ii  apache2-bin  2.4.25-3+deb9u2
 -- no debconf information
  
 
 

[Hosted Power Sales]

We are using this on a site with quite a lot of traffic (often about 1000 concurrent users) and we're using apache as a reverse proxy with mod_proxy and mod_proxy_http.

 
The issue is very reproducanle in our case and makes http2 completely unusable.

 

[Tehnic]

Hello,


i had the same problem and solved by rebuilding apache2_2.4.25-3+deb9u3
after replacing the http2 module source with version from
apache2_2.4.29-1 (plus small needed modifications to build because
of added/deleted files).

Works now for 2 weeks in production without any problem.

Regards

[Thomas]

Hi,

I can confirm this issue/bug in mod_http2. Occasional segmentation
faults when using mod_http2, reproducible by frequently accessing vhosts
with enabled h2 support.


I was also able to resolve it by upgrading mod_http2 to the version
included in Apache 2.4.29.


Coredump backtrace from segfaulted standard Debian (Stretch) 2.4.25
mod_http2 (for what it's worth):

#0 h2_stream_out_prepare (stream=***@entry=0x7f19584550a0,
plen=***@entry=0x7f1959ea2a00, peos=***@entry=0x7f1959ea29fc,
presponse=***@entry=0x7f1959ea2a08) at h2_stream.c:604
[Current thread is 1 (Thread 0x7f1959ea3700 (LWP 15463))]
(gdb) bt full
#0 h2_stream_out_prepare (stream=***@entry=0x7f19584550a0,
plen=***@entry=0x7f1959ea2a00, peos=***@entry=0x7f1959ea29fc,
presponse=***@entry=0x7f1959ea2a08) at h2_stream.c:604
c = <optimized out>
status = <optimized out>
requested = <optimized out>
b = <optimized out>
e = <optimized out>
#1 0x00007f19744786cb in on_stream_resume (ctx=0x7f195849f0a0,
stream=0x7f19584550a0) at h2_session.c:1576
session = 0x7f195849f0a0
status = <optimized out>
rv = <optimized out>
len = 0
eos = 0
headers = 0x0
#2 0x00007f197446fe3b in h2_mplx_dispatch_master_events
(m=0x7f195849f2d0, on_resume=***@entry=0x7f1974478500
<on_stream_resume>, on_ctx=***@entry=0x7f195849f0a0) at h2_mplx.c:1379
status = 0
acquired = 1
ids = {85, 75, 87, 73, 79, 77, 39, 53, 89, 91, 23, 93, 95, 97,
99, 101, 103, 107, 105, 109, 111, 123, 125, 119, 121, 115, 117, 113,
127, 129, 131, 133, 135, 137, 139, 141, 143, 145, 147, 149, 151, 153,
155, 157, 0, 32767, 0, 0,
58, 1, 0, 0, 0, 0, -1010297088, -1897350404, 1, 0, 1481241560,
32537, 1, 0, 1481241904, 32537, 1481241896, 32537, 8000, 0, 1784734664,
32537, 1950777061, 32537, 1508518904, 32537, 1508518896, 32537, 58, 0,
1508518912, 32537,
1508519092, 32537, -1, -1, 1784725616, 32537, 58, 0, 58, 0,
-1010297088, -1897350404, 1481241296, 32537, 1481240736, 32537, 0, 0, 0, 0}
stream = <optimized out>
i = 43
n = 44
#3 0x00007f19744797ab in h2_session_process (session=0x7f195849f0a0,
async=***@entry=0) at h2_session.c:2210
status = 0
c = 0x7f196a6112c8
rv = <optimized out>
mpm_state = 1
trace = 0
#4 0x00007f1974465b2a in h2_conn_run (ctx=***@entry=0x7f196a60e2f8,
c=***@entry=0x7f196a6112c8) at h2_conn.c:212
status = <optimized out>
mpm_state = 0
#5 0x00007f197446ba5b in h2_h2_process_conn (c=0x7f196a6112c8) at
h2_h2.c:658
status = <optimized out>
ctx = 0x7f196a60e2f8
c = 0x7f196a6112c8
#6 0x0000563011b2c730 in ap_run_process_connection
(c=***@entry=0x7f196a6112c8) at connection.c:42
pHook = <optimized out>
n = 2
rv = -1
#7 0x0000563011b2cc80 in ap_process_connection
(c=***@entry=0x7f196a6112c8, csd=***@entry=0x7f196a6110b0) at connection.c:226
rc = <optimized out>
#8 0x00007f1973338e4a in process_socket (bucket_alloc=0x7f19584b9028,
my_thread_num=4, my_child_num=2, sock=0x7f196a6110b0, p=0x7f196a611028,
thd=0x7f197af2dc78) at worker.c:631
current_conn = 0x7f196a6112c8
conn_id = <optimized out>
sbh = 0x7f196a6112c0
#9 worker_thread (thd=0x7f197af2dc78, dummy=<optimized out>) at
worker.c:992
ti = <optimized out>
process_slot = 2
thread_slot = 4
csd = 0x7f196a6110b0
bucket_alloc = 0x7f19584b9028
last_ptrans = <optimized out>
ptrans = 0x7f196a611028
rv = <optimized out>
is_idle = 0
#10 0x00007f197a557494 in start_thread (arg=0x7f1959ea3700) at
pthread_create.c:333
__res = <optimized out>
pd = 0x7f1959ea3700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139746859431680,
-6524335721044288688, 0, 139746901392863, 0, 139747414900800,
6577037691194661712, 6576965816350759760}, mask_was_saved = 0}}, priv =
{pad = {0x0, 0x0, 0x0, 0x0},
data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#11 0x00007f197a299aff in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
No locals.

[Gautas pranešimas]

Laba diena,


Noriu Jus informuoti apie šių metų pasikeitimą dėl atnaujintos visos Lietuvos įmonių bazės 2018 metų sausio vidurio.
Visi juridiniai asmenys pateikti bazėje yra veikiantys, realiai vykdantys veiklą, turintys įdarbintų darbuotojų. Duomenys pagal Sodrą, Registrų centrą.

Bazėje nurodoma ir apyvarta, darbuotojų atlyginimai, darbuotojų skaičius, transporto skaičius ir daug kitų duomenų, kuriuos matysite pavyzdyje.

Duomenis galima filtruoti pagal veiklas, miestus ir kitus duomenis.


Å ią bazę verta turėti visoms įmonėms. Pateiksiu prieÅŸastis:

1) Kontaktai pateikti bazėje direktorių ir kitų atsakingų asmenų, didelė tikimybė Jums surasti naujų klientų, partnerių, tiekėjų, kai tiesiogiai bendrausite su direktoriais, komercijos vadovais.

2) Konkurentų analizavimas, tiekėjų atsirinkimas pagal Jums reikalingus kriterijus, galite atsifiltruoti pagal įmonės dydį, bazėje nurodoma kiek įmonės skolingos Sodrai.

3) Lengva, greita ir patogu dirbti su šia baze, elektroninius pašto adresus galite importuoti į elektroninių laiškų siuntimo programas ar sistemas iš kurių siunčiate elektroninius laiškus.
Taip pat galite importuoti mobiliųjų telefonų numerius į SMS siuntimo programas.


Išsirinkite iš "Veiklų sąrašo" veiklas kurių Jums reikia.
( Sąrašas prisegtas laiške excel faile )

Parašykite, kurias veiklas išsirinkote
ir atsiųsime pavyzdį ir pasiūlymą su sąlygomis įmonių bazei įsigyti



Pagarbiai,
Tadas Giedraitis
Tel. nr. [phone]

[Debian Bug Tracking System]

Your message dated Mon, 02 Jul 2018 16:47:10 +0000
with message-id <E1fa1yo-000ARR-***@fasolo.debian.org>
and subject line Bug#873945: fixed in apache2 2.4.25-3+deb9u5
has caused the Debian Bug report #873945,
regarding apache2-bin: apache2 with http2 segfault
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)