Discussion:
Bug#912277: apache2: does not start any more: AH01903: Failed to configure CA certificate chain!
Thorsten Glaser
2018-10-29 19:31:54 UTC
Permalink
Package: apache2
Version: 2.4.35-1
Severity: important

After a recent upgrade, apache2 does not start any more:

***@tglase:~ $ cat /var/log/apache2/error.log
[Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Failed to configure CA certificate chain!
[Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fatal error initialising mod_ssl, exiting. See /var/log/apache2/error.log for more information
AH00016: Configuration Failed

The certificate itself and the chain are ok, though:

***@tglase:~ $ openssl verify -CApath /etc/ssl/certs -show_chain -purpose sslserver -verify_hostname tglase.lan.tarent.de -untrusted /etc/ssl/W_lan_tarent_de.ca /etc/ssl/W_lan_tarent_de.cer
/etc/ssl/W_lan_tarent_de.cer: OK
Chain:
depth=0: CN = *.lan.tarent.de (untrusted)
depth=1: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = RapidSSL TLS RSA CA G1 (untrusted)
depth=2: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2

Postfix, on the same system, using the same certificates…
smtpd_tls_cert_file = /etc/ssl/W_lan_tarent_de.cer
smtpd_tls_key_file = /etc/ssl/private/W_lan_tarent_de.key
smtpd_tls_CAfile = /etc/ssl/W_lan_tarent_de.ca
… runs fine, so this must be some regression in Apache2.

-- Package-specific info:

-- System Information:
Debian Release: buster/sid
APT prefers unreleased
APT policy: (500, 'unreleased'), (500, 'buildd-unstable'), (500, 'unstable')
Architecture: x32 (x86_64)
Foreign Architectures: i386, amd64

Kernel: Linux 4.18.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8)
Shell: /bin/sh linked to /bin/lksh
Init: sysvinit (via /sbin/init)

Versions of packages apache2 depends on:
ii apache2-bin 2.4.35-1
ii apache2-data 2.4.35-1
ii apache2-utils 2.4.35-1
ii dpkg 1.19.2
ii lsb-base 9.20170808
ii mime-support 3.61
ii perl 5.26.2-7+b1
ii procps 2:3.3.15-2

Versions of packages apache2 recommends:
ii ssl-cert 1.0.39

Versions of packages apache2 suggests:
ii apache2-doc 2.4.35-1
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii dillo [www-browser] 3.0.5-4
ii links2 [www-browser] 2.17-1
ii lynx [www-browser] 2.8.9rel.1-2
ii opera-static [www-browser] 9.64.2480.gcc4.qt3

Versions of packages apache2-bin depends on:
ii libapr1 1.6.3-3
ii libaprutil1 1.6.1-3+b1
ii libaprutil1-dbd-pgsql 1.6.1-3+b1
ii libaprutil1-dbd-sqlite3 1.6.1-3+b1
ii libaprutil1-ldap 1.6.1-3+b1
ii libbrotli1 1.0.6-1
ii libc6 2.27-6
ii libcurl4 7.61.0-1
ii libjansson4 2.11-1
ii libldap-2.4-2 2.4.46+dfsg-5+x32.1
ii liblua5.2-0 5.2.4-1.1+b1
ii libnghttp2-14 1.34.0-1
ii libpcre3 2:8.39-11
ii libssl1.1 1.1.1-1
ii libxml2 2.9.4+dfsg1-7+b1
ii perl 5.26.2-7+b1
ii zlib1g 1:1.2.11.dfsg-1

Versions of packages apache2-bin suggests:
ii apache2-doc 2.4.35-1
pn apache2-suexec-pristine | apache2-suexec-custom <none>
ii dillo [www-browser] 3.0.5-4
ii links2 [www-browser] 2.17-1
ii lynx [www-browser] 2.8.9rel.1-2
ii opera-static [www-browser] 9.64.2480.gcc4.qt3

Versions of packages apache2 is related to:
ii apache2 2.4.35-1
ii apache2-bin 2.4.35-1

-- Configuration Files:
/etc/apache2/apache2.conf changed:
DefaultRuntimeDir ${APACHE_RUN_DIR}
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
HostnameLookups Off
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf
Include ports.conf
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/>
Options Indexes FollowSymLinks ExecCGI
AddHandler cgi-script .cgi
AllowOverride None
Require all granted
</Directory>
AccessFileName .htaccess
<FilesMatch "^\.ht">
Require all denied
</FilesMatch>
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
IncludeOptional conf-enabled/*.conf
IncludeOptional sites-enabled/*.conf

/etc/apache2/sites-available/default-ssl.conf changed:
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin ***@localhost
DocumentRoot /var/www/html
# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
# SSL Engine Switch:
# Enable/Disable SSL for this virtual host.
SSLEngine on
# *.lan.tarent.de
SSLCertificateKeyFile /etc/ssl/private/W_lan_tarent_de.key
SSLCertificateFile /etc/ssl/W_lan_tarent_de.cer
SSLCertificateChainFile /etc/ssl/W_lan_tarent_de.ca
# A self-signed (snakeoil) certificate can be created by installing
# the ssl-cert package. See
# /usr/share/doc/apache2/README.Debian.gz for more info.
# If both key and certificate are stored in the same file, only the
# SSLCertificateFile directive is needed.
#SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
#SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
# Server Certificate Chain:
# Point SSLCertificateChainFile at a file containing the
# concatenation of PEM encoded CA certificates which form the
# certificate chain for the server certificate. Alternatively
# the referenced file can be the same as SSLCertificateFile
# when the CA certificates are directly appended to the server
# certificate for convinience.
#SSLCertificateChainFile /etc/apache2/ssl.crt/server-ca.crt
# Certificate Authority (CA):
# Set the CA certificate verification path where to find CA
# certificates for client authentication or alternatively one
# huge file containing all of them (file must be PEM encoded)
# Note: Inside SSLCACertificatePath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCACertificatePath /etc/ssl/certs/
#SSLCACertificateFile /etc/apache2/ssl.crt/ca-bundle.crt
# Certificate Revocation Lists (CRL):
# Set the CA revocation path where to find CA CRLs for client
# authentication or alternatively one huge file containing all
# of them (file must be PEM encoded)
# Note: Inside SSLCARevocationPath you need hash symlinks
# to point to the certificate files. Use the provided
# Makefile to update the hash symlinks after changes.
#SSLCARevocationPath /etc/apache2/ssl.crl/
#SSLCARevocationFile /etc/apache2/ssl.crl/ca-bundle.crl
# Client Authentication (Type):
# Client certificate verification type and depth. Types are
# none, optional, require and optional_no_ca. Depth is a
# number which specifies how deeply to verify the certificate
# issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth 10
# SSL Engine Options:
# Set various options for the SSL engine.
# o FakeBasicAuth:
# Translate the client X.509 into a Basic Authorisation. This means that
# the standard Auth/DBMAuth methods can be used for access control. The
# user name is the `one line' version of the client's X.509 certificate.
# Note that no password is obtained from the user. Every entry in the user
# file needs this password: `xxj31ZMTZzkVA'.
# o ExportCertData:
# This exports two additional environment variables: SSL_CLIENT_CERT and
# SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
# server (always existing) and the client (only existing when client
# authentication is used). This can be used to import the certificates
# into CGI scripts.
# o StdEnvVars:
# This exports the standard SSL/TLS related `SSL_*' environment variables.
# Per default this exportation is switched off for performance reasons,
# because the extraction step is an expensive operation and is usually
# useless for serving static content. So one usually enables the
# exportation for CGI and SSI requests only.
# o OptRenegotiate:
# This enables optimized SSL connection renegotiation handling when SSL
# directives are used in per-directory context.
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
# SSL Protocol Adjustments:
# The safe and default but still SSL/TLS standard compliant shutdown
# approach is that mod_ssl sends the close notify alert but doesn't wait for
# the close notify alert from client. When you need a different shutdown
# approach you can use one of the following variables:
# o ssl-unclean-shutdown:
# This forces an unclean shutdown when the connection is closed, i.e. no
# SSL close notify alert is send or allowed to received. This violates
# the SSL/TLS standard but is needed for some brain-dead browsers. Use
# this when you receive I/O errors because of the standard approach where
# mod_ssl sends the close notify alert.
# o ssl-accurate-shutdown:
# This forces an accurate shutdown when the connection is closed, i.e. a
# SSL close notify alert is send and mod_ssl waits for the close notify
# alert of the client. This is 100% SSL/TLS standard compliant, but in
# practice often causes hanging connections with brain-dead browsers. Use
# this only for browsers where you know that their SSL implementation
# works correctly.
# Notice: Most problems of broken clients are also related to the HTTP
# keep-alive facility, so you usually additionally want to disable
# keep-alive for those clients, too. Use variable "nokeepalive" for this.
# Similarly, one has to force some clients to use HTTP/1.0 to workaround
# their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
# "force-response-1.0" for this.
# BrowserMatch "MSIE [2-6]" \
# nokeepalive ssl-unclean-shutdown \
# downgrade-1.0 force-response-1.0
</VirtualHost>
</IfModule>


-- no debconf
Stefan Fritsch
2018-11-03 12:24:05 UTC
Permalink
Post by Thorsten Glaser
[Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Failed to
configure CA certificate chain!
[Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fatal
error initialising mod_ssl, exiting.
See /var/log/apache2/error.log for more information AH00016: Configuration
Failed
Have you looked into /var/log/apache2/error.log if there is more information?
If there is none, try adding loglevel ssl:debug and re-try.
Thorsten Glaser
2018-11-04 17:14:04 UTC
Permalink
Hi Stefan,
[Mon Oct 29 20:18:58.090841 2018] [ssl:emerg] [pid 17306] AH01903: Fail=
ed to
configure CA certificate chain!
[Mon Oct 29 20:18:58.090919 2018] [ssl:emerg] [pid 17306] AH02311: Fata=
l
error initialising mod_ssl, exiting.
See /var/log/apache2/error.log for more information AH00016: Configurat=
ion
Failed
=20
Have you looked into /var/log/apache2/error.log if there is more informa=
tion?=20

the thing you quoted was exactly what was in /var/log/apache2/error.log
as the =E2=80=9Ccat=E2=80=9D showed=E2=80=A6

I just hit this on another machine, it=E2=80=99s the 2.4.35-1 =E2=86=92 2.4=
=2E37-1 upgrade
that caused the failure.
If there is none, try adding loglevel ssl:debug and re-try.
OK, thanks for the debugging help.

That gives:

[Sun Nov 04 17:05:02.839408 2018] [ssl:info] [pid 18196] AH01887: Init: Ini=
tializing (virtual) servers for SSL
[Sun Nov 04 17:05:02.839427 2018] [ssl:info] [pid 18196] AH01914: Configuri=
ng server ci-busyapps.lan.tarent.de:443 for SSL protocol
[Sun Nov 04 17:05:02.839433 2018] [ssl:debug] [pid 18196] ssl_engine_init.c=
(1748): AH10083: Init: (ci-busyapps.lan.tarent.de:443) mod_md support is un=
available.
[Sun Nov 04 17:05:02.839729 2018] [ssl:emerg] [pid 18196] AH01903: Failed t=
o configure CA certificate chain!
[Sun Nov 04 17:05:02.839739 2018] [ssl:emerg] [pid 18196] AH02311: Fatal er=
ror initialising mod_ssl, exiting. See /var/log/apache2/error.log for more =
information
AH00016: Configuration Failed

So perhaps the mod_ssl backport / new feature was bad?

On a hunch, I tried a2enmod md, but that does not change much:

[Sun Nov 04 17:05:47.417353 2018] [ssl:info] [pid 18229] AH01887: Init: Ini=
tializing (virtual) servers for SSL
[Sun Nov 04 17:05:47.417371 2018] [ssl:info] [pid 18229] AH01914: Configuri=
ng server ci-busyapps.lan.tarent.de:443 for SSL protocol
[Sun Nov 04 17:05:47.417377 2018] [ssl:debug] [pid 18229] ssl_engine_init.c=
(1748): AH10083: Init: (ci-busyapps.lan.tarent.de:443) mod_md support is av=
ailable.
[Sun Nov 04 17:05:47.417663 2018] [ssl:emerg] [pid 18229] AH01903: Failed t=
o configure CA certificate chain!
[Sun Nov 04 17:05:47.417673 2018] [ssl:emerg] [pid 18229] AH02311: Fatal er=
ror initialising mod_ssl, exiting. See /var/log/apache2/error.log for more =
information
AH00016: Configuration Failed


More debugging data points: this did not occur immediately after
the package upgrade, only when I did an /etc/init.d/apache2 stop
followed by start.

Worse, this persists after downgrading apache2, apache2-bin,
apache2-data, apache2-utils to 2.4.35-1 (?!?!?!).

Dazed and confused,
//mirabilos
--=20
tarent solutions GmbH
Rochusstra=C3=9Fe 2-4, D-53123 Bonn =E2=80=A2 http://www.tarent.de/
Tel: +49 228 54881-393 =E2=80=A2 Fax: +49 228 54881-235
HRB 5168 (AG Bonn) =E2=80=A2 USt-ID (VAT): DE122264941
Gesch=C3=A4ftsf=C3=BChrer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Ale=
xander Steeg
Thorsten Glaser
2018-11-04 17:24:52 UTC
Permalink
Dixi quod=E2=80=A6
I just hit this on another machine, it=E2=80=99s the 2.4.35-1 =E2=86=92 2=
=2E4.37-1 upgrade
that caused the failure.
Given that I originally reported this against 2.4.35-1 and that=E2=80=A6
More debugging data points: this did not occur immediately after
the package upgrade, only when I did an /etc/init.d/apache2 stop
followed by start.
=20
Worse, this persists after downgrading apache2, apache2-bin,
apache2-data, apache2-utils to 2.4.35-1 (?!?!?!).
=E2=80=A6 this is obviously nonsense.

2.4.33-3+b1 is the oldest version I can downgrade to, and it
also exhibits the problem. WTF.

We did switch certificates recently, but OpenSSL accepts them=E2=80=A6

Still puzzled,
//mirabilos
--=20
tarent solutions GmbH
Rochusstra=C3=9Fe 2-4, D-53123 Bonn =E2=80=A2 http://www.tarent.de/
Tel: +49 228 54881-393 =E2=80=A2 Fax: +49 228 54881-235
HRB 5168 (AG Bonn) =E2=80=A2 USt-ID (VAT): DE122264941
Gesch=C3=A4ftsf=C3=BChrer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Ale=
xander Steeg
Thorsten Glaser
2018-11-04 17:36:19 UTC
Permalink
retitle 912277 apache2: SSLCertificateChainFile silently ignored, causing A=
H01903 startup failure
thanks
Post by Thorsten Glaser
2.4.33-3+b1 is the oldest version I can downgrade to, and it
also exhibits the problem. WTF.
This is a real WTF. I found https://serverfault.com/a/892300/189656
and thought =E2=80=9Chey, Apache 2 still documents SSLCertificateChainFile,
plus it=E2=80=99s the proper way to specify the chain given it=E2=80=99s no=
rmally
separate from the certificates, and there=E2=80=99s no warning message abou=
t
that directive, but let=E2=80=99s give it a shot=E2=80=9D.

So I did:

# cat /etc/ssl/W_lan_tarent_de.cer /etc/ssl/W_lan_tarent_de.ca >/etc/ssl/co=
mbined-cer-chain.pem

Then I edited /etc/apache2/sites-enabled/default-ssl.conf, commenting
out SSLCertificateFile and SSLCertificateChainFile, and adding

=09SSLCertificateFile /etc/ssl/combined-cer-chain.pem

***@tglase:~ $ sudo cleanenv / /etc/init.d/apache2 stop
Stopping Apache httpd web server: apache2.
Server was not running ... (warning).
***@tglase:~ $ sudo cleanenv / /etc/init.d/apache2 start
Starting Apache httpd web server: apache2 ..

=2EoO(wait, what?)

***@tglase:~ $ curl --head https://$(hostname -f)/=20
HTTP/1.1 200 OK
Date: Sun, 04 Nov 2018 17:34:29 GMT
Server: Apache/2.4.35 (Debian)
Content-Type: text/html;charset=3DUTF-8

=2EoO(what now?)

So it turns out that, ever since some upgrade, the directive
SSLCertificateChainFile is *silently* ignored, but this only
becomes apparent when you stop+start instead of restart (so
they are *still* not equivalent =E2=98=B9).

I don=E2=80=99t think this acceptable. Ideally, the option would be
still supported; it does no harm and has worked for decades.

If that=E2=80=99s not desired, it MUST yield a warning.

bye,
//mirabilos
--=20
tarent solutions GmbH
Rochusstra=C3=9Fe 2-4, D-53123 Bonn =E2=80=A2 http://www.tarent.de/
Tel: +49 228 54881-393 =E2=80=A2 Fax: +49 228 54881-235
HRB 5168 (AG Bonn) =E2=80=A2 USt-ID (VAT): DE122264941
Gesch=C3=A4ftsf=C3=BChrer: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Ale=
xander Steeg
Debian Bug Tracking System
2018-11-04 17:39:14 UTC
Permalink
retitle 912277 apache2: SSLCertificateChainFile silently ignored, causing AH01903 startup failure
Bug #912277 [apache2] apache2: does not start any more: AH01903: Failed to configure CA certificate chain!
Changed Bug title to 'apache2: SSLCertificateChainFile silently ignored, causing AH01903 startup failure' from 'apache2: does not start any more: AH01903: Failed to configure CA certificate chain!'.
thanks
Stopping processing here.

Please contact me if you need assistance.
--
912277: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=912277
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Stefan Fritsch
2018-11-04 18:11:50 UTC
Permalink
Post by Thorsten Glaser
This is a real WTF. I found https://serverfault.com/a/892300/189656
and thought “hey, Apache 2 still documents SSLCertificateChainFile,
plus it’s the proper way to specify the chain given it’s normally
separate from the certificates, and there’s no warning message about
that directive, but let’s give it a shot”.
SSLCertificateChainFile still does something, it's not a no-op in the source.
Maybe you are hitting https://bz.apache.org/bugzilla/show_bug.cgi?id=62880 .
That would cause non-deterministic behavior (depending on what went on before
the SSLCertificateChainFile is encountered). If you have time, you could try
the patch attached to that bug. Right now, I don't have time to prepare test
packages.

Cheers,
Stefan

Loading...