Netscape CA handling: When is a CA Not a CA?


Now that many people are using such packages as SSLeay to generate their own certificates for SSL and S/MIME use, I frequently have to mail people informing them that their CA certificate is invalid or I get asked about what the ca-fix program is for and why people should use it. This is not just restricted to SSLeay: I've seen invalid CA certificates from commercial packages like Netscape Certificate server. This document describes the problems and their solutions.

Note: newer versions of Netscape have changed their behaviour and the main problem appears to be fixed. If you are running 4.06 or later of 4.5PR1 or later then check out the latest versions section.

The Symptoms.

Anyone who uses Netscape Communicator for a mail or news reader may occasionally see signed messages. Often this is just the standard "signed" box but occasionally it is the cross with "invalid signature".

When you click on the "invalid signature" box it will give you a reason for the signature failure in a security window. There are several common reasons.

The message can be altered for several reasons, maybe a mail gateway or a mailing list server has modified it in some way and thus invalidated the signature. The CA problems are the ones that will be discussed here.

If you get the "not trusted" message then you can go through the list of signers (Security->Signers) find the CA and set it to be trusted if you so wish. If you then reopen the message the message shows the usual "signed" box.

If you get "Issuer Certificate is invalid" then the CA will not appear in the list and you can't trust it. If you try to add the CA in question yourself then it will say the CA is already in the database: but it is not listed.

At this point the database is stuck in this state. You can still use it but the CA in question will be left unstrusted and invisible. A natural question is how the message was sent it the first place if the issuer is invalid. This will be answered in subsequent sections but first a bit of backround.

CA flag: X509 V3 extensions.

A typical certificate will be signed by a CA and that CA may be signed by subsequent CAs until a self-signed root certificate is reached. For example a users certificate might be signed by an intermediate CA and the intermediate CA signed by a root authority. Since signing requires knowledge of the private key a normal user cannot forge a certificate signed by the intermediate CA because they do not possess the CA's private key.

A user will possess their own private key. What is to stop an end user issuing certificates and pretending to be a CA themselves?

The answer is that most certificates these days are V3 certificates which can include various certificate extensions. One of these is called basicConstraints. It includes a flag called cA to indicate whether the certificate belongs to a CA or not. Thus the end user certificate will not have cA set and the intermediate CA (and root CA) will have cA set.

If an end user tries to impersonate a CA anyone attempting to verify such a bogus certificate can see it is not signed by a CA and take appropriate action.

This is a considerable simplification of the overall picture but it will suffice for now. A fuller description is given in the PKIX standards documentation for example.

Auto inserting CA certificates.

If Netscape regards a CA as valid and it does not currently have the CA certificate in its database it can be auto added as untrusted. This can happen under several different circumstances. The criterion which Netscape uses for determining if a certificate is valid is described here. One test is if the basicConstraints cA flag is set. If the CA is considered invalid then it either gets ignored or (erroneously) permanently and invisibly added as untrusted and invalid.

Untrusted (the term Netscape uses) is not really the right term. If a CA is auto added, but its parent CA is trusted, then the untrusted CA is trusted because it inherits trust from its parent CA. If a CA is set to be trusted then it is trusted even if the parent CA is not.

Installing CA certificates: anything goes.

There is another way to get a certificate into Netscape. If Netscape opens a file (e.g. by clicking on a link) and it is sent as MIME type application/x-x509-ca-cert and it is a certificate then a sequence of dialog boxes appears allowing it to be added and trusted.

Note I said certificate and not CA certificate. With this method you can install end user certificates, certificates with invalid signatures just about anything that vaguely looks like a certificate can be added and trusted as a CA.

When you actually try to use an invalid CA installed by this method all seems fine because it is trusted. If you send S/MIME mail with this CA then the poor recipient gets it added using the validity criteria and it is deemed invalid.

A whole organisation could install certificates for their users by first adding the CA certificate and everything seem fine because they all had the CA added as trusted. Only when they start sending signed mail to people who haven't installed the CA certificate does the problem become apparent.

Diagnosing the problem.

If all seems OK to someone sending mail with an invalid CA certificate, how can the problem be diagnosed? There are several ways, one is described below.
  1. Open messenger and create a new message, click the sign box and save it as a draft.
  2. Open the message in the Drafts folder and check it appears as signed.
  3. Exit netscape.
  4. Locate your user directory and move the files key3.db and cert7.db somewhere safe.
  5. Restart Netscape and open the draft signed message again.
  6. It should say that the signature is invalid.
  7. Click on the box and check the reason.
  8. If the reason says that the issuer certificate is invalid then you have a problem: as does anyone you send signed mail to: stop at this point and fix the problem. If it merely says that the CA is not trusted then you are OK.
  9. Select the new CA in Security->Signers and set it to trust email users.
  10. Reopen the message and it should now say just have the regular "signed" box.
  11. Exit communicator are restore your key3.db and cert7.db files.
What the above is doing is simulating sending mail to a user who hasn't installed your CA certificate.

Solutions to the problem.

If you have a commercial package it should include some documentation on how to add certificate extensions to your CA certificate, specifically basicConstraints.

If you are using SSLeay then the normal method of generating a CA ( -newca) generates an invalid V1 CA. I have written certificate patching program called ca-fix that can be used to fix such certificates. For more information see the ca-fix documentation.

Latest Versions of Netscape.

If you receive mail from Netscape Communicator 4.06 or 4.5 or later then you may notice something odd. Even if is is signed by a valid root CA, Netscape will no longer appear to auto add it when you receive mail signed by it.

The reason for this is that the S/MIME behaviour has changed recently. The root CA is no longer included with signed messages. This means that if you have an uninstalled S/MIME root CA the recipient has to add it manually.

I reported this as a bug and apparently it is an intentional change. Personally I think this is a very bad idea and I've requested that the old behaviour should be possible via an option: if you agree then I suggest you complain to Netscape. At best this is a kludge to avoid the problem though there are potentially other commercial reasons for its inclusion.

The good news though is that the problem appears to be fixed. If you now receive mail signed by an invalid CA then checking the signature can still give "issuer certificate is invalid" but an invalid CA is not added to the database: so you can add it manually.

As should be apparent the above test for the problem no longer works because the root CA is not auto added when you read signed S/MIME mail (because it isn't there!). There is however an easier way to diagnose if Netscape doesn't like your CA.

  1. Select the CA it in Security->Signers.
  2. Select Edit, uncheck the three "Accept boxes" and click on OK.
  3. Click on Verify.
  4. If it says just "Certificate not trusted" then all is OK. If it also says "Not a valid Certificate Authority" then it regards it as invalid.
  5. Click Edit and retrurn the "Accept boxes" to their original state.
If you try this on some of the auto installed CAs you will find it regards some of those as invalid.