There are four different authentication methods allowed in either
Main or Aggressive Mode. These methods are: digital signature, two forms of authentication with public key encryption, or a
pre-shared key.
For an authentication with digital signatures, message hashes are
signed and verified; for authentication with either public key
encryption or pre-shared keys, the hashes directly authenticate
the exchange.
When using public keys for authentication, the Phase 1 exchange can
be accomplished either by using signatures or by using public key
encryption. With signatures, the exchange is authenticated by signing
a mutually obtainable hash. RSA signatures must be encoded as a private
key encryption in PKCS #1 format and not as a signature in PKCS #1
format. This is because the latter includes the Object Identifier
(OID) of the hash algorithm, and that OID is invalid in IPSec. Also,
authentication with public key encryption allows for identity
protection with Aggressive Mode.