Upon the receipt of a packet with both protocol headers, the processing sequence should be authentication followed by decryption. Why decrypt if you are not certain of the origin?

So, the sender should first apply ESP and then AH to the outbound traffic. In fact, this sequence is an explicit requirement for transport mode IPSec processing.

When using both ESP and AH, one must consider whether ESP authentication should be turned on since AH authenticates the packet anyway. The answer depends on the relative extent of the SAs used. Turning ESP authentication on makes sense when the ESP SA extends beyond the AH SA. Here, ESP can avoid spoofing attacks in the intranet.

In general, the transport mode is used between the endpoints of a connection, and tunnel mode is used between two machines when at least one of them is a gateway.