When a wave hits the hole, part of it continues into the chamber and part of it is reflected. The wave travels through the chamber, hits the back wall of the muffler and bounces back out of the hole. The length of this chamber is calculated so that this wave leaves the resonator chamber just after the next wave reflects off the outside of the chamber. Ideally, the high-pressure part of the wave that came from the chamber will line up with the low-pressure part of the wave that was reflected off the outside of the chamber wall, and the two waves will cancel each other out.
The animation below shows how the resonator works in a simplified muffler.
Waves canceling inside a simplified muffler
In reality, the sound coming from the engine is a mixture of many different frequencies of sound, and since many of those frequencies depend on the engine speed, the sound is almost never at exactly the right frequency for this to happen. The resonator is designed to work best in the frequency range where the engine makes the most noise; but even if the frequency is not exactly what the resonator was tuned for, it will still produce some destructive interference.
Some cars, especially luxury cars where quiet operation is a key feature, have another component in the exhaust that looks like a muffler, but is called a resonator. This device works just like the resonator chamber in the muffler -- the dimensions are calculated so that the waves reflected by the resonator help cancel out certain frequencies of sound in the exhaust.
There are other features inside this muffler that help it reduce the sound level in different ways. The body of the muffler is constructed in three layers: Two thin layers of metal with a thicker, slightly insulated layer between them. This allows the body of the muffler to absorb some of the pressure pulses. Also, the inlet and outlet pipes going into the main chamber are perforated with holes. This allows thousands of tiny pressure pulses to bounce around in the main chamber, canceling each other out to some extent in addition to being absorbed by the muffler's housing.