Peter T Lesson writes that the trial by ordeal was "an effective test of guilt", contrary to its brutal suggestion of divine judgment. As unscientific as it seems to determine guilt by dunking someone's arm in boiling water, the threat of it is a cunning way to elicit truth in the absense of evidence—so long as the subject is confident God will protect them if they are innocent.
It's rather like modern lie detectors, generally inadmissible in court but used pervasively in interrogations.
Suppose you're guilty: you know you stole your neighbour's cat, and so does God. In this case, you expect that if you undergo the ordeal, God will let the boiling water burn you, evidencing your guilt. Thus, you'll have to pay the large fine – and your hand will be boiled to rags to boot. In contrast, if you confess, you'll save a bit of money, not to mention your hand. So, if you're guilty, you'll confess.
Now suppose you're innocent: you know you didn't steal your neighbour's cat, and again so does God. In this case, you expect that if you undergo the ordeal, God will perform a miracle that prevents the boiling water from burning you, evidencing your innocence. Thus, you won't have to pay any fine – and you'll keep your hand intact. This is better than if you confess to stealing the cat, in which case you'd have to pay a fine for a theft you didn't commit. So, if you're innocent, you'll undergo the ordeal.
The trick is, of course, that the water was usually a safe temperature. The heat of the water is the priest's verdict, not his trier of fact.
It's fascinating and convincing, not least because it describes a complex ritual performance dependent on a stable, predictable social context. The sort of thing that becomes dangerous when the context changes.