I don’t know if this has become standard in other modules, but a lot of the unique enemies in Pathfinder adventure paths include a tactics section that indicates at what HP threshold they’ll try to flee (e.g., “at 10 HP or less, he tries to escape”). I figure this could be generalized into a standard morale system as follows.
For each encounter, the enemy NPCs have a Morale rating (or, more accurately, a Morale Break Threshold) that usually starts at 0. It’s tracked for the whole encounter, not per individual combatant. Any enemy whose HP drop to or below this rating (including if it equals or exceeds that NPC’s maximum HP) attempts to flee (and counts as Demoralized if prevented from fleeing and forced to fight).
This rating increases in the following ways:
- +1/HD of allies downed in the fight (e.g., an 8 HD enemy is dropped and the Morale rating goes up +8)
- +1/2 HD of allies that successfully flee (after they, themselves, became demoralized)
- Intimidate Check: PC makes an Intimidate check as a standard action, DC 10 + Current Morale, +1/2 Intimidating PC’s HD
- Ad Hoc for other demoralizing events
This rating reduces in the following ways:
- -2/HD of PCs and PC allies downed in the fight
- Rally Check: NPC leader makes Leadership check (HD + Cha mod + misc mods) as a standard action, DC 10 + Current Morale, -1/2 rallying NPC’s HD
- Ad Hoc for other morale-boosting events
Particularly brave or cowardly NPCs may treat the current rating as half or double (or more fine-grained distinctions if you’re willing to do more math).
The upshot of this system is that it allows the GM to throw in more mooks without overwhelming the PCs, as technically harder encounters of multiple NPCs will actually end earlier as the PCs manage to eliminate some foes.