saw 夫人 out of textbook order and im mad

Step one, you master the N5 kanji. Step two, you do N4. But today I saw 夫人 on a news app and it completely broke the sequence. 夫 is husband, so why does adding 人 make it Madame? If you are trying to learn this by 'guessing' from context like those lazy immersion-only people, you are wasting your time. You have to learn the official radical order first or you're just faking it. Now my daily study plan is ruined.