He had to worry about money all right. He never rich and found it difficult to pay the bills. Being upper class is not the same thing as being rich. He regarded having a team of servants as indispensable for life, so he had to work hard to earn enough money to pay for them.