The main value comes from facilitating the allocation of capital to sectors such as health care and technology and in enabling people to "time shift" their investment and use of capital. In other words, when you are young and buy a house with a big mortgage, you're using other people's capital. As you get older and (in theory) have saved for your own retirement, you've paid down your mortgage and you've become a source of capital for others.
Part of the reason for the large scale of this sector is that pretty much every company as well as most individuals have significant needs for financial services and this need increases as the size of the company or the individuals portfolio increases.
The "lucrative" aspect stems from a few factors - first, you can charge fees based on your expertise, your processes, your own capital, and your ability to raise capital from various sources. Second, companies and individuals are willing to pay for specialized expertise when it can enable new value for the company or individual - for example - if you need to manage currency in 14 countries, you may find it quite helpful and valuable to have an experienced partner assisting you with the strategy and necessary transactions. Third, there can be a non-linear relationship between input and output. In other words, the effort required to manage a $100M transaction is not 100X the effort to manage a $1M transaction.
In theory, competitive pressures should force the rates for a $100M transaction to be more in line with the effort required and thus decrease the margin. However, there's a significant "reputation" factor here - if you're on a board and you're involved with a $100M transaction and Goldman Sachs says their fee is $2.5M and another firm with zero reputation says they can do it for $1.5M, you're likely to pick Goldman to avoid any future potential risk to you if there were to be a problem with the transaction.
Nice work if you can get it!