You have two options in the UK, operate as a sole trader, or form a private limited company.
As a sole trader you pay special rate of income tax and national insurance, but you are personally liable for your work and any debts you take on board.
As a private LC, you would generally take a salary up to the national insurance start band and pay the rest in share dividends to minimise your tax burden. You just need to be careful of IR35, the rule that says words to the effect of "you must really be working for yourself, not indirectly for someone else". If you did fall under IR35, your tax burden would be much heavier.
The advantage of a private LC is your liability for it's debt is limited (thus the name) to the unpaid share capital, this is generally 100GBP by default. The disadvantage is the administrative overhead is more than for a sole trader.
The main insurance you'd require is professional indemnity insurance, which covers you for liability for costs resulting from problems in your professional work. Any insurance will also add general public liability insurance to that in the case of a limited company, that's compulsory.
We're a 2 person company, incorporated as a private limited company in the UK. The overhead is greater, though the accounting effort required in about the same for both. We use a piece of UK centric online accounting software called Free Agent. It's designed for small companies and sole traders, I can't recommend it enough, it's saved me probably 10-20 hours per month of time. It you don't intend to earn much, it's price tag much be an issue, although if you recommend it to 10 people who go on to use it, you get it for free. (I'm not trying to do this, I recommend against it, in fact, until you're earning enough).
For professional indemnity we found Hiscox Insurance to provide the cheapest cover we located. The price varies by your revenue and whether you work in the US, etc., but overall I could recommend them. But certainly do a search yourself.
I wish you luck, it's one of the toughest possible markets out there, you will need some angle on your business to stand out over others.