publish a website or selling a software product, is it mandatory to register a company?
Generally speaking, it is not mandatory to incorporate and many people run a personal blog or very small lifestyle business without being incorporated.
But a incorporating is very important, because it can shield your person from legal liabilities (lawsuits etc). Without a proper company structure you are personally liable, and if you do something wrong in the extreme (and very rare) case you can loose everything you own including your house. For some ideas and perspectives have a look at Dana Shultz' posts (targeted at California, USA) and the conversations here on the site. Which company structure to choose depends on several things, most importantly the expected sales and risk for the venture.
Since website and software can easily sold cross countries boundaries, is there anything one must know before he does this?
Yes, depending on location there can be tax code considerations, VAT rules and much more.