Having spent my first years as a developer doing just that, I can confirm it's viable and (as long as you really are scrupulous about how you do it) you can keep away from infringing protected intellectual property.
Personally, though, I'd expand the range of options somewhat.
A bigger, better product selling for less is a full-frontal attack on a bigger, richer competitor. You're guaranteeing a long lead time, and there's often the danger of confusing "more" with "better." But this is fertile ground for some products. For instance, is there a technology product that's easy to map out in detail, but that has poor associated tooling? If that affects customers, then a product that's comparable/compatible but has improved tools could be an attack at their point of weakness.
A smaller, simpler product solving one specific problem super well is a more common startup tactic. Your competitor's big, complex product has a number of use cases: it's the perfect tool for some, cumbersome for others, barely adequate for maybe a few. This approach has two huge benefits. First, it naturally leads you to engage with customers and their problems rather than spending all your time studying the competition. Second, it's very hard for the established player to respond to you once they notice you're stealing market share.
A comparable product being sold differently is another approach worth pursuing. Does the competitor sell direct? Then see how you could create a channel-friendly offering. Does the established player focus on enterprise sales? Then focus your product on the individual user. Fight packaged software with SAAS, desktop systems with super shiny mobile apps. Turn their pricing on its head. Find one thing to change that gives you a clear run at a better-defined slice of the market.
Good luck, with whatever applecart you decide to overturn!