Right on. I've worked for a number of startups early in my career and they tended to be fun since you have wide range of responsibilities and will actually have to produce something in the end. The upside of working at startup is that you will learn a lot and be directly hands on, whether its coding the program, creating a CAD design, or learning how to use a new manufacturing technique (3D printing parts for my previous job).
There were very few layers of bureaucracy or management at the startups where I worked and a lot of what you learn early on will be a useful skill to fall back on no matter where you end up careerwise. Plus there's always the possibility that your startup will turn into a Unicorn (like Google, Facebook, or Netflix 15 years ago) and you end up becoming a multi-millionaire with an IPO if you are among the early hires.
The downside of startup, like the other mentioned, and that you probably know from all the horror stories out there, is that there is no work life balance, very few layers or bureaucracy that you can hide behind if you can't produce results, and there is always the risk that the startup will fold literally overnight and you only find out about it when you show up to work the next day and find all the locks to your office have been changed by the banks or creditors. But if you're young and have no family obligation, and you have a passion for whatever product or service and feel that you can get along with the personalities at these startups, then I would definitely take a leap of faith and give it a go.