Paddling can, potentially, be a very expensive hobby. Look at the people who take it semi-seriously who own (multiple) boats that cost almost as much as my car, the price of good drysuits, the carbon fibre paddles and the list goes on. But the cost of entry doesn’t need to be prohibitive – you do not need to spend that much money to get into it.
Start by going out with a club or by hiring a kayak or paddleboard at the beach – somewhere you’ll have people keeping an eye on you and you can try things out at your own pace. There’s no point in buying your own boat/board before you’ve even tried it. A club will provide everything you need and a hire place will provide a buoyancy aid.
Water shoes / something for your feet

I think the first and possibly most important thing is to buy something to protect your feet. A pair of old trainers might be good enough or you might want to invest in a pair of neoprene beach shoes or surfing booties. Personally, I still haven’t figured out exactly what I like – I have a pair of neoprene socks from my caving days which work great on the water but are really bad on the shore when I feel every single stone and pebble right through my feet. Something with soles might be good.
Buoyancy aid

You might be thinking this is the absolute basic most-essential item of kit, why haven’t you put this first??But as a beginner, you’ll probably be borrowing or hiring equipment and going out with someone who knows what they’re doing and they will probably provide you with a buoyancy aid. But once you’ve decided that this is something you want to do, a buoyancy aid is a good investment. You can spend a fortune on a buoyancy aid or you can go to Decathlon and buy one that’s basic but functional and yours. If, for whatever reason, you’re not going out with someone who’ll lend you a buoyancy aid in the early days, you should probably find someone who will but if not, the buoyancy aid needs to go right up your shopping list. There is nothing more important than your safety.
Glasses cord

Bit of a wildcard here but sunglasses are a great thing if there’s sun on the water and if you capsize, those sunglasses – or your normal glasses – will end up on the bottom of the sea/lake/river/wherever you’re paddling. I’ve had one cord that I’ve worn for years but one of the ends is tearing and I’ll need to get a new one for next year. The boathouse has also invested in a packet of them to lend our girls since we’ve had so many worried about being able to see vs keeping their only (expensive) glasses safe on the water.
Honestly, as a beginner that’s about all you really need. As a kayaker, I like to wear a wetsuit over a swimsuit but our girls come to the boathouse in old t-shirts and leggings, and paddleboarders seem to wear anything from bikinis up to tracksuits – it depends on how likely you are to end up in the water, the weather, how comfortable you are and so on. But as a beginner, someone who’s not quite ready to drop half your annual salary on kit in one go, someone who doesn’t know whether paddling is something you’re doing to love or not, something for your feet and something for your glasses are a good start and a buoyancy aid will never be a bad decision.