When starting a new business it's important to have a plan. Some people jump into business because they have a great product but don't really think of the business side of things. If you're not going to split your focus between your product or service it's important to bring someone in that you trust with business aspects.
Alot of people prefer to have their CPA take care of taxes and other loose ends when starting a business, and it's a wise move. If you're working by yourself, you want to stay in compliance, not only with the state, but the city and county as well.
Another way to get around having someone help you is to take a business class at a community college if you do not have any business experience, many colleges even offer weekend workshops focusing on business.
Once you've got all your basics out of the way, you should focus on marketing, any new business wants the world to know they exist, if you're starting a service or retail store, it's imporant to be found online, you should create a website or some kind of web presance, lots of small businesses only rely on websites like Yelp, which can be really helpful (or hurtful if your service isn't up to par). It's important to have a plan when it comes to your marketing strategies also, if you're in a metorpolitan area, it's not uncommon to see businesses handing out fliers or coupons to their store or restaurant, sure you might have to give a little to gain a client, but if your product or service is good, they'll come back.
There are lots of things to think about when starting a business, so organization is key, keep that in mind.