The other thing to remember is that IRAs are very portable. If you're not happy with your current provider, it's a simple process to move the money to a new provider. I recommend a direct or trustee-to-trustee transfer if you decide to move the accounts. But with IRAs, you also can get a check, and you then have 60 days to invest the money in the new self-directed IRA.
I'm curious about how you will measure your success in managing this account without the advice of your financial planner. There are a lot of things to consider in comparing the two portfolios' performances. For example, if your asset allocation is different than your planner's, you need to account and/or adjust for that difference in comparing the two portfolios.
If you're just comparing your self-directed IRA account performance to your other account's net of fees, the self-directed account does have the advantage of no adviser fees.
If your adviser is actively managing your IRA accounts, the expectation is that these accounts should earn a higher-yield net of fees than the account you're managing on your own. Still, that won't always be the case, so one bad quarter isn't a justification to move the accounts.
In addition, a financial planner often does more than just oversee your investments. The compensation model in place for your advisory relationship establishes how your financial planner is paid. If you're not writing a check and the adviser is paid on commission or with assets under management, then you'd expect your investment returns to be lower because of how the investment professional is being paid.