Everyone envisions their golden years as, well, golden – plenty of time to do what you want and the money to do it, too.

But when it comes to saving for retirement, working Americans are having a surprisingly tough time, according to a new national poll from Bankrate. More than half of American workers (52 percent) reported that they were behind their own savings goals for the future golden years.

A mere 16 percent said that they were right where they needed to be, and only 11 percent said they were ahead of where they ought to be. Meanwhile, 20 percent responded that they didn’t know where their retirement plan stood.

Despite an August 2019 Bankrate study showing that Americans have been increasing their retirement contributions throughout the economic recovery, the total balance of those savings remains well behind where they need to be, according to data from the latest survey.

The survey also revealed that having a retirement account may be less popular than many would expect. About 38 percent of all respondents indicated they had never had a retirement account, led by a whopping 43 percent of millennials (of respondents age 22 and over).

Who’s furthest behind on retirement savings?

Annual income appears to play a role in the ability of workers to save for retirement. The tendency to be behind on savings goals is highest among households with income between $30,000 and $49,999, with 62 percent saying that they are behind.

Meanwhile, 52 percent of households earning less than $30,000 say they’re behind, and 57 percent of those earning between $50,000 and $79,900 said they were lagging.

And the numbers don’t fall off that substantially at higher incomes. About 48 percent of households making $80,000 or more reported being short of their goals for retirement savings. So, significant numbers of workers at various levels of income say they haven’t saved enough.

However, households earning more than $80,000 were the most likely to report that they had saved enough or were ahead of where they should be — with a total of 45 percent providing these responses. This group was also the least likely to respond “don’t know” to the question of where their savings stood, with just 8 percent of interviewees responding this way.

Not surprisingly, lower-earning households (below $30,000) were the most likely to report that they had never had a retirement account, with 58 percent of respondents saying they hadn’t taken any action.

It’s not clear that age had a significant impact on whether workers were right on track with where they should be, with the major generations answering similarly: millennials (age 23-38) came in at 16 percent; Gen X (age 39-54) at 14 percent; boomers (age 55-73) at 18 percent; and the Silent Generation (age 74+) at 12 percent.

Perhaps surprisingly, the group most likely to say it was ahead of where it should be was millennials. Nearly 16 percent of millennials said they were ahead of plan, compared to 8 percent of Gen X and 9 percent of boomers.

Gen X was also the most likely to report that it was behind where it should be, at 63 percent of respondents. In contrast, millennials came in at 48 percent and boomers at 54 percent.

Having a specific savings goal was correlated with workers being ahead of where they should be. Of those who said they were ahead of plan, about 71 percent had a specific amount of savings that they were striving to accumulate. In contrast, 45 percent of those who were right on track said they had a savings goal, while only 26 percent of those who were behind had a goal.

Other culprits for low retirement savings

Americans have a hard time staying away from making early withdrawals on their retirement accounts. Of those with retirement accounts, nearly half (49 percent) have withdrawn money before they hit retirement age.

Although it may vary from household to household, many say they tap those funds for urgent needs. The most common reasons for accessing retirement savings early included: unemployment, medical bills or other unplanned expenses, and debt repayment. Other reasons cited include home purchases, home improvement or repair, and higher education expenses.

The tendency to withdraw funds early fell with higher annual income. Almost 60 percent of households with income below $30,000 said that they had taken an early withdrawal. More than 51 percent of households earning $30,000 to $49,999 had done so, compared with 50 percent of those earning $50,000 to $79,999. But only 44 percent of those workers making more than $80,000 said that they’d tapped their retirement funds.

The number of workers taking early withdrawals was relatively closely grouped, with the Silent Generation at 43 percent, millennials at 46 percent, Gen X at 54 percent and boomers at 51 percent.

Among those who are behind where they ought to be, nearly 61 percent reported taking an early withdrawal, while the figure was 18 percent for those who said they were ahead.

Of those who had taken an early withdrawal, only 31 percent said they were right on track, compared with 52 percent who said they were behind. In contrast, about 69 percent of those who had not taken an early withdrawal said their retirement savings were right on track.

“Early withdrawals from your retirement account deal a permanent setback to your retirement planning,” says Greg McBride, CFA, Bankrate chief financial analyst.

“In addition to the taxes and 10 percent early withdrawal penalty you may be subjected to, the money comes out but doesn’t go back in,” says McBride. “You don’t get to make higher contributions in subsequent years to make up for what you’d taken out early.”

How can savers get back on track?

“Getting your retirement savings on track begins by fully utilizing your tax-advantaged retirement savings options such as a workplace 401(k) and supplementing that with an IRA,” says McBride.

Take full advantage of any matching funds that your employer offers on a 401(k) plan. Workers should contribute enough to their own plan to get the maximum match offered by their employer. That’s an easy, risk-free return if your employer offers it, though not all do.

“Aim to save at least 10 percent – and ideally 15 percent – of your income specifically for retirement,” says McBride. “The best time to start is ‘today,’ and the worst time to start is ‘some day.’”

If they have their money in a tax-advantaged retirement account, savers need to make every effort to avoid withdrawing their money early. Not only will they have to pay a tax on any earnings in the account, but they may also be socked with a 10 percent penalty. That’s a real hit to their savings, not just now but over time, when the money could otherwise be compounding.

Of course, some expenses – medical bills or the costs of being unemployed, for example – may not be able to be postponed. But many other expenses can be shifted until later, allowing savers to continue to pile up retirement funds in tax-advantaged accounts.

If your employer doesn’t have a retirement plan, you can still contribute to an IRA, as long as you have earned income. An IRA offers many of the same benefits, such as tax-deferred growth on earnings and a tax break on contributions, as an employer’s plan.

Only after you’ve taken advantage of a 401(k) and IRA should you start looking at other taxable avenues to grow your savings, such as high-yield CDs and savings accounts.

Savers looking to get back on track should also consider investing in higher-return assets such as stocks, especially if they have a long time before they retire. Savers can purchase a basic diversified index fund without specialized knowledge, and the long-term returns on indexes such as the Standard & Poor’s 500 have been truly outstanding, averaging about 10 percent annually.

Methodology

This study was conducted for Bankrate via online interview by YouGov. Interviews were conducted from October 23-25, 2019 among a sample of 2,697 adults. Data are weighted and are intended to be representative of all U.S. adults, and therefore are subject to statistical errors typically associated with sample-based information.