Millions of Americans made the decision to retire during the pandemic, taking advantage of a booming stock market and the time to rethink how they wanted to spend their lives. But as concerns around the pandemic ease and stock market declines have investors and retirees dealing with lower 401(k) balances, many are looking to re-enter the workforce.

But does heading back to work mean giving up Social Security benefits? The answer depends on how old a person is and what their current benefits are.

Here are four key things “unretirees” should consider when it comes to their Social Security benefits before switching back to being a worker.

1. A portion of your Social Security income may be withheld

Age is the biggest determining factor for how Social Security benefits would be altered, should a retiree return to work. Depending on where someone falls in regards to “full retirement age,” benefits may be withheld or adjusted.

The earliest a person can start receiving Social Security retirement benefits is 62, but that doesn’t mean they are considered to be at full retirement age.

Full retirement age is 66 for people born between 1943 and 1954; those born from 1955 to 1959 have two months added for every birth year until the full retirement age reaches 67, which is the age for those born in 1960 or later, according to the Social Security Administration (SSA) website. (Here’s a chart of birth years and full retirement ages.)

Those younger than full retirement age for the entire year they return to work, while still receiving benefits, have $1 deducted for every $2 earned above the annual income limit. For 2022, the annual limit is $19,560.

For those working during the year they reach full retirement age (but haven’t reached it yet), the deductions are slightly less. The SSA will deduct $1 for every $3 earned above an earnings limit of $51,960. The administration adds that only earnings before the month of reaching full retirement age are counted toward the threshold.

Here’s the best time to start collecting Social Security, say experts. 

2. You might have to pay back any benefits you’ve received

“If you are under 70 years old and decide to come out of retirement within 12 months of applying for Social Security, you can withdraw your application. This requires submitting a form to the Social Security Administration,” says Leslie H. Tayne, Esq. founder and director of the Tayne Law Group, in New York. “You’ll also have to pay back any benefits you’ve already received, including anything that’s been withheld from checks. If you go this route, you’ll be able to reapply later.”

3. At full retirement age, you’re still eligible for full benefits

If you’re at full retirement age but choose to return to work, your benefits won’t be affected.

The SSA adds that the benefit amount will be recalculated to “leave out the months when [they] reduced or withheld benefits due to your excess earnings.”

4. Know the special rule for retiring and then unretiring mid-year

Those who choose to return to work mid-year have a special rule applied to their earnings for one year — usually the first year of their retirement.

The SSA gives the following example: If someone retires around June but creates their own business around October, they’ll still receive full benefits for the months they’re considered fully retired — regardless of their total earnings for that year.

The special rule will pay you a full benefits check for any whole month you’re considered retired, under the following circumstances, according to the SSA:

  • Be under full retirement age for all of 2022, you are considered retired in any month that your earnings are $1,630 or less and you did not perform substantial services in self-employment.
  • Reach full retirement age in 2022, you are considered retired in any month that your earnings are $4,330 or less and you did not perform substantial services in self-employment.

The administration defines “substantial services in self-employment” as working more than 45 hours a month to a business, or between 15 and 45 hours to a business in a “highly skilled occupation.”

Social Security’s annual earnings test

If you’re below normal retirement age, not currently working and receiving Social Security benefits, the earnings test can help you determine if your benefits will be withheld if you return to the workforce.

Here’s how the earnings test works: Social Security withholds benefits if your earnings exceed a certain level and if you’re below normal retirement age. One of two different exempt amounts apply — a lower amount in years before the year you reach normal retirement age and a higher amount in the year you reach it. In 2022, people who will reach normal retirement age after this year, the exempt amount is $19,560, while people who will reach that age in 2022, the exempt amount is $51,960. This higher amount applies only to earnings made in the months of the year before reaching normal retirement age.

The SSA adds that any benefits withheld while working aren’t “lost.” Monthly benefits will be increased to account for the time in which your benefits were withheld.

Experts advise workers to use the earnings test as a way to keep the SSA up-to-date on your earnings, and to help avoid any necessary repayments in the future.

“The key to avoiding an unexpected (and unwanted) letter demanding you repay previous benefits due to the earnings test is to provide Social Security with an estimate of how much you expect to earn each year before attaining full retirement age,” says Tim Adams, a certified public accountant and Social Security advisor. “If your estimate changes during the year, contact Social Security right away so they can re-adjust if necessary.”

Other things to consider before heading back to work

Social Security benefits aren’t the only financial aspect that are affected by a retiree choosing to go back to work. There are other things, like 401(k)s and taxes, that should be considered.

We asked experts for advice on how “unretirees” can navigate a number of other tricky financial scenarios:

How going back to work might affect Medicare coverage

Once someone turns age 65, they are automatically enrolled in Medicare Part A, which is usually free and covers hospital insurance.

At 65, people are also eligible for Part B (doctor and outpatient services) and D (prescription costs) if they are receiving Social Security benefits, and premiums are deducted from the benefits check.

If you have applied for Social Security benefits while receiving Part B coverage, withdrawing your application will have implications. If you keep the Part B coverage, you will be billed for future premiums — and failure to pay them on time will put your coverage at risk of removal.

Also keep in mind that individuals earning above $91,000 are charged more for Part B premiums than the standard $170.10 per month. The Medicare website details monthly payments over multiple income thresholds.

Think about your taxes

If you adjust your Social Security benefits, your taxes will be changed as well.

“Your income determines how much of your Social Security benefits are taxable,” Tayne says. “Additionally, drawing from your retirement savings as well as earning income from a full- or part-time job could affect which tax bracket you’re in, meaning you could owe more.”

Tayne adds that filing jointly affects tax situations, too. For those filing jointly with a combined income of over $44,000 or filing alone with an income over $34,000, 85 percent of Social Security benefits are taxable. For incomes below those figures, 50 percent of the benefits are taxable.

People who return to work after retiring might find it helpful to consult with a tax professional to try to avoid any penalties for inaccurate reporting or withholding come tax season.

Take advantage of 401(k)s

Someone heading back to their career field might want to consider taking a position that offers a 401(k) plan, says Timothy S. Bickmore CFP, director of financial planning and co-founder at LBW Wealth Management in the Madison, Wisconsin area. Doing so could increase future wealth.

“The law permits someone who is deemed to still be working to delay their required minimum distributions (RMD) inside a 401(k),” Bickmore says. “This means the individual could delay taking money out of their 401(k) and continue to defer growth into the future.”

Another strategy to stretch retirement dollars further would be to roll over any old 401(k) or IRAs into the new plan, Bickmore says. Rolling over into a single plan allows consumers to delay RMDs on that money as well, giving it the opportunity to grow as you keep working. Not all plans allow rollovers, so be sure to check with your provider.

Check in on your pension

Although pensions aren’t as common as they used to be, retirees who have them should keep in mind how returning to the workforce might impact them.

“It would be important to look at how this would affect their current or future pension benefit,” Bickmore says. “Every pension is different, be it a union or state pension, so it would be recommended to look specifically at the pension plan’s details.”

Bottom line

Going back to work after you’ve already retired can make sense for a lot of people. However, be sure you understand how that decision will impact your Social Security benefits. The answer will largely depend on your age, but heading back to work could result in benefits being withheld for a period of time, or having to repay benefits you’ve already received. It’s worth checking how you will be affected before you re-enter the workforce, so that there are no surprises along the way.

Note: Bankrate’s Brian Baker also contributed to an update of this story.