35 years is the magic number
The Social Security website offers an explanation of how your benefits are calculated, but it's a little hard to follow. You can find a simpler explanation at MyRetirementPaycheck.org, a website sponsored by the National Endowment for Financial Education.
Your Social Security payment is figured using a complex calculation based on a 35-year average of your covered wages. Each year's wages are adjusted for inflation before being averaged. If you worked longer than 35 years, the government will use the highest 35 years. If you worked fewer than 35 years, they'll average in zeros for the years you are lacking. You don't have to be a math genius to figure out the impact of that -- it drags down your average. If you can avoid zeros by working a couple of years longer, you'll increase your Social Security payment.