"The recent flattening out in housing activity could prove more protracted than currently expected," Yellen said in prepared remarks to the Joint Economic Committee. She said uncertainty in the housing market bears "close observation."
The housing sector had been recovering in recent years, but that recovery has slowed, with new-home sales falling 14.5 percent in March. While rough weather in the early part of this year played a factor in the slowdown, experts say the sluggishness appears to be more than just cold-weather hibernation.
The statement released by the Federal Open Market Committee following its meeting last week had briefly noted the slowing housing market, but Yellen's speech Wednesday highlighted the housing sector as a concern.
Much of the rest of Yellen's testimony was cautiously upbeat and consistent with previous outlooks and guidance.
"With the harsh winter behind us, many recent indicators suggest that a rebound in spending and production is already underway, putting the overall economy on track for solid growth in the current quarter," she said.
She added that she expects that "economic activity will expand at a somewhat faster pace this year than it did last year."
She also brushed away concerns that the Fed was fueling a bubble, saying that while the "committee recognizes that an extended period of low interest rates has the potential to induce investors to 'reach for yield' by taking on increased leverage, duration risk or credit risk," she didn't see anything too worrisome out there at this point.
Yellen is expected to testify again Thursday, this time in front of the Senate Budget Committee.
For more on what the Fed has been up to, check out Bankrate's page of all things Fed.
Follow me on twitter: @allisonsross.