[jIM_Ohio]

Here is how I am thinking about this. You have 9-13 years before child #1 hit college.

Can you do any of the following
1) pay off mortgage sooner
2) sell house and not lose money
3) have wife go back to work once 5 year old starts school. Even 5k-10k per year helps

I see your need as planning for kids education- that is the question you are asking indirectly.

I think your retirement plan is sound
I think your budget (outside of home price) is sound

My initial reaction is do not chase bad money after bad money. Just because things are tight budget wise does not mean you should (or should not) move.

If you can downsize without losing money, do it and that will work.
If you keep house and it holds it's value long term over 10-15 years, you will have a HUGE retirement benefit come your way at the time you need the most capital.
If wife's earnings if she returned to work could do the following:
$3,600/year to a general fund for kids education (in 9 years this would be $53,000)
$24,000/year to pay off mortgage before retirement (in 9 years this would be $355,000)

It's possible you keep all $27,000 (each year) I am suggesting in an investment which grows faster than 5.675%, you will come out way ahead. I am suggesting second income sent around $270,000 into an investment which is centered around kids education and financial independance). If you need a suggestion or two, look at PRPFX or RPSIX as a conservative mutual fund for this purpose. Meaning in 9 years you have a mutual fund with $400,000 in it and a mortgage balance of around $150k. You could cash out mutual fund some to pay off the $150k, and RPSIX would be shooting off 4% dividend payments of close to $10,000 per year which could subsidize some education payments too (and still leave $250,000 in account for emergencies, other kids education 3-7 years later, retirement or other). $10,000 in interest each year is quite a bit, IMO.

The goal I am suggesting is to pay off mortgage around time first kid starts college, then use mortgage payment to allocate to paying a portion of kids education.

Around time youngest starts college you could sell house and retire.

The situation is NOT bad. Yo are correct that you could do better (by downsizing). I think this only makes sense if you do not lose money when selling the house now. The longer you wait (to sell), I think the better off you will do overall (financially) if you cannot sell the house for what you owe.

Do you owe more than the house would sell for?