Apparently people don't shrink quite as much with age as they think they do:
Self-report overestimates true height loss: implications for diagnosis of osteoporosis
Abstract The Newcastle Thousand Families birth cohort dates from 1947; assessments have included height measurement at 22 and 50 years, when height loss was also assessed by self-report. A total of 388 attended for 50-year review of bone health, of whom 57 reported a median height loss of 2.5 cm, and 8 reported height loss of >3.5 cm. However, of 24 subjects for whom true height loss could be calculated, 7 had gained height, 9 were unchanged and only 8 had lost height since age 22 years. Self-report leads to over-reporting of height loss, and therefore should not be the sole measure of height loss. In clinical practice, objective confirmation of reported height loss should be undertaken, wherever possible, prior to further investigation.
At 180 lbs I'm basically the same weight as when I was a senior in high school. In case you are wondering, I do a very compressed paleo style fitness routine -- only 30 minutes per workout. When I was younger I would have considered one of my current workouts a "rest day" :-)
If you are pressed for time and a former athlete, I recommend Tabata training and sprints. (The guy in the video, Mark Sisson, is in his fifties! Love the footgear :-) I dream that I'll someday be able to get back on the mat and train BJJ/MMA, but who knows when that will happen... Sometimes, to get motivated, I watch videos like these.