Let's revisit the basic points of the argument:
The Sun is a young star. There are billions of stars in the galaxy that are billions of years older;
Some of these stars likely have Earth-like planets which, if the Earth is typical, may develop intelligent life;
Presumably some of these civilizations will develop interstellar travel, a technology Earth is investigating even now;
At any practical pace of interstellar travel, the galaxy can be completely colonized in a few tens of millions of years.
According to this line of thinking, the Earth should have already been colonized, or at least visited. But no convincing evidence of this exists. Furthermore, no confirmed signs of intelligence elsewhere have been spotted, either in our galaxy or the more than 80 billion other galaxies of the observable universe. Hence Fermi's question "Where is everybody?"
Let me know if you find an answer.