[Mr. Jan]

" to have built" indicates the past tense
We could have said: The Ancient Egyptians are said to build the pyramids when the sun sets
However we are talking about a past tense