It's a few too many. I assume it's where "he" spent his childhood, not where his father spent his childhood? The father is the most recently mentioned male person, so "he" in "where he spent his childhood" could refer to the father.
Try this: It's in the small house where he spent his childhood, the house built of stone by his father.
If you have a question about the English language and would like to ask one of our many English teachers and language experts, please click the button below to let us know: