Dude, I completely understand! My last employer did one here in Texas (about 60 miles southwest of Houston). I was there for almost the entire thing (took off 1 day out of the 4 days we were there). It was quite the maddening experience. Nothing like having your legs used as a sawhorse for a guy putting in the hardwood floors (I was installing a lavatory at the time).
For the most part, the project was extremely well thought out and planned. Each trade had at least 2 people in a supervisory position to handle issues as they arose (me being one of them). All supervisors and project managers were in contact via Nextels so getting a problem handled was done quickly. We prefabricated the piping for the ground in manageable sections (houses here are primarily built on concrete slabs) and had the whole thing installed in 45 minutes and on test! The make-up crews were right behind us and got the foundation ready for pour about an hour after us. Concrete was poured overnight. Framing began bright and early the next morning and as soon as the second floor was started, we began the rough-in. By the time we were done with the first floor, the house was essentially completely framed and we finished shortly after that. Then a cold front came in (with the attendant cold rain!) That day was miserable, what I call a NIRBY day (nasty, ichy, rotten, blechy, yucky). But we got through it all.
I have to admit though, from a standpoint of just being able to say I've been there, done that, it was pretty cool. And when they had the reveal for the family, wow. One of the most touching moments I've ever witnessed. And it made me feel proud to know I was a major part of it, even if the actual show hardly "showed" anything of the actual construction of the house (I think they did a 5 second time-lapse of it).
And yes, we did have to go back a couple of times but mainly for things that were beyond our control during the construction (it had a black Kohler toilet that arrived broken and took a couple weeks to get replaced), and I believe there was a leak at a kitchen sink, but other than that I think we did a pretty good job.
Would I do it again? I don't know, probably. I'm too good of a person to say no.