how to build an astrophysics program?

In group meeting today John Forbes (Flatiron) asked an interesting question: How do you build a good astrophysics program at a small place? He's thinking about this because he is on the job market. My own answer is a bit weird: It is to work with people you trust, since when a place is small, and resources are shared, trust is paramount. But this conversation was interesting, and it was also an illustration of an amusing fact, which is that many of the most interesting discussion topics at the Astronomical Data Group meeting at Flatiron are often not about astronomical data analysis!