Maintaining 15 Tb of astronomical data takes a significant amount of human effort! If I were clever, I would have perl scripts take care of all contingencies. Blanton has written some great stuff that keeps our directory structure (built out of many cross-mounted partitions on many large servers) all consistent and unified even when machines go down or data are moved from one to another. But we still hit problems relating to the finite size of our directories and partitions (and the human-factors value in having related data close in hardware). I spent a good fraction of my research time this weekend cleaning up.
No comments:
Post a Comment