a conditional form of domain adaptation?

I have been discussing with Soledad Villar (JHU) projects related to domain adaptation (also with Thabo Samakhoana and Katherine Alsfelder), in which you have two different instruments (say) taking data and you want to find the transformation between the instruments such that the data are the same from the two sources. The modification we want (or need) to make is to create a conditional version of this: In SDSS-V, there are two observatories; they take very similar (but not identical) data. What are the transformations that make the data identical? The problem is: The two observatories also observe different stars on average (because they see different parts of the Galaxy). So we need to find the transformations that make the data identical, conditional on other data (like the ESA Gaia data) that we have for the stars. Great problem, and we came up with some non-elegant solutions. Are there also elegant solutions?