Re: [ccp4bb] Does NCS bias a randomly-chosen test set (even if not enforced)?

- Protein crystallography

Main steps:

- Protein purification
- Crystallisation

Special:

- Programs for crystallography
- X-ray detectors

Basic tutorials:

   - Chemistry
   - Protein
   - Peptide
   - Amino Acids

Xtal community:

- CCP4BB

CCP4bb navigation

CCP4bb <-- 1999 <-- November 1999 <-- 30 November 1999

Previous message:
Subject: Re: shipping heavy-atom solutions using Fedex
From: Chavas Leo ccp4hnaalas {- at -} GMAIL {- dot -} COM
Date: 2008-02-11

Next message:
Subject: 5th Annual SER-CAT symposium - March 7th - Charleston, SC
From: Chris Davies davies {- at -} MUSC {- dot -} EDU
Date: 2008-02-11

Subject: Re: Does NCS bias a randomly-chosen test set (even if not enforced)?
From: Edward Berry eaberry {- at -} LBL {- dot -} GOV
Date: 2008-02-11

Dirk Kostrewa wrote:
> Dear Ed,
>
> although, I don't think that a comparison of refinement in a higher and
> a lower symmetry space group is valid for general NCS cases, I will try
> to answer your question. Here are my thoughts for two different cases:
>
> (1) You have data to atomic resolution with high I/sigma and low Rsym (I
> assume high redundancy). The n copies of the asymmetric unit in the unit
> cell are really identical and obey the higher symmetry (so, not a
> protein crystal). When you process the data in lower symmetry (say, P1),
> the non-averaged "higher-symmetry"-equivalent Fobs will differ due to
> measurement errors, and thus reflections in the working-set will differ
> to "higher-symmetry"-related reflections in the test-set due to these
> measurement errors. If you then refine the n copies against the
> working-set in the lower P1 symmetry, you minimize
Fobs(work)-Fcalc
,
> resulting in Fcalcs that become closer to the working-set Fobs. As a
> consequence, the Fcalcs will thus diverge somewhat from the test-set
> Fobs. However, since this atomic model is assumed to be very well
> defined obeying the higher symmetry, and, furthermore, the working-set
> contains well measured "higher-symmetry"-equivalent Fobs, the resulting
> atomic positions, and thus the Fcalcs, will be very close to their
> equivalent values in the higher-symmetry refinement. Therefore, the
> Fcalcs will also be still very similar to the
> "higher-symmetry"-equivalent Fobs in the test-set, and I would expect a
> difference between Rwork and Rfree ranging from "0" to the value of
> Rsym. In other words, the Fobs in the test-set are not really
> independent of the reflections in the working-set, and thus Rfree is
> heavily biased towards Rwork.
> In this case, I would not expect large differences in the outcome due to
> the additional application of "NCS"-constraints/restraints.

As I see it, this is clearly a case of
Fo-Fc
for the test reflectins
decreasing because the model is getting better, and there is no bias.
Lets say the higher symmetry really does apply, so the correct structure
is perfectly symmetrical and the "NCS-related" reflections agree to within
the error level.
Lets also say the initial model is perfectly symmetrical (you solved the
molecular replacement with two copies of the same monomer, and rigid-
body refinement positioned them exactly). But let's say it is completely
unrefined- the search model is from a different organism in a different
space group, and modified by homology modeling to your sequence.
So the Fo obey the NCS within error, The Fc obey the NCS, but the
Fobs don't fit the Fcalc very well. Initially there is no Free-R bias,
because the model has not been refined agaist the data. The free set
can only be biased by refinement, since it is only during refinement
that the the free set is treated differently. Thus it doesn't matter
that the ncs-related Fo are correlated and the ncs-related Fc
are correlated: it is only the CHANGES in Fc that could introduce
model bias, and they are uncorrelated if you do not enforce ncs.

Now as we refine, the model will converge toward the correct symmetrical
model as a result of minimizing the
Fo-Fc
for the work reflections.
At the same time the
Fo-Fc
for the test reflections will also decrease
on the average, but to a lesser extent. I argue that the only mechanism
for refinement to reduce
Fo-Fc
at a test reflection is by improving
the structure, and I think that constitutes an unbiased Free-R value.

If you can think of any mechanism to reduce
Fo-Fc
for a test reflection
because you are refining against a symm-related work reflection, then
the R-free would be biased. This is not the case if you do not enforce
symmetry. On the average no decrease in
Fo-Fc
(test) will result from
changes that reduce
Fo-Fc
for the work reflection: given an arbitrary
change in the structure, the change in
Fc
at arbitrary reflections
is a pseudo-random variable with expected value zero, and there is no
correlation between the change at ncs-related reflections.

The value of
Fo-Fc
at a test reflection goes down, not due to
changes which improve the fit at a sym-related working reflection,
but because of changes that improve the fit at all test reflections,
and then only because the structure is improving. The atoms moved into
symmetrical positions not because they were constrained to do so,
but because that fits the data better, in turn because the true structure
is symmetrical. If the symmetry doesn't hold for some atoms, they will
tend to move into asymmetric positions to minimize
Fo-Fc
at work
reflections, now *decreasing* the correlation with sym-related work
reflections. But again this will tend to reduce
Fo-Fc
at free
reflections, simply because the model is better approximating the
true structure.

To make a more obvious parallel, suppose you are refining a low-resolution
dataset from a microcrystal (with no NCS). In another directory on the
same disk you have a high resolution structure refined against a larger
but isomorphous crystal from the same well, same cryo treatment,
using a different or no free set. The Fo's will be highly correlated
between the two dataets, because they are isomorphous crystals
of the same protein.

Now if you constrain your low resolution model to be close to
the high resolution one, your free set will be biased because
those reflections were used in refining the other structure,
and you are constraining the new structure to be the same.

If you DON'T impose any restraints between the two models, the
new model will STILL tend toward the high-resolution structure,
because it is a good approximation of the true structure.
Hence the Fc's will become highly correlated to the Fc's of
that structure. And
Fo-Fc
of the test reflections will decrease,
not because the structural changes you are making improved the fit
of the high-resolution structure to the reflection in that dataset
which is a test reflection in the new dataset, but only because
the model is improving.
Using your logic, because the model (and hence Fc's) are approaching
those of the structure which was refined against the test reflections,
so the test reflections must be biased.

Thanks for taking the time to help me work this out,
Ed

>
> (2) You have data to non-atomic lower resolution, weak I/sigma and poor
> Rsym. It is impossible to say whether the n copies of the asymmetric
> unit in the unit cell are really identical, but they are treated so
> assuming the higher symmetry (so, a real protein crystal). For data
> processing, the same holds true as for case (1). In contrast, here I
> think that it makes a difference, whether
> you apply "NCS"-constraints/restraints between the n copies in the lower
> symmetry P1, or not. If you apply "NCS"-constraints or strong
> "NCS"-restraints, the n copies are made equal and you get n times the
> average structure. This is similar to the refinement in the higher
> symmetry, except that again you minimize the discrepancy between Fcalcs
> and working-set Fobs, which will increase the discrepancy to the
> "higher-symmetry"-related Fobs in the test-set. But since the Fobs in
> the test-set are still not really independent to the Fobs in the
> working-set, I would again expect maximum differences between Rwork and
> Rfree in the same order of magnitude as Rsym. So, Rfree is still biased
> towards Rwork, but it might be more difficult to notice this. But if you
> do not apply "NCS"-constraints/restraints, you give the less
> well-defined atomic model more freedom to converge against the
> working-set Fobs, resulting in a higher discrepancy between Rwork and
> Rfree. But since the Fobs in the working set still contain
> "higher-symmetry"-equivalent Fobs, you will end up with a model that
> still shows some similarity to the refined structure in the higher
> symmetry. As a result, the Rfree is even then not really independent of
> Rwork, but it might be even more difficult to notice this, depending on
> data resolution and quality. Here, I can't give a range of differences
> between Rwork and Rfree.
>
> So, this is still not quantitative, and I hope that I'm not completely
> wrong with my argumentation.
>
> These lower vs. higher symmetry examples given above are only
> transferable to reality in special NCS-cases with pseudo-higher symmetry
> (what Dale Tronrud discussed). Taking these special cases aside, what do
> the NCS experts say to my original statement that precautions against
> NCS bias in Rfree must only be taken if NCS-constraints/restraints are
> really applied during refinement?
>
> Best regards,
>
> Dirk.

CCP4bb navigation

CCP4bb <-- 1999 <-- November 1999 <-- 30 November 1999

Previous message:
Subject: Re: shipping heavy-atom solutions using Fedex
From: Chavas Leo ccp4hnaalas {- at -} GMAIL {- dot -} COM
Date: 2008-02-11

Next message:
Subject: 5th Annual SER-CAT symposium - March 7th - Charleston, SC
From: Chris Davies davies {- at -} MUSC {- dot -} EDU
Date: 2008-02-11