7 Self-improvement
Throughout this chapter, \((\psi ,A,B,L)\) is an \((\varepsilon ,\delta ,\gamma )\)-good symmetric strategy for the \((m,q,d)\) low individual degree test.
7.1 Non-projective Output
A large part of this chapter is devoted to proving Lemma 7.1, which is a slightly weaker form of the projective self-improvement theorem proved in Section 7.2. The key difference is that the output family \(H\) is only required to be a submeasurement, rather than a projective submeasurement. Once this non-projective statement is available, Theorem 4.4 upgrades \(H\) to a projective family without losing control of the four quantitative conclusions.
Compared with the projective theorem, the helper lemma records strong self-consistency using the explicit inequality from Definition 3.34, keeps the boundedness conclusion in terms of an auxiliary operator \(Z\), and produces a much smaller error parameter.
Let \(G \in \mathrm{PolyMeas}(m,q,d)\) be a measurement with the following property:
(Consistency with \(A\)): On average over \(u \sim \mathbb {F}_q^{m}\),
\[ A^{u}_a \otimes I \simeq _{\nu } I \otimes G_{[g(u)=a]}. \]
Let
Then there exists \(H \in \mathrm{PolySub}(m,q,d)\) with the following properties:
(Completeness): If \(H = \sum _h H_h\), then
\[ \langle \psi \rvert H \otimes I \lvert \psi \rangle \geq (1-\nu )-\zeta . \](Consistency with \(A\)): On average over \(u \sim \mathbb {F}_q^m\),
\[ A^u_a \otimes I \simeq _{\zeta } I \otimes H_{[h(u) = a]}. \](Strong self-consistency):
\[ \sum _{h} \langle \psi \rvert H_h \otimes H_h \lvert \psi \rangle \geq \langle \psi \rvert H \otimes I \lvert \psi \rangle - \zeta . \](Boundedness): There exists a positive-semidefinite matrix \(Z\) such that
\[ \langle \psi \rvert Z \otimes I \lvert \psi \rangle -\mathbb {E}_{u} \sum _a \langle \psi \rvert A^{u}_{a} \otimes H_{[h(u)=a]} \lvert \psi \rangle \leq \zeta \]and for each \(h \in \mathcal{P}(m,q,d)\),
\[ Z \geq \left(\mathbb {E}_{u} A^{u}_{h(u)}\right). \]
Let \(T=\{ T_g\} \) and \(Z\) be the optimal solutions to the SDPs 25 and 26 given by Lemma 7.8. Then \(T\) is a measurement, and
For each \(u \in \mathbb {F}_q^m\), define \(H^u=\{ H_h^u\} _{h \in \mathcal{P}(m,q,d)}\) by
and let
The pointwise family \(H^u\) is a submeasurement because
where the inequality uses that \(T\) is a measurement and the last identity uses that \(A\) is projective. Averaging over \(u\) gives
so \(H\) is also a submeasurement and therefore lies in \(\mathrm{PolySub}(m,q,d)\).
Set
Lemma 7.9 below is the technical transfer from the averaged family \(H\) back to the sandwiched operators \(A^u_{h(u)}T_hA^u_{h(u)}\). We use it in the completeness, consistency, strong self-consistency, and boundedness estimates that follow.
We now verify the four conclusions of Lemma 7.1.
Proof of 1. The completeness of \(H\) is
Grouping the polynomials according to their value at \(u\) gives
We first move the leftmost copy of \(A^u_a\) across the bipartition:
The difference is bounded by Cauchy–Schwarz as in 37; the first factor is at most \(\sqrt{2\delta }\) by self-consistency of \(A\), and the second is at most \(1\) because \(T_{[h(u)=a]} \le I\). We next remove the remaining copy of \(A^u_a\) on Bob’s side:
The first Cauchy–Schwarz factor is at most \(1\) because the fiber operators form a submeasurement after grouping \(T\) by the value of \(h(u)\). Here the second Cauchy–Schwarz factor is
which is at most \(\delta \) because \(A\) is self-consistent and projective. Reindexing by \(h\) and using 2,
Thus
Since \(G\) is a measurement, 1 and Lemma 3.18 give
Therefore the completeness is at least \(1-\nu -3\sqrt{\delta }\), which is bounded below by \((1-\nu )-\zeta \).
Proof of 2. The inconsistency of \(H\) with \(A\) is
The theorem-side off-diagonal selection used for this ‘add-in-u‘ application is , and the corresponding left/right scalar quantities are and . The selected scalar chain for this specialization is recorded by . Apply Lemma 7.9 with \(\mathcal O=\mathbb {F}_q\), \(M=A\), and \(S_u=\{ (a,h):h(u)\neq a\} \). Then
and the right-hand side is \(0\) because \(A\) is projective. Hence
Since
this proves 2.
Proof of 3. From the definition of \(H_h^u\) and the projectivity of \(A\),
The strong self-consistency expression is
Apply Lemma 7.9 with \(\mathcal O=\mathcal{P}(m,q,d)\), \(M=H\), and \(S_u=\{ (h,h):h\in \mathcal{P}(m,q,d)\} \). This gives
We next enlarge the sum from \(h'=h\) to all pairs \(h,h'\):
The added off-diagonal contribution is
where the second equality is 11. To estimate 15, first replace the left copy of \(A^u_{h(u)}\) by \(A^v_{h(v)}\):
The corresponding Cauchy–Schwarz bound is
The first factor is bounded by \(\sqrt{\zeta _{\mathrm{variance}}}\) via Lemma 6.6, and the second by \(1\) because \(T\) and \(H^u\) are submeasurements.
Repeating the same argument for the remaining copy of \(A\) yields
The same intermediate estimate gives
so Schwartz–Zippel bounds the indicator average by \(md/q\) and proves 14.
Using 11, the right-hand side of 14 is
Replace the remaining copy of \(A^u_{h(u)}\) by \(A^v_{h(v)}\):
Here the first Cauchy–Schwarz factor is at most \(1\) because \(T\) and \(H^u\) are submeasurements, and the second is again bounded by Lemma 6.6. Finally move this last copy of \(A^v_{h(v)}\) to Bob’s side:
Substituting 2, then 1, and finally the already proved bound 9, one obtains
Combining 13, 14, 23, and 24 gives
The Lean formalization records this step as named internal bounds, starting with the structure and the exact residual-side expansion and then uses the proved lemma together with to recover the final helper-stage self-consistency conclusion. The theorem-level derivation is therefore not moved into an additional hypothesis. The paper’s arithmetic shows that the right-hand error is at most
which proves 3.
Proof of 4. The helper-stage boundedness quantity is
The reindexing of the sum by \(h\) is the algebraic identity
whose averaged scalar form reads
Using \(\sum _a A_a^u=I\), the pointwise difference between the right-placed total \(I \otimes H.\mathrm{total} = \sum _h I \otimes H_h\) and the helper-agreement operator at \(u\) reindexes as the off-diagonal sum
whose averaged scalar form reads
Combined with 9 this gives
By 7, the right-hand side is at most \(3\sqrt{\delta }+4\sqrt{\zeta _{\mathrm{variance}}}\). Substituting the definition of \(\zeta _{\mathrm{variance}}\) and collecting terms gives
The formalization records a strengthened helper theorem whose output carries the complementary-slackness equations used in the completeness chain. The previous top-level residual-domination assembly theorems have been removed: the full self-improvement conclusion is now stated only by the paper theorem below. Its proof follows the paper’s expectation-level total-difference transport route; issue #1642 records why the sharper generic operator-total strengthening is not supplied by the present route.
7.1.1 A Semidefinite Program
The uniform primal family \(T_g=(2|\mathcal{P}(m,q,d)|)^{-1}I\) has total mass \((1/2)I\), and the dual operator \(Z=2I\) is positive semidefinite and dominates the identity.
This is the elementary Slater-type feasible witness recorded in the basic self-improvement definitions.
In every concrete matrix realization, the uniform family \(T_g=(2|\mathcal{P}(m,q,d)|)^{-1}I\) has total mass \((1/2)I\). The dual operator \(Z=2I\) dominates the identity and satisfies \(Z-A_g\geq I\), hence \(Z\geq A_g\), for every polynomial \(g\). In the canonical block SDP, the corresponding block matrix is feasible, its slack block is \((1/2)I\), and the canonical dual slack for the same dual witness also dominates the block identity. More generally, dual feasibility implies positivity of \(Z\), since the averaged point operators \(A_g\) are positive semidefinite.
The formal proof is the matrix specialization of the uniform SDP witness recorded in the basic definitions. The inequality \(A_g\leq I\) follows by averaging the point-measurement effects and using the submeasurement bound at each point.
From a matrix-level strong-duality witness with complementary slackness of the shape asserted by Lemma 7.8, one obtains a complete measurement \(\{ T_g\} \), a feasible dual operator \(Z\), equality of the primal and dual objectives, and the equations \(T_gZ=T_gA_g\) for all \(g\). Lean isolates the saturated canonical optimal-pair output required from strong duality: a feasible canonical primal matrix, a dual-feasible operator with equal objective value, canonical complementary slackness, and vanishing of the extra canonical slack block. This datum gives the matrix-level slackness-carrying SDP statement, whose witness is already a complete matrix measurement, and is then transported to the abstract Section 9 statement without assuming \(I \le Z\). The displayed measurement witness is read directly from that abstract statement. The canonical saturation datum is now the only matrix-level input used to pass from strong duality to the abstract SDP statement. The paper-facing self-improvement helper consumes the resulting abstract SDP statement.
The identity \(\sum _gT_g=I\) turns the optimal primal submeasurement into a measurement; Lean performs this completion before storing the matrix and abstract slackness statements. The zero-defect form \(T_g(Z-A_g)=0\) is equivalent, by distributivity, to the displayed equation \(T_gZ=T_gA_g\).
Earlier Lean versions exposed top-level conditional construction theorems that combined the matrix slackness output with residual-dominating orthonormalization and QXP-repair inputs. These declarations have been removed. The matrix slackness material is retained only up to the slackness-carrying helper output, and the full projective-output conclusion is represented by the paper theorem MIPStarRE.LDT.SelfImprovement.selfImprovement.
Earlier Lean versions also contained a reduced theorem sdp proving only the measurement-total and dual-feasibility fragment used before strong duality was supplied, as well as dominance-carrying declarations with the auxiliary condition \(I\le Z\). These declarations have been removed, and they are not advertised as formalizations of Lemma 7.8. The active Lean route for Lemma 7.8 is the slackness-carrying statement sdp_statement_with_slackness.
Let
Then the primal is
and the dual is
These semidefinite programs are dual to each other. Moreover there is an optimal pair of solutions \(\{ T_g\} \) to 25 and \(Z\) to 26 such that \(\sum _g T_g = I\) and
Rewrite 25 in canonical block form. Let \(r\) be the dimension of the space on which \(A\) acts, let \(M = |\mathcal{P}(m,q,d)|\), and fix an ordering \(g_1,\ldots ,g_M\) of the polynomials. Consider
where
and, for \(i,j \in \{ 1,\ldots ,r\} \),
Writing \(X=\sum _{i,j=1}^{M+1}\lvert i \rangle \langle j \rvert \otimes X_{ij}\), the constraints say precisely that \(\sum _{i=1}^{M+1} X_{ii}=I\), hence \(\sum _{i=1}^{M}X_{ii}\le I\), and the objective is
Thus 29 is equivalent to 25 under the identification \(T_{g_i}=X_{ii}\).
The canonical dual is
Since
the constraint 31 is exactly \(Z\ge A_{g_i}\) for every \(i\), and the objective is \(\operatorname{Tr}(Z)\). In Lean, the latter statement is recorded as the trace-pairing identity
for every feasible canonical primal matrix \(X\), where \(D(Z)\) denotes the block-diagonal canonical dual operator. Consequently the canonical primal-dual gap is the real trace pairing of \(X\) with the canonical dual slack \(D(Z)-C\). Since the real trace pairing of positive semidefinite operators is nonnegative, Lean also records the resulting weak-duality inequality for any feasible canonical primal and dual pair. Hence 30 is equivalent to 26.
Slater’s condition holds because
are strictly feasible for the primal and dual respectively. Therefore strong duality holds. For an optimal pair \((X,(z_{ij}))\) in canonical form, complementary slackness gives
We may assume that \(X\) is block diagonal. Translating 32 back to the variables \(\{ T_g\} \) and \(Z\) yields \(T_g(Z-A_g)=0\) for every \(g\) and \(S Z=0\), where \(S=I-\sum _gT_g\) is the slack block. The paper then treats this as forcing \(S=0\). The Lean development records this saturation as part of the canonical optimal-pair datum for the block SDP. The canonical calculation still uses the auxiliary expression matrixSdpComplementarySlacknessDefect to read off the polynomial blocks of 32. The matrix optimal-witness interface, however, stores the displayed equation \(T_gZ=T_gA_g\) directly; MatrixSdpOptimalWitness.primalMeasurement expresses \(\sum _g T_g=I\) as a complete measurement. The statement MatrixSdpStatementWithSlackness records this matrix-level strong-duality output and the abstract theorem sdp_statement_with_slackness specializes it to the paper’s self-improvement setting. The explicit Slater witnesses used above are isolated in Lemma 7.4 and separately recorded by matrixSdpStrictPrimalSubmeasurement, matrixSdpStrictDualWitness, and matrixSdpFeasibleBounds_canonical; the dual feasibility of \(2I\) follows from matrixAveragedPointOperator_le_one. The block-diagonal reduction is also recorded for arbitrary feasible canonical matrices: extracting the diagonal blocks preserves the objective and transfers canonical complementary slackness to the extracted paper primal submeasurement.
Let \(T=\{ T_h\} \) be an optimal solution of the primal SDP, let
and set
Suppose \(M = \{ M^u_o\} \) is a submeasurement with outcomes in some set \(\mathcal O\). For each \(u \in \mathbb {F}_q^m\), let \(S_u\) be a subset of \(\mathcal O \times \mathcal{P}(m,q,d)\). Then
The reduced variance-bound specialization keeps only the variance-bound consequence used later in the helper theorem; it is not itself the full formal counterpart of Lemma 7.9.
We begin by expanding
We claim that
To show this, we bound the magnitude of the difference:
The first factor is controlled by summing over the value \(a = h(v)\):
where the inequality uses that \(M^u\) and \(T\) are submeasurements. By Lemma 3.21 and the self-consistency of \(A\), this is at most \(2\delta \). The second factor is
which is at most \(1\) because \(M^u\) and \(H^v\) are submeasurements.
Next, we claim that
Again, we bound the magnitude of the difference:
The first factor is bounded by
using first that \(M\) is a submeasurement, then that \(A^v_{h(v)} \le I\), and finally that \(T\) is a submeasurement (\(\sum _g T_g \le I\)). The second factor is exactly the same expression as in 37, hence at most \(\sqrt{2\delta }\).
Having moved both copies of \(A\) to the left-hand side, we next show that
The difference is bounded by
The first factor can be rewritten as
because \(M^u\) is a submeasurement. Lemma 6.6 bounds this by \(\zeta _{\mathrm{variance}}\) since \(T \in \mathrm{PolySub}(m,q,d)\). The second factor is exactly the first factor from 39, so it is at most \(1\).
Finally, we show that
The same Cauchy–Schwarz estimate as in 41, with the final copy of \(A^v_{h(v)}\) replaced by \(A^u_{h(u)}\), gives the same bound \(\sqrt{\zeta _{\mathrm{variance}}}\). Indeed, the factor containing \(A^u_{h(u)} \cdot M^u_o \cdot A^u_{h(u)}\) is bounded by \(1\) by the same submeasurement argument as above, while the factor containing \((A^v_{h(v)} - A^u_{h(u)})^2\) is unchanged. Summing the four transports gives an error of \(2\sqrt{2\delta } + 2\sqrt{\zeta _{\mathrm{variance}}}\), and the lemma follows from \(2\delta \leq \zeta _{\mathrm{variance}}\).
7.2 Projective Output
This final step applies orthonormalization to the non-projective family produced by Lemma 7.1. The bookkeeping here is entirely about showing that completeness, consistency, self-consistency, and boundedness survive the passage from \(\widehat H\) to a projective submeasurement \(H\).
Several auxiliary results are used to prove Theorem 7.11 from the helper output and the orthonormalization theorem. They record local intermediate estimates and final-field assembly lemmas. The former orthonormalization spectral-transport record has been retired; the spectral-truncation part of the construction is now represented directly by spectralTruncationStatement_of_sourceAlmostProjective. This result is not a locality-preserving repair theorem. The theorem below records the paper-facing conclusion. Its proof follows the expectation-level completeness argument from the paper; the obstruction to a generic operator-total strengthening is recorded in issue #1642.
Let \(G \in \mathrm{PolyMeas}(m,q,d)\) be a measurement with the following properties:
(Consistency with \(A\)): On average over \(u \sim \mathbb {F}_q^{m}\),
\[ A^{u}_a \otimes I \simeq _{\nu } I \otimes G_{[g(u)=a]}. \]
Let
Then there exists a projective submeasurement \(H \in \mathrm{PolySub}(m,q,d)\) with the following properties:
(Completeness): If \(H = \sum _h H_h\), then
\[ \langle \psi \rvert H \otimes I \lvert \psi \rangle \geq (1-\nu )-\zeta . \](Consistency with \(A\)): On average over \(u \sim \mathbb {F}_q^m\),
\[ A^u_a \otimes I \simeq _{\zeta } I \otimes H_{[h(u) = a]}. \](Strong self-consistency):
\[ H_h \otimes I \approx _{\zeta } I \otimes H_h. \](Boundedness): There exists a positive-semidefinite matrix \(Z\) such that
\[ \langle \psi \rvert Z \otimes (I - H) \lvert \psi \rangle \leq \zeta \]and for each \(h \in \mathcal{P}(m,q,d)\),
\[ Z \geq \left(\mathbb {E}_{u} A^{u}_{h(u)}\right). \]
The bound is trivial if one of \(\varepsilon \), \(\delta \), or \(d/q\) is at least \(1\), so assume all three are at most \(1\). In Lean, the third assumption is recorded as \(d \le q\), which is equivalent to \(d/q \le 1\) since \(q{\gt}0\). Apply Lemma 7.1 to \(G\) and let \(\widehat H \in \mathrm{PolySub}(m,q,d)\) be the output, with
Therefore Theorem 4.4 supplies a projective submeasurement \(H \in \mathrm{PolySub}(m,q,d)\) such that
where
In addition, Lemma 3.39 implies that
where
The four conclusions are transferred one by one.
Proof of 1. Lemma 3.38 implies
where the second inequality is 1 of Lemma 7.1.
Proof of 2. Lemma 3.20 applied to 2 of Lemma 7.1 and the post-processed comparison 46 gives
In Lean the corresponding transport is stated for submeasurements with the total-overlap displacement recorded explicitly; this is the term which vanishes in the measurement-valued form of Lemma 3.20. The monotone-total route records a sufficient structural replacement: if the orthonormalization repair preserves the completed residual outcome, then the projective total is bounded by the helper total as an operator, hence also has no larger right-register expectation in the strategy state. Under this structural hypothesis the alphabet-size displacement term is not introduced. The QXP-layer interface used by the formalization records both the total domination of the canonical projective family and the sharper residual comparison for the fresh option-completion outcome. At present the formal QXP algebra only isolates the construction-level statement \(Q_{\mathrm{none}} \leq P_{\mathrm{none}}\), obtained when the repair preserves the fresh row block \(\widehat X_{\mathrm{none}}=X_{\mathrm{none}}\). The former generic “RestrictSome” monotone-total route would have required one to turn this into the source-facing inequality \(\widehat A_{\bot } \leq \widehat P_{\bot }\), by adding a comparison between the completed source residual and the “Q”-layer fresh outcome. The present formal development no longer uses that generic route, and the comparison is false without additional hypotheses; see docs/reports/issue-1642-restrictsome-residual-domination-obstruction.md. Since \(A^u_{\mathrm{tot}}=I\) and postprocessing does not change the total operator, Lean also records the equivalent form in which this displacement is the single scalar difference
between the right-register totals of \(H\) and \(\widehat H\). The outcomewise data-processing bound controls this scalar by applying the vector triangle inequality to the sum over \(a\in \mathbb {F}_q\); the formal estimate gives the additional term
Thus the total-overlap term is no longer an independent hypothesis, although the numerical absorption of this alphabet-size loss is recorded separately. There is also a sharper formal route: if the right-register total of the projective family has no larger expectation than that of \(\widehat H\), then the total-overlap term can only improve the consistency defect. Under this monotonicity hypothesis the transported error is again \(\widehat{\zeta }+\sqrt{\widehat{\zeta }_{\mathrm{dataprocess}}}\), and the standard final-stage numerical estimate absorbs it into \(\zeta \).
Proof of 3. Lemma 3.36 implies
Combining this with 45 on both sides and summing the three transports with Lemma 3.27 yields
Proof of 4. The boundedness quantity is
Proposition 3.24 upgrades 46 to the scalar estimate
and now 4 of Lemma 7.1 bounds this by \(\widehat{\zeta } + \sqrt{\widehat{\zeta }_{\mathrm{dataprocess}}}\). The formal boundedness field is the corresponding operator-monotonicity step: once the SDP dual witness satisfies \(I\le Z\), the total mass of any left-placed submeasurement, and in particular of the final projective submeasurement, is bounded by \(Z\otimes I\) with zero boundedness error.
Thus the four conclusions hold with errors
respectively. The paper’s exponent bookkeeping then shows the following bounds. The formal bookkeeping uses the three-term power sum \(\varepsilon ^p + \delta ^p + (d/q)^p\), its monotonicity on the unit interval, and the corresponding square-root estimate. The displayed Lean theorems record these arithmetic identities together with the completeness, self-closeness, and projective-residual absorptions \(2 \widehat{\zeta } + 2\sqrt{\widehat{\zeta }_{\mathrm{ortho}}} \le \zeta \), \(6 \widehat{\zeta } + 6\widehat{\zeta }_{\mathrm{ortho}} \le \zeta \) and \(\widehat{\zeta } + \sqrt{\widehat{\zeta }_{\mathrm{dataprocess}}} \le \zeta \). The two threshold lemmas involving \(|\mathbb F_q|\) record the scalar part of the alphabet-size obstruction in the current final-fields point-consistency transport: the data-processing threshold contains \(8\widehat{\zeta }\), but the transported estimate still carries the factor \(\sqrt{|\mathbb F_q|}\).
and hence
Since \(2\widehat{\zeta } + 2\sqrt{\widehat{\zeta }_{\mathrm{ortho}}} \le \widehat{\zeta }_{\mathrm{dataprocess}} \le \zeta \), all four bounds are at most \(\zeta \).