A Possible Blind Spot in difFOCI’s Differentiable Relaxation

difFOCI is a method that makes the rank-based dependence measure underlying FOCI differentiable, so that it can be used as an objective for representation learning [1, 2]. This measure is closely related to Chatterjee’s rank correlation [3].

I have been thinking about one possible limitation of the way difFOCI performs this differentiable relaxation.

Averaging Ranks Before Measuring Dependence

In difFOCI, the ranks of neighboring samples are aggregated using a weighted average. Roughly speaking, it constructs

$$\bar R_i=\sum_j W_{ij}R_j,$$

and then uses this averaged rank inside the objective.

This is useful because replacing a hard nearest-neighbor selection with a smooth weighted average gives us smoother gradients and makes gradient-based optimization possible.

However, there is a small issue that seems worth thinking about.

In general,

$$\min\left(R_i,\sum_j W_{ij}R_j\right)$$

is not the same as

$$\sum_j W_{ij}\min(R_i,R_j).$$

In other words, averaging the neighboring ranks first and then evaluating their relationship with $R_i$ is not equivalent to evaluating the relationship with each neighbor first and then averaging.

This raises the possibility that averaging the ranks may discard some information about the conditional distribution.

A Case Where the Conditional Mean Does Not Change

A particularly interesting case is when

$$E[U\mid X]=\frac12,$$

while at the same time

$$U\not\!\perp X.$$

Here, the conditional mean rank does not change with $X$, even though the conditional distribution itself does.

A simple example of this kind of behavior is heteroscedasticity. Consider

$$Y=Z\varepsilon,$$

where $Z$ and $\varepsilon$ are independent and $E[\varepsilon]=0$.

Then

$$E[Y\mid Z]=0,$$

but

$$\operatorname{Var}(Y\mid Z) = Z^2\operatorname{Var}(\varepsilon).$$

So the conditional mean remains unchanged, while the spread of the distribution clearly depends on $Z$.

Thus, $Z$ and $Y$ are dependent even though their relationship is invisible at the level of the conditional mean.

Could the Relaxation Lose This Information?

Something similar may happen when difFOCI averages neighboring ranks.

If the neighborhood weights are sufficiently spread out, dependencies satisfying

$$E[U\mid X]=\text{constant}$$

may become difficult for the relaxed objective to distinguish, even though

$$P(U\mid X)$$

still changes with $X$.

This does not mean that difFOCI itself is incorrect.

The original difFOCI construction is designed so that, as the soft neighborhood becomes increasingly sharp, it approaches the hard nearest-neighbor behavior used by FOCI [1].

The concern is instead about what happens away from that limiting regime, when multiple neighbors contribute substantially to the weighted average.

So perhaps there is a trade-off of the following form:

$$\boxed{ \text{smoother gradients through neighborhood averaging} \quad\Longleftrightarrow\quad \text{possible loss of some distributional information} }$$

This makes me wonder whether averaging the ranks themselves is always the best way to obtain a differentiable relaxation.

Perhaps a differentiable dependence objective should preserve more than just a conditional average rank. For example, it might be useful to retain multiple features of the conditional distribution rather than compressing the neighborhood into a single quantity such as

$$E[U\mid X].$$

At this point, I am not claiming that this is a general failure mode of difFOCI. It is more of a possible structural limitation that seems worth investigating.

The broader question is:

When we make a dependence measure differentiable, how much of the dependence information in the original measure survives the relaxation?

I think this question may be just as important as whether the resulting objective has useful gradients.

Thank you for reading.

References

1. K. L. Pavasovic, G. Biroli, and L. Sagun.
   “A Differentiable Rank-Based Objective for Better Feature Learning.”
   International Conference on Learning Representations (ICLR), 2025.

2. M. Azadkia and S. Chatterjee.
   “A Simple Measure of Conditional Dependence.”
   The Annals of Statistics, 49(6):3070–3102, 2021.
   DOI: 10.1214/21-AOS2073

3. S. Chatterjee.
   “A New Coefficient of Correlation.”
   Journal of the American Statistical Association, 116(536):2009–2022, 2021.
   DOI: 10.1080/01621459.2020.1758115

Related Work

A separate but related line of work has pointed out that Chatterjee’s original rank correlation can have low power against certain alternatives and that incorporating multiple nearest neighbors can improve its statistical efficiency [4].

This is not the same issue discussed above. The concern here is specifically about possible information loss caused by the differentiable averaging operation.

4. Z. Lin and F. Han.
   “On Boosting the Power of Chatterjee’s Rank Correlation.”
   Biometrika, 110(2):283–299, 2023.
   DOI: 10.1093/biomet/asac048