Permutation Strategy In Two-Sample Tests: A Deep Dive

This article addresses a question regarding the permutation strategy used in the two-sample discriminability test, specifically within the context of the discrim_two_sample.py implementation. It explores the rationale behind the current implementation's approach to convex combinations compared to the method described in the original research paper by Bridgeford et al. (2021). Let's delve into the details and understand the underlying reasons for these choices.

Understanding the Question

The core question revolves around an observed discrepancy between the permutation strategy implemented in the discrim_two_sample.py code and the permutation method detailed in the Bridgeford et al. (2021) paper. The current implementation performs convex combinations within each input matrix (x1 and x2) separately. This is evident from the following code snippet:

permx1 = self._get_convex_comb(self.x1, random_state)  # convex combination within x1
permx2 = self._get_convex_comb(self.x2, random_state)  # convex combination within x2

However, the paper suggests creating "randomly combined datasets" through "random convex combinations of the observed data from each of the two methods choices." This description implies mixing data between the two input matrices, a different approach than the current implementation.

The main point of inquiry is to understand why the implementation deviates from the paper's description. What are the theoretical or practical reasons that justify performing convex combinations within each matrix instead of mixing them between the two?

Exploring the Rationale Behind the Implementation

To fully grasp the rationale behind the current implementation, it's essential to consider the goals of the two-sample discriminability test and how permutation strategies contribute to achieving those goals. The two-sample test aims to determine whether two datasets (samples) originate from the same underlying distribution. In other words, it assesses whether there is a statistically significant difference between the two groups.

Permutation tests are a non-parametric approach to hypothesis testing. They work by creating a null distribution of the test statistic under the assumption that the two samples are drawn from the same population. This is achieved by repeatedly shuffling or permuting the data and recalculating the test statistic for each permutation. The p-value is then calculated as the proportion of permutations that result in a test statistic as extreme or more extreme than the observed test statistic.

Why Convex Combinations? Convex combinations are used to create variations of the original data while preserving some of its underlying structure. This is particularly useful when dealing with complex data where simple shuffling might disrupt important relationships. By creating convex combinations, we generate new data points that are still representative of the original samples but introduce some degree of randomness. The use of convex combinations aims to generate null samples that are “close” to the original samples under the null hypothesis.

Rationale for Within-Sample Permutations: The key insight lies in understanding what the null hypothesis implies in this specific context. If the two samples come from the same distribution, then any combination of points within each sample should also resemble a sample from that same distribution. The current implementation might be designed to specifically test this null hypothesis by creating variations within each sample. By performing convex combinations within each matrix, the implementation is essentially asking: "If these two samples are truly from the same distribution, would we still be able to discriminate between them after creating these slight variations within each sample?"

By keeping the convex combinations within each sample, we maintain the inherent structure and relationships within that sample. If, even after these internal permutations, the discriminator can still distinguish between the two groups, it provides strong evidence against the null hypothesis.

Contrasting with Between-Sample Permutations

Now, let's consider the alternative: performing convex combinations between the two input matrices, as suggested by the paper's description. This approach would involve mixing data points from x1 and x2 to create new, combined samples. While this might seem like a more direct way to create "randomly combined datasets," it could also have unintended consequences.

Potential Issues with Between-Sample Permutations: Mixing data between the two samples could potentially obscure genuine differences between them. If the two samples are indeed different, then creating convex combinations between them might dilute the signal and make it harder to detect the difference. Furthermore, this approach might not be as effective at preserving the underlying structure within each original sample. By mixing data, we could be introducing artificial relationships that weren't present in the original datasets.

When Between-Sample Permutations Might Be Appropriate: Between-sample permutations are appropriate in many settings. In the context of this test, it might be appropriate if the goal is to test a slightly different null hypothesis. For example, if the goal is to test whether the overall distribution of the combined data is the same, regardless of which sample a data point originally came from. However, it seems the implementation and the paper are focusing on a slightly different (and perhaps more nuanced) null hypothesis related to the discriminability between the samples even when internal variations are introduced.

Possible Reasons for the Discrepancy

Several factors could explain the discrepancy between the paper's description and the implementation:

• Different Interpretations: It's possible that the paper's description is open to interpretation. The phrase "randomly combined datasets" could be interpreted in different ways, and the authors of the implementation might have chosen a specific interpretation that aligns with their theoretical framework.
• Specific Theoretical Considerations: The implementation might be based on specific theoretical considerations that are not explicitly detailed in the paper. The authors might have found that within-sample permutations are more effective or more appropriate for the specific type of data or problem they are addressing.
• Practical Considerations: Practical considerations, such as computational efficiency or ease of implementation, could also have influenced the design choice. Within-sample permutations might be easier to implement or computationally less expensive than between-sample permutations.
• Errata/Clarification: It is possible that the paper has an erratum or clarification that addresses this discrepancy. Checking for updates or contacting the authors of the paper might provide additional insights.

Conclusion

In conclusion, the choice of performing convex combinations within each matrix in the discrim_two_sample.py implementation likely stems from a desire to specifically test the null hypothesis that the two samples are drawn from the same distribution, even after introducing internal variations within each sample. This approach might be more effective at preserving the underlying structure within each sample and avoiding the dilution of genuine differences. While the paper's description might suggest a different approach (between-sample permutations), the current implementation likely reflects a deliberate design choice based on theoretical or practical considerations. Further investigation, such as consulting the paper's authors or examining related literature, could provide even greater clarity.

For more information on permutation tests and hypothesis testing, you can visit this Wikipedia article.