Does h depend on sample size?

Its point value depends on the two proportions. Sampling uncertainty and the power of a test depend on sample sizes and design.

Yes. Reversing which proportion is subtracted reverses the sign.

Can I use h alone to calculate a study's required sample size?

No. A power calculation also needs the test, significance level, desired power, allocation and relevant dependence assumptions.

AI Glossary / Statistics and probability / Intermediate

Cohen's h

Definition

Cohen's h is an effect-size measure for the difference between two proportions after applying an arcsine-square-root transformation. It is defined as h = 2 arcsin(sqrt(p1)) - 2 arcsin(sqrt(p2)), with angles measured in radians. It is often used in power and sample-size calculations for proportion comparisons. It describes a transformed difference, not a percentage-point difference, risk ratio or probability that one intervention is better.

What it means

A difference of ten percentage points has a different transformed magnitude near the middle of the probability range than near its edges. Cohen's h captures that feature through a nonlinear transformation of each proportion. Its sign follows the order of subtraction, so positive h means p1 exceeds p2. The magnitude is sometimes discussed using conventional effect-size labels, but practical importance depends on the outcome, baseline, consequences and setting. Always retain the original proportions when explaining the result to a business audience.

How it works

Express both proportions on the zero-to-one scale. Take each square root, apply the inverse sine in radians and multiply by two, then subtract the second transformed value from the first. The result lies between minus pi and pi. Reversing the group order changes the sign but not the magnitude. The transformation was developed in the context of variance stabilization for binomial proportions, which motivates certain approximate power methods. Computing h itself requires only the two proportions; conducting inference or planning a study also requires sample sizes, dependence structure, significance level, power and a specified testing approach.

Three practical examples

Experiment planning: express a proposed difference in independent conversion proportions on the scale required by a power-analysis routine.
AI evaluation reporting: supplement observed binary success rates with a clearly labeled transformed effect size.
Sensitivity analysis: compare how the same percentage-point improvement translates into h at several plausible baseline rates.

Think of it this way

Think of translating distances from a curved scale onto another measuring ruler. Equal distances on the original ruler need not remain equal after the translation. Cohen's h provides a particular mathematical ruler for proportions, but the original rates remain essential for understanding what actually changed.

Worked example

Suppose two hypothetical groups have success proportions 0.60 and 0.50. Their absolute difference is 0.10, or ten percentage points. Cohen's h is 2[arcsin(sqrt(0.60)) - arcsin(sqrt(0.50))] = 0.20136 approximately. Now compare 0.20 and 0.10. The absolute difference is again ten percentage points, but h is approximately 0.28379. Thus equal raw differences need not produce equal h values. If the first comparison arose from 60/100 versus 50/100, or from 600/1,000 versus 500/1,000, the point estimate of h would be unchanged. The precision of the estimates and power of an appropriately specified test would differ. The effect-size formula alone does not provide a p-value or confidence interval.

Illustrative case study

Fictional case: Birch Product Research plans a study of an assistant's effect on task completion. Analysts use a proportion-based power tool that accepts Cohen's h. They calculate h from a baseline rate and a minimum improvement agreed with product specialists, then examine alternative assumptions rather than relying on one optimistic forecast. Their report shows the original rates and percentage-point improvement beside h so stakeholders understand the operational target. They separately consider whether users contribute repeated tasks, which would invalidate an independent-observation calculation if ignored. This invented case illustrates planning, not evidence of a successful product intervention.

Common misconceptions and mistakes

Cohen's h is not Cohen's d, which concerns standardized mean differences, or Cohen's kappa, which concerns agreement. Entering percentages such as 60 instead of proportions such as 0.60 is invalid. Using degrees instead of radians changes the conventional scale. A numerically small h may still matter when outcomes have major consequences or affect many people. Conversely, a conventionally large h does not guarantee a worthwhile business decision. The transformation does not remove confounding, measurement error or dependence, and boundary observations do not eliminate uncertainty about the underlying population proportions.

Business use

Businesses may encounter h in software for experiment design and power analysis. Treat it as a bridge between an operational target and a statistical calculation, with the assumptions made explicit. Define the outcome consistently and identify whether observations are independent, paired or clustered. Report costs, benefits and absolute differences alongside the transformed measure. When comparing studies, check that outcome definitions and group ordering match. A common numerical scale does not make studies with different populations or measurement rules directly interchangeable.

Check your understanding

Why do the two ten-percentage-point comparisons in the worked example have different h values?

Answer: The arcsine-square-root transformation is nonlinear. A change from 0.10 to 0.20 covers more distance on that transformed scale than a change from 0.50 to 0.60. The raw operational changes remain ten percentage points in both cases.

Related terms

Effect size
Proportion
Sample size
arcsine transformation
power analysis
risk difference
risk ratio
binary outcome
variance stabilization
Cohen's d

Learn next

compare absolute differences, relative differences and transformed effect sizes for the same proportions. Study how sample size changes precision without changing a fixed effect-size target. Then inspect the assumptions of the power method being used, especially when AI evaluations contain repeated tasks from the same person or related benchmark items.

Sources

R pwr package, ES.h: Effect size calculation for proportions: https://search.r-project.org/CRAN/refmans/pwr/html/ES.h.html statsmodels, proportion_effectsize documentation: https://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_effectsize.html R pwr package manual, including proportion power functions: https://mirror.math.princeton.edu/pub/CRAN/web/packages/pwr/pwr.pdf

Previous: Cognitive science Next: Cohen's kappa