Major considerations:

Wheel mass
Motor maximum torque
Motor revolution rate

Needed information:

Rocket maximum revolution rate
Rocket moment of inertia

Conservation of angular momentum gives that I_rw_r + I_ww_w = 0, which can be rearranged to say that (I_w / I_r = w_r / w_w), to find the rotational inertia the wheel should have, knowing (1) rocket's and wheel's maximum revolution rates and (2) the rocket's moment of inertia. However, the rocket's moment of inertia is relatively harder to measure, so the mass can be used to approximate:

The "ideal" moment of inertia for a given mass is mr². The "efficiency" of some mass as a rotational inertia can be expressed using a parameter α, where (I = αmr²), in which case α covers the interval (0,1). We can reasonably assume that α_r < α_w, as the wheel will be designed to use mass "efficiently"; rockets generally are not. Then the ratio of inertias (assuming equal radii) is given by I_r/ I_w = α_rm_r / α_wm_w. Assuming α_r < α_w, we have that I_r / I_w < m_r / m_w; that is to say, the mass ratio gives an upper bound for the rotational inertia ratio, and is therefore safe to use for preliminary sizing according to w_w / w_r = m_r / m_w.

Notable model inaccuracies:

Underestimate of reaction wheel mass: The maximum revolution rate underestimates the angular momentum the wheel needs to remove from the rocket.
Overestimate of reaction wheel mass: The mass ratio (intentionally) overestimates the necessary inertia ratio, as described.
Underestimate of reaction wheel mass: The radii of the wheel and rocket are assumed to be equal; however, the wheel's radius is necessarily at least a little bit smaller.

Browser not supported