In circuit QED, protocols for quantum gates and readout of superconducting qubits often rely on the dispersive regime, reached when the qubit-photon detuning $Delta$ is large compared to their mutual coupling strength. For qubits including the Cooper-pair box and transmon, selection rules dramatically restrict the contributions to dispersive level shifts $chi$.
By contrast, without selection rules many virtual transitions contribute to $chi$ and can produce sizable dispersive shifts even at large detuning. We present theory for a generic multi-level qudit capacitively coupled to one or multiple harmonic modes, and give general expressions for the effective Hamiltonian in second and fourth order perturbation theory. We apply our results to the fluxonium system and show that the absence of strong selection rules explains the surprisingly large dispersive shifts observed in experiments. Quantitative predictions from our theory are in good agreement with experimental data over a wide range of magnetic flux and reveal that fourth-order resonances are important for the phase modulation observed in fluxonium spectroscopy.