$\DeclareMathOperator\GL{GL}$Let $ H $ be an irreducible finite subgroup of $ \GL(n,\mathbb{C}) $. Define $ N^r(H) $ inductively by $$ N^{r+1}(H)=\{ g \in \GL(n,\mathbb{C}): g H g^{-1} \subset N^r(H) \} $$ starting from $ N^1(H):= H $.
Note that, since $ H $ is finite, the "semi-normalizer" $ N^2(H) $ is just the normalizer of $ H $ (see comment from LSpice). And thus is closed under inverses. But for $ r \geq 3 $ the set $ N^r(H) $ is generally not a group.
My question: Is $ N^r(H) $ closed under inverses?
The answer appears to be yes for $ n=2 $ (and is yes trivially for $ n=1 $ since then $ N^r(H)=\GL(1,\mathbb{C}) $ for all $ r \geq 2 $).
Context:
This construction is used for a naturally occurring object called the Clifford hierarchy which appears in quantum computing. For example, the (single qubit) Clifford hierarchy is the case where $ n=2 $ and $ H= \left\langle \begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}, \begin{bmatrix} 1 & 0 \\ 0 & -1 \end{bmatrix} \right\rangle $ is the single qubit Pauli group.
For $ H $ infinite there are many counterexamples. There is a counterexample with reducible $ H $ due to Arturo Magidin in this MathSE question with $ H = \left\{\left.\left(\begin{array}{cc} 1 & m\\ 0 & 1\end{array}\right)\in G\ \right|\ m\in\mathbb{Z}\right\} $ where $ x = \left(\begin{array}{cc} 2 & 0\\ 0 & 1\end{array}\right) $ is in $ N^2(H) $ but $ x^{-1} $ is not.
And there is a counterexample for infinite $ H $ with $ H $ irreducible given by Moishe Kohen in the answer to this MathSE question
More generally we can consider representations over fields other than $ \mathbb{C} $.
Some thoughts: If $ H $ is projectively self-normalizing (meaning the normalizer is just $ H $ with scalar matrices) then clearly the hierarchy $ N^r(H) $ is closed under inverses because $ N^r(H)=<H,\mathbb{C}^\times I > $ for all $ r \geq 2 $. This applies to all projectively self-normalizing subgroups $ H $ of $ GL(n,\mathbb{C}) $, including all $ H $ such that $ H/Z(H) $ is a maximal finite subgroup of $ PGL(n,\mathbb{C}) $ (since irreducible finite subgroups always have a projectively finite normalizer). For example all subgroups for $ n=1 $. And the subgroups $ H=2.A_5 $ and $ H=2.S_4 $ for $ n=2 $.