As stated, the answer is no, Freiman's inequality no longer holds. The counter example is $A$ being the vertices of an equilateral triangle on the unit circle. I found this by looking at the proof of Freiman's inequality (given as Lemma 5.13 of Tao-Vu book on Additive Combinatorics) and see where the proof fails.
In my particular setting, the set $A$ satisfies some additional convexity conditions, so perhaps the result may hold for such situations.
UPDATE: it turned out that the proposed inequality held under one extra assumption: that the origin is out sideoutside the convex hull of $A$. That will rule out the examples such as above, and still implies the classical inequality. The proof follows the same line as the original proof, with some extra care needed.