Perhaps it may be helpful to bring up J.R.R. Tolkien's own views on the subject of magic in Middle Earth?
In "The Letters of J.R.R. Tolkien" (letter 155), Tolkien apparently apparently became concerned about his overly casual use of the word "magic" in his writings, and -- disagreeing with the lady Galadriel's explanation -- tried to explain Middle Earth magic more clearly.
Magia (true effects) and goeteia (illusions)
Morality of magic not intrinsic
I suppose that, for the purposes of the tale, some would say that there is a latent distinction such as once was called the distinction between magia and goeteia [this is defined in the O.E.D. as 'witchcraft or magic performed by the invocation and employment of evil spirits; necromancy'] Galadriel speaks of the 'deceits of the Enemy'. Well enough but magia could be, was, held good (per se), and goetia bad. Neither is, in this tale, good or bad (per se), but only by motive or purpose or use. Both sides use both, but with different motives.
The supremely bad motive is (for this tale, since it specifically about it) domination of other 'free' wills. The Enemy's operation are by no means all goetic deceits, but 'magic' that produces real effects in the physical world. But his magia he uses to bulldoze both people and things, and his goeteia to terrify and subjugate.
Their magia the Elves and Gandalf use (sparingly): a magia, producing real results (like fire in a wet faggot) for specific beneficent purposes. Their goetic effects are entirely artistic and not intended to deceive: they never deceive Elves (but may deceive or bewilder unaware Men) since the difference is to them as clear as the difference to us between fiction, painting, and sculpture, and 'life'.
Magic was rarely used
Tolkien then pointed out that characters, whether good or evil, generally lived by ordinary (non-magical) means even if they were capable of magic. Regardless of the reasons why the elves or wizards chose restraint, Sauron chose mundane methods out of practicality. For magic was difficult, and slavery, machinery and other ordinary means were far easier ways to achieve the same results.
Of course another factor then comes in, a moral or pathological one: the tyrants lose sight of objects, become cruel, and like smashing, hurting, and defiling as such. It would no doubt be possible to defend poor Lotho's introduction of more efficient mills; but not of Sharkey and Sandyman's use of them
Magic is inherent, not lore
Tolkien's last clarification is that true magic wasn't learnable. It wasn't "lore." Rather it was "inherent power not possessed or attainable by Men as such." He noted that Aragorn's "'healing' might be regarded as 'magical', or at least a blend of magic with pharmacy and 'hypnotic' processes." But this was not an issue because Aragorn wasn't a pure Man. He was "at long remove one of the 'children of Lúthien'." However distant from this ancestry, Aragorn was part Elf, part Maiar, and part Man.
He also pointed out the seeming contradiction that the Numenorians used "spells" when making weapons. He didn't answer this contradiction. His letter cuts off. But others have noted that some Numenorians, the minority with royal ancestry, had elvish blood.
(As a sidenote, I -- not Tolkien -- will point out that the only sorcerers other than Sauron I recall implied in "The Lord of the Rings" or "The Silmarillion" were I believe "Black Numenorians," which was a term that referred not to skin color but rather to Numenorians who worshipped Morgoth. Sauron convinced many Numenorians to worship Morgoth during Sauron's imprisonment on Numenor before the island's downfall. Those Black Numenorians overseeing Numenor's conquests and ports in Middle Earth, such as Umbar, survived and continued to call themselves Numenorians. In contrast, those of Numenor who worshipped Eru, revered the Valar, remained friends with the Elves, and fled the island before it sank called themselves Dunadan afterwards. Some of the Black Numenorians retreated from the Dunadan to Harad and still existed even in the time of the War of the Ring. The Mouth of Sauron at the Black Gate was a Black Numenorean.)