It is easy for a campaign to get out of control, magic wise. Take "light", a first level wizards spell. It lasts one hour per level of the wizard. (In original rules, it lasted until it was put out). With enough wizards running around in your campaign world, every single town could have streetlights lit all night long with light spells. At this point, is magic really magical? Make something common enough, and it isn't magic at all. It is simply mundane. What about flying? The most interesting, well planed out architectural challenges can be thwarted by a flying character. And adventure that could last an entire evening can be circumvented by one annoying fly spell. Oh sure, you can embed the structure with all sorts of counter magic, making it impossible to fly. But it becomes an arms war... upping the magic ante to make basic things feel basic. Teleportation? Again, a bane of well planned adventure design. Simple common spells that force a GM to think several steps ahead and up the ante, making magic so ubiquitous that all feeling of mystery and antiquation is lost. Think hard about what spells you allow in your campaign. You may get some harsh protests, but limiting the spells your players have access to can create a much richer experience for all.
How common are magic items?
Is there a "magic shop" in every village, littering the landscape like Starbucks? Or are magic items mystical, mysterious, and rare. Sure there are a lot of creatures that require a magic items to be struck, but a little rule bending (and reliance on traditional folklore) and that item could be made of a specific mundane material.
Vampires and werewolves have an aversion to silver. Many fae creatures abhor iron. A nice silver backup dagger, or an impromptu fireplace poker, can make a nice stand in for traditional magic items.
As for combat advantage, nothing wrong with having quality weapons that afford a +1 or +2 without being magical.
Keeps everyone on their toes, and makes finding the right weapon for the right foe part of the adventure.
What about wands, rings, etc...
Lets get back to that "light" spell. How about a type of rock, a material component for the spell, that is required. So you can't just make anything light up. It has to be a specific element, perhaps only found in a certain part of the campaign world. Lets say rose quartz for the heck of it. So you have one rock, non magical, the size of an orange, that you can make light up with a light spell. If you lose it, you have to go adventuring. Or maybe find a lapidary in a decent sized town, or an old woman, shunned by the villagers, who deals in folk magic and lives in a hut at the end of a village.
Wands and rings, rather than having powers of their own, can magnify the powers of those using it. Want a magic wand? Go make one! One that maybe casts a couple of extra first level spells a day. YOUR spells, not spells "embedded" in the wand. Maybe that rose quartz is embedded in a ring, so that light spell can be cast from your hand.
Leave the "real" magic to adventuring. And don't be afraid to let a group get ahold of a magic item that is far more powerful than their level would seem to allow. They don't have to keep it. Let them have fun playing with items before Fate steps in and takes it away, keeping game balance in play but having the excitement of powerful magic in the world.
It doesn't have to be an all-or-nothing experience. Done right, low level adventures can be very exciting, and the players don't have to be in a race to attain high levels just to have a good time and a chance at survival, and get to play with the "cool" stuff without throwing off game balance and escalating the magic arms race in a campaign world.