Exactly. As I approached P4 and started planning my promotions, I realized that a lot of heroes' 3/6 starting gear at P4 made for a decidedly or even clearly inferior hero, and that trend would seem to hold true for promotions to Orange where a legendary skill isn't crucial.
The heroes I promoted to P4 right away didn't really gain or lose much from their promotion, like Crimson Witch or Moon Drake, or had good enough starting 3/6 P4 gear, like Unicorgi and Roller Warrior, IIRC (not looking these up now). Some heroes lose out with an immediate promotion to P4, like Plant Soul losing her large improved healing buff until High Tea becomes available. Some heroes have IMO pretty abysmal 5/6 P4 gear compared to their decently enchanted P3 gear, like King Kaiju, Cyclops Shaman, Dark Dracul (IMO), and Unstable Understudy (IMO, the large crit buff at P3 is ideal for her design). I'm having that last group bypass P4 all together to save me a fair amount of gold and much more gear, on top of an arguable loss in efficacy.
The biggest example by far, though, is me keeping Medusa at P3. At P3, she can get +6 expertise and +45% conservation. Both of those stats allow me to compete with Medusa above my team level in a big way. Her P4 gear gives her an appreciable damage boost, but Medusa isn't used for her damage. In my specific situation, I think I'll pass on promoting her immediately to Orange too. The way I see it, P3 Medusa helps me win in fight pit and expedition, but theorycrafting suggests Orange Medusa with her legendary skill would only help me win in expedition against Bone Dragons, which are actually becoming steadily easier yet more prevalent as I close the level gap. My expedition runs aren't hurting for that energy reserve or a tougher Medusa. Yet.
TL;DR Actually review the gear you would be giving up compared to the gear you would be gaining. How important are those stats to your heroes' designs? When might promotion result in a net gain in areas you find acceptable? That's when you should promote.