Along with the "Curveball" thread dealing with advanced Goblins (good at warcraft, engineering, mining and arcane magic) and the d20 Chaos Magic resource book, I also re-thought the humble troll. My thought goes that they really should be a little more powerful, but also varied. Having split off from Faerie Folk like the goblins (giving homage to their faerie tale roots versus Tolkien), they developed skills at traps, poison, and illusion magic in a sub-breed, harder skin and natural "rocky" camouflage in the mountain breed, and these new sub-breeds have these abilities instead of regeneration. The regen ability still remains in one breed.

So the obvious question is how much tougher from normal stats, and then what is the equivalent level of illusion that can replace the regeneration? Or, in the case of mountain breeds, how good is the camo, and how much harder are they to hit? I would say that all breeds like ambush and traps, that just seems a good fit.

There is a good reason for this, as the troll tribes in the south of my first continent are greatly feared, and normal D&D trolls cease very quickly to instill any fear, being fairly milk-toast stat wise.

Obviously the stats should depend upon the level of the creature itself, as a level one troll early on would not have the same as a level twenty troll later, but that's not really the question at hand.

As far as Illusion goes, it should be set high enough to off-set the loss of regeneration, therefore it should be rather high as they need it to survive in an encounter. An good example would be if the normal Trolls had a regeneration of 5, then I would say the Troll with traps, poison, and illusion should be 3, 2, and 4 respectively. Combined they out-weigh the regeneration but alone are weaker than it, a good mix.

Now for the case of camouflage, I think that as long as the Troll is in a position where the environment behind it relates to the camouflage then they should receive the bonus, and that bonus should be around 2 or 3. It takes just 1 point more to go from hitting to missing so even the small differences go a long way.

Overall I like your ideas on changing up the Trolls some and wish you good luck in your campaigns, hopefully my advice was of some use!

Nothing well. Regeneration is actually good. SLA DC is based on charisma, something trolls lack. Unless you given them Shadow Evocation and don't change the CR, you're not going to have any ability on the level of regeneration.

Actually, what you could do is use the Primordial Giant template from Secrets of Xen'drik can give some major mental stat boosts and Invisibility at-will, but it would take a big chunk out of their melee stats. You could throw Magic Blooded in for good measure, making them passable sorcerers. Best part? They'd still have regen.
Camo is equally pointless on a monster, as it won't matter much, and AC is lulz. If you want to buff them, why would you get rid of their strongest ability just to add stats that don't matter?

That said, the Mountain Troll, MMIII, gave them +8 to hide in mountains, has higher AC than a regular troll, and has fast healing instead of regen, so you might want to steal ideas from it, or just flat-out use it instead.
Uh, dude? Trolls are one of the best-statted monsters in the MM. If you want to make them threatening to higher leveled enemies, I'd just give them levels. Tome of Battle is especially good for this, as their RHD given them IL bonuses and access to higher level maneuvers, but ToB is the best book in 3.5, so that's not surprising. Or at least you could give them Power Attack. Seriously, the hell is up with them not having it?

But, meh, I like using the rules when at all possible, so that's all I've really got here.

That aside, though, a race of creatures that are CR 5 before class levels would be damned terrifying.

I thought about it, and to quote that old, hilarious beer commercial with the talking paper heads, "BRILLIANT!" No, seriously, I totally over-thought this, thanks for poking holes because it really helped.

I'm glad to help. Just remember that a lot of classes will be gimped on that pile of CR. Incarnum works well, and Tome of Battle is awesome, as usual. I'd go Warblade, personally, as the melee class on anything that you expect to matter much, because IHS and Moment of Perfect Mind are both kickass.