The reason he doesn't care for Big Baby is due to the fact that he mistreated and abused him so badly after taking him away from Daisy while pretending to act like a father figure to him. His minions would eventually turn against him after Woody exposes his true colors. While his henchmen are abusive and behaving like bullies towards Andy's toys, this is due to the fact that they are following Lotso's orders, making their actions entirely attributable to him.

He doesn't show any care for his minions (Twitch, Chunk, Sparks, Stretch, Ken, Big Baby, the Monkey, and the Bookworm), merely using them as tools/pawns for doing the work of ruling Sunnyside.

After detaining Andy's toys, he shows them Woody's hat, which he left behind during his escape, under the false pretense that he killed him to break their spirits and discourage them from leaving Sunnyside.

He reprogrammed Buzz into becoming one of his henchmen when he refused to join the Butterfly Room alone, manipulating him into thinking that his friends are helpers of Emperor Zurg and having him beat down and lock up his own friends.

It is implied that he had Chuckles beaten, broken and injured after he decided that he wasn't going to have any part of his empire, despite him once being his friend.That's why he offered Buzz Lightyear a chance to go there, because Buzz could be a useful pawn. Even though he lets obedient toys live safely in the Butterfly Room, he doesn't care about them at all and just wants to keep them submissive to him.Upon arriving at Sunnyside Daycare, he became its ruler and turned it into a toy prison where only toys who he views as special have the privilege of being played nicely in the Butterfly Room while he forces the rest to be beaten, abused, and played with roughly by younger kids in the Caterpillar Room, where those who break are tossed in the trash and sent to the dump, which may very well lead to their incineration, and those that defy Lotso are placed into the sandbox.

Chuckles had described him as starting out as a caring toy only to throw all of that away once he saw Daisy with another Lotso and became a wrathful and abusive tyrant who imprisoned and hurt toys for his amusement.

Toy story 3 incinerator scene lotzo