Nah. The saviors have what appears to be at least a couple hundred men, and the offerings from the communities are often as small as 12 pieces of fruit. Hilltop has maybe half a dozen people who actually leave the walls to scavenge. Yet the sanctuary was shown to be plush with goods and supplies, even having their own 'marketplace'. They are likely taking stuff from more communities we don't know about, and several dozen saviors probably contribute by scavenging themselves. Taking all the food and supplies from the new communities they find has become about greed, not necessity.

As dirty Daryl once noted "this place ain't about gettin by. Its about gettin it all".

 

I feel like we've never been shown Negan's group in full. Even in the last episode when he announced "We are going to war," the numbers didn't seem (to me) to be that huge.

 

I never got the impression that they were surviving on what the other groups gave them anyway.

We know his outposts collect from their designated group. Simon (formerly thr satellite location) collects from hilltop, Gavin collects from kingdom, and Negan himself collects from alexandria. They pick up their tribute, give some to the sanctuary and keep the rest. The sanctuary has workers who scavenge and grow crops, tend to livestock, and prepare processed foods. So they will be fine.

I believe Negan will tell his workers that they need to tighten their belts due to the lack of collections. But that will turn them against him even more. I don't think the lack of tributes will hurt Negan as much as the effect it will have on his, already unhappy, workers.