Two papers indicate that tofu is connected with cognitive problems. Maybe the legumes are displacing brain-healthy nutrients like fish?
One study involved Americans. This longitudinal demonstrated that higher tofu consumption at midlife was associated with poor cognitive test performance, as well as low brain weight measured with neuroimaging or autopsy.
Another study took place at three cities in Indonesia. It found that tofu consumption was associated with worse memory. The authors speculate that this could be due to hormonal effects or toxins.
Brain aging and midlife tofu consumption. (Link)
High tofu intake is associated with worse memory in elderly Indonesian men and women. (Link)