First I want to say I was a little concerned as to what your post was going to be. I thought you were going to be one of those early learning, nay Sayers that wished to bash early learning because there are no proven benefits or something like that. Instead I was quite piqued by your question.
First I will say my husband and I are "poor" we are a low income, single income family of 3. And we are very happy. We have been blessed to live in a town affordable enough that we can own our tiny home for less than the price of rent. Sometimes I yearn for more.. But we live frugally, make the best choices we can and save what we can to enjoy the little splurges. I did many of the things you did. We cloth diapered and breastfed. I buy my son gifts from second hand stores. And I seldom pay full price for anything.
Neither my husband or I are career driven, money yearning types of people. For us the joy comes from raising a family and enjoying the simple life we have.
So why do I do early learning with my son? First why not?

we can and we do. But more seriously... I think there are a lot of reasons. And these come in no particular order. I am sure there are more.
I want my son to have an advantage with school. I want him to have some struggles, as it will make him stronger, but I want him to have the chance to have a running start. I think early learning gives him that.
I want my son to become anything he wants to be. If he wants to be a physicist and he struggles with history or geography in school, I want him to have had that head start so that he can cover all those "unnecessary" for his degree subjects by having more time to apply himself to them. And to still maintain good academic standing. If he wants to have a low paying job and be happy, so be it. But I don't want him to regret not having had the chance to be anything he wanted to be when he was younger and his brain was more supple and things were easier to learn.
Also... Scholarships help pay for college.

This one is very important to me. Because it is fun! And early learning enhances life. My son just turned 4 and his life is so rich. Not in a monetary way. But he has so much knowledge that his play, his creativity, and his daily life is richer, more full of variety and ideas. People have mentioned that his interests are so complex and he loves such a wide variety of things. One moment he is playing engineer as he constructing an elaborate contraption, that we learnt about from learning about, simple machines and other basic physics concepts like forces, inertia, energy and more. The next moment we are discussing how the mammary glands of a cow makes milk as he is playing farmer. Next moment we are acting out scenes from A Little Princess as we discuss charity and kindness and having a positive attitude. Many of my friends children are mostly just acting out the latest Disney movies and are obsessed with certain characters to the expense of all the wonderful things in our world. This leads to me next reason.
This world is a wonderful amazing place and I think knowing how things work is joy. Knowing why we have seasons, how to predict weather, knowing how to speak several languages, or how to converse with pretty much anyone, anywhere about a variety of subjects because one is so well read is bliss. It makes for a happier life. At least I think it is. And it is my ultimate goal to be sage like.
The above reasons were important before my son was born. This next reason mattered more after I met him. My son had some issues at birth and some medical issues for the first few years of life. There was an 80% chance of mental retardation from lack of oxygen at birth. Then an additional 80% chance on top of that due to failure to thrive. He was in an early intervention program, red flagged for autism, later with a preliminary diagnosis, he had social issues and speech regressions. Now at 4 years old he is above average developmentally, he has extensive vocabulary and his autistic traits are manageable. He is independent, reads well, loves math and science. He asks hundreds of challenging and well thought out questions. And he is a great little companion for me to spend my days with. Which makes us both happy.