If you don't mind me asking, what is your total acreage?
Your process seems similar to what interests me. Start slow, don't build right away (or until I can afford to) and with the intention that it might be a retirement property in the future. I would be curious to know what were the factors that lead to to finally pulling the trigger and making the purchase?
A ski friend told me (after I asked him how much land he had) that is was not polite to ask or tell. I never knew that. I'll say it is bigger than Camp's "small lot" size.
I bought the smaller lot to impress a girl. Not sure if she was impressed but she did marry me.
I made the purchase on the bigger piece primarily because it came up for sale.
We put our cabin on the most "fun" spot on the smaller lot, which was pretty close to the property line, barely within the rules 30 ft from the edge. With the land coming on the market bordering me on two sides, I imagined someone buying it, sub dividing it into four lots, and running two driveways right past my cabin to get to "flag lots."
That idea really bummed me out, this land is almost my religion now. Anyway I KNEW I couldn't afford it, but I also knew I'd always regret it if I never tried to get it. I borrow every cent I could, somewhat responsibly. My new loan was big and long (22 years), but at a lower rate than my old loan. I worked it out so my mortgage payment from the old load to the new loan was the same, I knew I could afford it.
So I borrowed the money, made my offer, and included a heartfelt plea, about how I loved that land.
He agreed to sell it to me for $5k less than my offer, because he "knew I'd have closing costs."
The guy who sold me the land is in his eighties. He bought the land originally, well he had done pretty well in life, and just didn't want to see another driveway on our road. Seriously he could have given me the land for free and still had far more than he needed. He was known to be extremely generous.
The whole thing seems like destiny to me.