If they don't bring stuff to a building in progress it could be that you don't have enough serves or you forgot one part of road somewhere.
or if you paly the peasants rebellion they do not bring stuff if you have too many blueprints at once.
If the problem reoccurs, make sure you don't have too many building foundations laying around.
I have the same problem and none of the solutions above works for me. In TSK mission 2 and 4, I am unable to construct a wine yard and a leather armorer. At some point I had plenty of resources, plenty of serfs and builders available (most of them with question marks above their heads), zero building projects in progress and a road to where I wanted to construct the wine yard. Whenever I placed the wineyard, one builder would go to it to level the surface and... thats it! No other builders got involved, even if they stood right next to it and no building materials were being supplied by serfs. I keep destroying the construction site and rebuilding it, but the same thing happens over and over again, even if I put it somewhere else and even if there are plenty of serfs and builders available. The same goes for the leather armorer.
Now I have read on this forum that wine yards aren't actually that important, which doesn't make sense to me (people need to drink, right?), but the lack of leather armor really is a pain in the @#$.
Is there a possibility that this bug only applies to the Shattered Kingdom campaign? I was able to build a wine yard in the first mission, for no apparent reason.
EDIT: I see I AM able to build a leather armorer in TSK mission 4, but it's not producing anything because it isn't getting any leather. Instead of the leather armorer I now have this problem with the fisherman's hut.