In a finite lifetime, you won't be able fully to inspect an object with parts that are infinitely far from you, at least if we assume that you are limited by the speed of light. But there's other evidence. For example, you may be able to measure the gravitational pull of the ladder. If this pull turns out to be exactly what our theory would predict for a ladder that's like the piece of it we have before us (same material, thickness, density, etc.) and infinitely long, then this would be evidence for infinite length. (Note here that the gravitational pull exerted by any one inch of ladder declines with the square of its distance from you. So no matter how long the ladder its, its gravitational pull will not be infinite.) It's also possible that the ladder is expanding (as our universe is), or perhaps contracting. In that case you get a nice Doppler effect: a transformation of light reaching you from distant parts of the ladder -- the farther the light has traveled, the more strongly transformed it arrives. So evidence can provide reasonable assurance. And we have such reasonable assurance now that the universe is in fact finite.

But it's worth noting, as you suggest, that such evidence depends on "reasonable" assumptions. We assume that the laws of nature we've found to hold true around our space-time location also hold millions of light years away and millions of years in the past. Do we have reasonable evidence for this assumption? Well, yes, sort of, if with this assumption we can fit all our observations into one coherent account. Physics hasn't quite achieved this yet. But once physics delivers such a comprehensive theory, then it'll strike us as unreasonable -- bizarre -- to defend an infinite size of the universe by appeal to diverse laws of nature holding at different regions of space-time. But you should ask: the fact that some hypothesis strikes us as bizarre, how much reassurance is this that it's actually false?

In relation to my earlier answer, the following article from the Economist may be of interest. It's advertised as follows: "Can the laws of physics change? Curious results from the outer reaches of the universe." The link is

www.economist.com/node/16941123?story_id=16941123&fsrc=nlw|hig|09-02-2010|editors_highlights

This is not exactly what I had in mind, but relevant nonetheless.

BTW, this question is probably best classified under "physics" rather than "mathematics."