What drives the functional organization of the visual system? All proposals balance the causal roles of two pressures: innately specified cortical patterning mechanisms, establishing large-scale network architecture, and self-organizing mechanisms driven by the statistics of natural experience, effecting local organization. In part 1, I will characterize the functional organization of object-selective cortex and how well it converges with network architecture as revealed by functional connectivity measures. We find that long-range network-level distinctions mirror category-selective organization of object cortex, which points to the viability of "built-in" organizational pressures. However, we also find evidence for an extensive local interconnected network, and use deep neural network models to provide insight into this finding. In part 2, I will consider the large-scale organization of the early visual areas using Kohonen mapping techniques. This computational work suggests that V1/V2/V3 do not have to be specified by distinct patterning mechanisms but can emerge from self-organizing processes. Considering the results of both of these projects together, we will discuss these local and long-range organizing pressures and the cortical scales at which these two causal pressures meet.