Rome received all of its water (according to Encarta, about 38 million gallons a day) through a system of aqueducts. All water flowed to the city by gravity, but because it was arriving from surrounding hills, it could be stored in large cisterns very similar in concept to today's water towers (the main difference is that cisterns are filled from the top).
Water flowed from the cisterns either through pipes to individual houses or to public distribution points. Fountains served both decorative and functional purposes, since people could bring their buckets to the fountain to collect water. The cisterns provided the height needed to generate water pressure for the fountains to spray. As discussed in How Water Towers Work, a foot of height generates 0.43 pounds per square inch (psi) of water pressure, so a cistern does not have to be that tall to develop enough pressure to give a fountain a reasonable display.
Here are some interesting links: