Either an open or a short could cause what you're describing.
Say you start at the point representing an open, (1, 0) in cartesian coordinates, then if there's any transmission line between your calibration plane and the location of the open, there will be a phase delay in the reflected wave. The phase delay increases at higher frequency because the wavelength decreases while the physical length of the tranmission line segment stays the same. So on the Smith chart you see a rotation around the outside of the chart as frequency increases. Same thing if you start with a short at (-1, 0).
To tell the difference you need to look at the phase plot near 0 Hz. If the phase at 0 Hz is 0, you have an open. If it's 180 you have a short. If your sweep doesn't go down to low enough frequencies to extrapolate to 0 Hz you could probably just measure with an ohmmeter.
the phase plots look like sawtooth waves
Phase is usually measured either in the range of (-180, 180) degrees or (0, 360) degrees. So if the sweep passes through more than a full cycle of phase delay, you'll see that as a sawtooth on the phase plot. It doesn't mean there's a real discontinuity, only that the phase/magnitude plot can't show continuous paths from 180 to -180 the way a polar plot can.