I have a mystery tree at the back of the yard. It has small fruits, slightly larger than cherries, but with a pronounced crease in the side. The tree also sports some long spines in a few spots
Yes, they look like little plums. I see that cherry plums, Prunus cerasifera, can have spines, though I have never noticed that - did the flowers look like this: Myrobalan (Prunus cerasifera) ? You can see if the pits look like plum pits. The fruits can be tasty.