It should be noted, first off, that "mammal" is a term that has a bit of fuzziness around the edges of it. Marsupials, for example, are classified as mammals despite the fact that some of them lay eggs instead of using a womb for incubation.
Secondly, fossils are only one kind of record and in fact are not nearly as good as the DNA evidence is. Many animals, not just mammals, are vertebrates, and the way the body builds bone structure looks pretty much the same regardless of whether an animal is cold blooded or warm blooded for example, and for a long time people made the mistake of thinking Dinosaurs were cold-blooded lizard ancestors. The point is that the definitional things that differentiate mammals from other groups are all tied to the soft tissues - mammary glands, hair, warm-blooded, etc - Most of the time these are not preserved in a fossil and all you see is the skeleton. The skeleton of a cold-blooded tetrapod and a warm-blooded tetrapod don't look that different. We share the same basic body plan with lizards - 2 arms, 2 legs, optional tail, 1 head, food goes in the head and waste comes out the back near the tail, and internal organs consisting of lungs, heart, stomach, intestines, throat. The basic body plan for the torso is therefore the same.
The fossil record is really the hardest place to look for the dividing line between mammals and what came before. What is a lot easier to see is the DNA record in modern animals. You can identify genes that are associated with those soft-tissue traits that are missing from fossils, and compare them in modern animals to check how similar or different they are from each other, and analyzing that is the best place to get that sort of evidence.