It is graphite for sure in the cases I have had. For one it is also in the keyway, this would be brass on brass wearing so shouldn't be black and slippery to the tough. The area where the knob turns doesn't usually wear much as all the wearing areas are inside the lock body and are lubricated with heavy grease. Old-timers will recall the green grease for example that Schlage used in their A line knobs. The grease would get all over you if breaking a lock down completely to remove the cylinder.
The only lock I know of that used to really wear the knobs was the old residential Weslock knob.