I've never actually thought about that, but I'd be really surprised if they scraped the dirt off the top before grading the track. It would just be way too time consuming, so I'm going with the dirt is full of rubber.
I also think at dirt tracks like Eldora they're constantly bringing in and adding new dirt. Every spectator there brings home a little dirt in their hair, and just think about the dirt on the cars that get loaded up and hauled away to be cleaned back at the race shop.