I've been using Flex, a tool for generating scanners (tokenizers), for a while now, and I never worried about memory management of the C scanners it generates: I thought memory would be handled automatically, i.e., freed when the scanner finished its execution.
Actually, this is not the case: you have to deal with it yourself! If you don't you will end up with a memory leak of about 16k (see also the manual); you can use valgrind to check this.
Basically, all you have to do is to call yy_delete_buffer when the scanning is over. For instance, when you reach the end of file. Here's what I do in my flex scanners:
Unfortunately, you'll still experience a memory leak of about 4 bytes, and this looks like a bug. Hope it will be fixed soon...