Google App Engine’s
dev_appserver prevents filesystem writes because GAE has no filesystem – but sometimes you want to write anyways!
I was working over a big set of data from the Google App Engine High-Replication Datastore and I found that, somewhere in my loop, memory was slipping away…
So I got objgraph, which creates a graphic based on your memory usage. But I had a problem: Google App Engine doesn’t have a filesystem, so the development server prevents you from writing to files.
objgraph couldn’t create my graphic!
Luckily, the fix was simple: I found the line that threw the error and commented it out. (The file is
# Starting on line 242: def __init__(self, filename, mode='r', bufsize=-1, **kwargs): """Initializer. See file built-in documentation.""" # if mode not in FakeFile.ALLOWED_MODES: # raise IOError(errno.EROFS, 'Read-only file system', filename) if not FakeFile.is_file_accessible(filename): raise IOError(errno.EACCES, 'file not accessible', filename) super(FakeFile, self).__init__(filename, mode, bufsize, **kwargs)
By the way, it didn’t turn out to be a memory leak. It was GAE’s NDB caching in action – I just disabled it by:
for d_key in query.iter(keys_only=true): d = d_key.get(use_cache=False, use_memcache=False) # ... d.put(use_cache=False, use_memcache=False)
Then I was all clear!