feOffset is a rather simple filter primitive: it just moves the given image a given distance. If we don't go into sub-pixel rendering accuracy, the framework in Inkscape makes it possible to implement a renderer for this filter by merely updating image location info. So far so good.
What's the problem then? Well, even for simple images rendering this filter requires us to actually render another part of the picture: suppose we wanted to render the rectangle from (x0, y0) to (x1, y1). When we have a feOffset filter with distance (dx, dy), we actually need to render area (x0+dx, y0+dy) (x1+dx, y1+dy) and then apply the filter.
This can be accomplished, rendering blur needs similar functionality and it works quite OK. The bigger problem is, when feOffset is used on background image. In this case, at the time when background is drawn, it's not known, that some filter later on will try to use background outside this area. All the ways to remedy this I can think of, are rather complex. Still, this whole background access functionality is a rather complex subject. It will likely require some kind of re-entrant rendering, that can go back to rendering background when this kind of situation arises.
Also I've spent a way too much time trying to fix background access functionality, that wasn't broken in the first place. Obviously it's dangerous to modify one's own old code, because it's easy to assume remembering its workings. If that was someone else's code, I would have first taken a good look at how it works.