Opened 3 weeks ago

#827 new enhancement

Investigate background fetching of documents from remote

Reported by: Olly Betts Owned by: Olly Betts
Priority: normal Milestone: 2.0.0
Component: Backend-Remote Version:
Severity: minor Keywords:
Cc: Blocked By:
Blocking: Operating System: All

Description

(Spun off #353)

The old "network" backend (removed in e7c42c964728652432d5530b8f6913bc22bd19c6) implemented specialised request_document() and collect_document() methods. These were used to allow the MSet::fetch() methods to request the documents which the user wanted in bulk, rather than having a request-fetch conversation across the link for each document.

The new "remote" backend which replaced it doesn't have this specialisation. My testing at the time didn't show any speed up, but that may not be true in every situation (e.g. higher latency connection, or with a cold disk cache).

Since then, MSet::fetch() was hooked up for local backends to trigger a pre-fetch request, which can help a lot when the disk cache is cold.

I've just pushed 02bfcf0541c6beacf25501d95f759baf485b058b which makes MSet::fetch() send a request to the remote which triggers a prefetch there, which should help the case of a remote with a cold disk cache.

We should prototype and profile to investigate if it would be worthwhile implementing something more like the old network backend had.

Change History (0)

Note: See TracTickets for help on using tickets.