capreolus.extractor.bagofwords

Module Contents

Classes

BagOfWords

Bag of Words (or bag of trigrams when datamode=trigram) extractor. Used with the DSSM reranker.

Attributes

logger

capreolus.extractor.bagofwords.logger

class capreolus.extractor.bagofwords.BagOfWords(config=None, provide=None, share_dependency_objects=False, build=True)

Bases: capreolus.extractor.Extractor

Bag of Words (or bag of trigrams when datamode=trigram) extractor. Used with the DSSM reranker.

module_name = 'bagofwords'

dependencies

config_spec

pad = 0

pad_tok = '<pad>'

load_state(qids, docids)

cache_state(qids, docids)

get_trigrams_for_toks(toks_list)

exist()

preprocess(qids, docids, topics)

id2vec(q_id, posdoc_id, negdoc_id=None, *args, **kwargs)

Creates a feature from the (qid, docid) pair. If negdocid is supplied, that’s also included in the feature (needed for training with pairwise hinge loss) Label is a vector of shape [num_classes], and is supplied only when using pointwise training (i.e cross entropy) When using pointwise samples, negdocid is None, and label is either [0, 1] or [1, 0] depending on whether the document represented by posdocid is relevant or irrelevant respectively.

transform_txt(term_list, maxlen)