:mod:`capreolus.tokenizer`
==========================

.. py:module:: capreolus.tokenizer


Package Contents
----------------

Classes
~~~~~~~

.. autoapisummary::

   capreolus.tokenizer.Tokenizer
   capreolus.tokenizer.AnseriniTokenizer
   capreolus.tokenizer.BertTokenizer



.. py:class:: Tokenizer

   Bases: :class:`profane.ModuleBase`

   .. attribute:: module_type
      :annotation: = tokenizer

      


.. py:class:: AnseriniTokenizer

   Bases: :class:`capreolus.tokenizer.Tokenizer`

   .. attribute:: module_name
      :annotation: = anserini

      

   .. attribute:: config_spec
      

      

   .. method:: build(self)


   .. method:: tokenize(self, sentences)



.. py:class:: BertTokenizer

   Bases: :class:`capreolus.tokenizer.Tokenizer`

   .. attribute:: module_name
      :annotation: = berttokenizer

      

   .. attribute:: config_spec
      

      

   .. method:: build(self)


   .. method:: convert_tokens_to_ids(self, tokens)


   .. method:: tokenize(self, sentences)