utils Package¶
bittorrent
Module¶
Torrenting utils, mostly for handling bencoding and torrent files.
- class flexget.utils.bittorrent.Torrent(content: bytes)¶
Bases:
object
Represents a torrent
- encode() bytes ¶
- flexget.utils.bittorrent.clean_meta(meta: Dict[str, Any], including_info: bool = False, log_func: Callable[[...], None] | None = None)¶
Clean meta dict. Optionally log changes using the given logger.
See also http://packages.python.org/pyrocore/apidocs/pyrocore.util.metafile-pysrc.html#clean_meta
@param log_func: If given, a callable accepting a string message. @return: Set of keys removed from C{meta}.
- flexget.utils.bittorrent.decode_item(src_iter: Iterator[bytes], token: bytes) bytes | str | int | list | dict ¶
- flexget.utils.bittorrent.encode_bytes(data: bytes) bytes ¶
- flexget.utils.bittorrent.encode_list(data: list) bytes ¶
- flexget.utils.bittorrent.is_torrent_file(metafilepath: str) bool ¶
Check whether a file looks like a metafile by peeking into its content.
Note that this doesn’t ensure that the file is a complete and valid torrent, it just allows fast filtering of candidate files.
@param metafilepath: Path to the file to check, must have read permissions for it. @return: True if there is a high probability this is a metafile.
cached_input
Module¶
- class flexget.utils.cached_input.InputCache(**kwargs)¶
Bases:
VersionedBase
- added¶
- entries¶
- hash¶
- id¶
- name¶
- class flexget.utils.cached_input.InputCacheEntry(**kwargs)¶
Bases:
VersionedBase
- cache_id¶
- entry¶
- id¶
- class flexget.utils.cached_input.IterableCache(iterable: Iterable, finished_hook: Callable[[List], None] | None = None)¶
Bases:
object
Can cache any iterable (including generators) without immediately evaluating all entries. If finished_hook is supplied, it will be called the first time the iterable is run to the end.
- class flexget.utils.cached_input.cached(name: str, persist: str | None = None)¶
Bases:
object
Implements transparent caching decorator @cached for inputs.
Decorator has two parameters:
name in which the configuration is present in tasks configuration.
key in which the configuration has the cached resource identifier (ie. url). If the key is not given or present in the configuration :name: is expected to be a cache name (ie. url)
Note
Configuration assumptions may make this unusable in some (future) inputs
- cache = TimedDict({})¶
- load_from_db(load_expired: bool = False) List[InputCacheEntry] | None ¶
database
Module¶
- class flexget.utils.database.CaseInsensitiveWord(word: str | CaseInsensitiveWord)¶
Bases:
Comparator
Hybrid value representing a string that compares case insensitively.
- operate(op, other)¶
Operate on an argument.
This is the lowest level of operation, raises
NotImplementedError
by default.Overriding this on a subclass can allow common behavior to be applied to all operations. For example, overriding
ColumnOperators
to applyfunc.lower()
to the left and right side:class MyComparator(ColumnOperators): def operate(self, op, other, **kwargs): return op(func.lower(self), func.lower(other), **kwargs)
- Parameters:
op – Operator callable.
*other – the ‘other’ side of the operation. Will be a single scalar for most operations.
**kwargs – modifiers. These may be passed by special operators such as
ColumnOperators.contains()
.
- prop: RODescriptorReference[MapperProperty[_T_co]]¶
- flexget.utils.database.entry_synonym(name: str) SynonymProperty ¶
Use serialization system to store Entries in db.
- flexget.utils.database.ignore_case_property(text_attr)¶
- flexget.utils.database.json_synonym(name: str) SynonymProperty ¶
Use json to serialize python objects for db storage.
- flexget.utils.database.pipe_list_synonym(name: str) SynonymProperty ¶
Converts pipe separated text into a list
- flexget.utils.database.quality_property(text_attr)¶
- flexget.utils.database.quality_requirement_property(text_attr)¶
- flexget.utils.database.text_date_synonym(name: str) SynonymProperty ¶
Converts Y-M-D date strings into datetime objects
- flexget.utils.database.with_session(*args, **kwargs)¶
” A decorator which creates a new session if one was not passed via keyword argument to the function.
Automatically commits and closes the session if one was created, caller is responsible for commit if passed in.
If arguments are given when used as a decorator, they will automatically be passed to the created Session when one is not supplied.
- flexget.utils.database.year_property(date_attr)¶
imdb
Module¶
log
Module¶
Logging utilities
qualities
Module¶
- class flexget.utils.qualities.Quality(text: str = '')¶
Bases:
Serializer
Parses and stores the quality of an entry in the four component categories.
- property components: List[QualityComponent]¶
- classmethod deserialize(data: str, version: int) Quality ¶
Returns an instance of the original class, recreated from the serialized form.
- class flexget.utils.qualities.QualityComponent(type: str, value: int, name: str, regexp: str | None = None, modifier: int | None = None, defaults: List[QualityComponent] | None = None)¶
Bases:
object
- class flexget.utils.qualities.RequirementComponent(type: str)¶
Bases:
object
Represents requirements for a given component type. Can evaluate whether a given QualityComponent meets those requirements.
- allows(comp: QualityComponent, loose: bool = False) bool ¶
- class flexget.utils.qualities.Requirements(req: str = '')¶
Bases:
object
Represents requirements for allowable qualities. Can determine whether a given Quality passes requirements.
- allows(qual: Quality | str, loose: bool = False) bool ¶
Determine whether this set of requirements allows a given quality.
- property components: List[RequirementComponent]¶
- flexget.utils.qualities.all_components() Iterator[QualityComponent] ¶
requests
Module¶
- class flexget.utils.requests.Session(timeout: int = 30, max_retries: int = 1, **kwargs)¶
Bases:
Session
Subclass of requests Session class which defines some of our own defaults, records unresponsive sites, and raises errors by default.
- add_cookiejar(cookiejar)¶
Merges cookies from cookiejar into cookiejar for this session.
- Parameters:
cookiejar – CookieJar instance to add to the session.
- add_domain_limiter(limiter: DomainLimiter, replace: bool = True) None ¶
Add a limiter to throttle requests to a specific domain.
- Parameters:
limiter (DomainLimiter) – The DomainLimiter to add to the session.
replace – If True, an existing domain limiter for this domain will be replaced. If False, no changes will be made.
- request(method: str, url: str, *args, **kwargs) Response ¶
Does a request, but raises Timeout immediately if site is known to timeout, and records sites that timeout. Also raises errors getting the content by default.
- Parameters:
raise_status (bool) – If True, non-success status code responses will be raised as errors (True by default)
disable_limiters – If True, any limiters configured for this session will be ignored for this request.
- set_domain_delay(domain, delay)¶
DEPRECATED, use add_domain_limiter Registers a minimum interval between requests to domain
- Parameters:
domain – The domain to set the interval on
delay – The amount of time between requests, can be a timedelta or string like ‘3 seconds’
- class flexget.utils.requests.TimedLimiter(domain: str, interval: str | timedelta)¶
Bases:
TokenBucketLimiter
Enforces a minimum interval between requests to a given domain.
- class flexget.utils.requests.TokenBucketLimiter(domain: str, tokens: float | int, rate: str | timedelta, wait: bool = True)¶
Bases:
DomainLimiter
A token bucket rate limiter for domains.
New instances for the same domain will restore previous values.
- flexget.utils.requests.get(url: str, **kwargs) Response ¶
Sends a GET request. Returns
Response
object.- Parameters:
url – URL for the new
Request
object.kwargs – Optional arguments that
request
takes.
- flexget.utils.requests.head(url: str, **kwargs) Response ¶
Sends a HEAD request. Returns
Response
object.- Parameters:
url – URL for the new
Request
object.kwargs – Optional arguments that
request
takes.
- flexget.utils.requests.is_unresponsive(url: str) bool ¶
Checks if host of given url has timed out within WAIT_TIME
- Parameters:
url – The url to check
- Returns:
True if the host has timed out within WAIT_TIME
- Return type:
- flexget.utils.requests.limit_domains(url: str, limit_dict: Dict[str, DomainLimiter]) None ¶
If this url matches a domain in limit_dict, run the limiter.
This is separated in to its own function so that limits can be disabled during unit tests with VCR.
search
Module¶
simple_persistence
Module¶
NOTE:
Avoid using this module on your own or in plugins, this was originally made for 0.9 -> 1.0 transition.
You can safely use task.simple_persistence and manager.persist, if we implement something better we can replace underlying mechanism in single point (and provide transparent switch).
- class flexget.utils.simple_persistence.SimpleKeyValue(task, plugin, key, value)¶
Bases:
VersionedBase
- added¶
- id¶
- key¶
- plugin¶
- task¶
- value¶
- class flexget.utils.simple_persistence.SimplePersistence(plugin=None)¶
Bases:
MutableMapping
Store simple values that need to be persisted between FlexGet runs. Interface is like a dict.
This should only be used if a plugin needs to store a few values, otherwise it should create a full table in the database.
- class_store = {}¶
- classmethod flush(task=None)¶
Flush all in memory key/values to database.
- classmethod load(task=None)¶
Load all key/values from task into memory from database.
- property store¶
- class flexget.utils.simple_persistence.SimpleTaskPersistence(task)¶
Bases:
SimplePersistence
- property plugin¶
- flexget.utils.simple_persistence.db_cleanup(manager, session)¶
Clean up values in the db from tasks which no longer exist.
- flexget.utils.simple_persistence.flush_task(task)¶
Stores all in memory key/value pairs to database when a task has completed.
- flexget.utils.simple_persistence.flush_taskless(manager)¶
- flexget.utils.simple_persistence.load_task(task)¶
Loads all key/value pairs into memory before a task starts.
- flexget.utils.simple_persistence.load_taskless(manager)¶
Loads all key/value pairs into memory which aren’t associated with a specific task.
soup
Module¶
sqlalchemy_utils
Module¶
Miscellaneous SQLAlchemy helpers.
- class flexget.utils.sqlalchemy_utils.ContextSession(bind: _SessionBind | None = None, *, autoflush: bool = True, future: Literal[True] = True, expire_on_commit: bool = True, autobegin: bool = True, twophase: bool = False, binds: Dict[_SessionBindKey, _SessionBind] | None = None, enable_baked_queries: bool = True, info: _InfoType | None = None, query_cls: Type[Query[Any]] | None = None, autocommit: Literal[False] = False, join_transaction_mode: JoinTransactionMode = 'conditional_savepoint', close_resets_only: bool | _NoArg = _NoArg.NO_ARG)¶
Bases:
Session
sqlalchemy.orm.Session
which automatically commits when used as context manager without errors
- flexget.utils.sqlalchemy_utils.create_index(table_name: str, session: Session, *column_names: str) None ¶
Creates an index on specified columns in table_name
- Parameters:
table_name – Name of table to create the index on.
session – Session object which should be used
column_names – The names of the columns that should belong to this index.
- flexget.utils.sqlalchemy_utils.drop_index(table_name: str, index_name: str, session: Session) None ¶
Drops an index by table name and index name
- Parameters:
table_name (string) – Name of table
index_name (string) – Name of the index
session (Session) – SQLAlchemy Session
- flexget.utils.sqlalchemy_utils.drop_tables(names: List[str], session: Session) None ¶
Takes a list of table names and drops them from the database if they exist.
- flexget.utils.sqlalchemy_utils.get_index_by_name(table: Table, name: str) Index | None ¶
Find declaratively defined index from table by name
- Parameters:
table – Table object
name (string) – Name of the index to get
- Returns:
Index object
- flexget.utils.sqlalchemy_utils.index_exists(table_name: str, index_name: str, session: Session) bool ¶
Use SQLAlchemy reflect to check index existences.
- flexget.utils.sqlalchemy_utils.table_add_column(table: Table | str, name: str, col_type: TypeEngine | type, session: Session, default: Any | None = None) None ¶
Adds a column to a table
Warning
Uses raw statements, probably needs to be changed in order to work on other databases besides SQLite
- Parameters:
table (string) – Table to add column to (can be name or schema)
name (string) – Name of new column to add
col_type – The sqlalchemy column type to add
session (Session) – SQLAlchemy Session to do the alteration
default – Default value for the created column (optional)
- flexget.utils.sqlalchemy_utils.table_columns(table: str | Table, session: Session) List[str] ¶
- Parameters:
table (string) – Name of table or table schema
session (Session) – SQLAlchemy Session
- Returns:
List of column names in the table or empty list
- flexget.utils.sqlalchemy_utils.table_exists(name: str, session: Session) bool ¶
Use SQLAlchemy reflect to check table existences.
- flexget.utils.sqlalchemy_utils.table_index(table_name: str, index_name: str, session: Session) Index ¶
Finds an index by table name and index name
- Parameters:
table_name (string) – Name of table
index_name (string) – Name of the index
session (Session) – SQLAlchemy Session
- Returns:
The requested index
template
Module¶
- class flexget.utils.template.CoercingDateTime¶
Bases:
DateTime
Datetime with some features that make it better when used in our templates: - Avoids crashing when comparing tz aware and naive datetimes.
When this happens, it will assume the naive datetime is in the same timezone as the dt aware one.
Allows comparisons with plain dates, where the date is assumed to be at midnight in the same timezone.
Returns `Interval`s with a nicer string representation for our templates
This allows us to introduce tz aware datetimes into entry fields without breaking old configs, or old plugins.
- class flexget.utils.template.FlexGetEnvironment(block_start_string: str = '{%', block_end_string: str = '%}', variable_start_string: str = '{{', variable_end_string: str = '}}', comment_start_string: str = '{#', comment_end_string: str = '#}', line_statement_prefix: str | None = None, line_comment_prefix: str | None = None, trim_blocks: bool = False, lstrip_blocks: bool = False, newline_sequence: te.Literal['\n', '\r\n', '\r'] = '\n', keep_trailing_newline: bool = False, extensions: ~typing.Sequence[str | ~typing.Type[Extension]] = (), optimized: bool = True, undefined: ~typing.Type[~jinja2.runtime.Undefined] = <class 'jinja2.runtime.Undefined'>, finalize: ~typing.Callable[[...], ~typing.Any] | None = None, autoescape: bool | ~typing.Callable[[str | None], bool] = False, loader: BaseLoader | None = None, cache_size: int = 400, auto_reload: bool = True, bytecode_cache: BytecodeCache | None = None, enable_async: bool = False)¶
Bases:
Environment
Environment with template_class support
- class flexget.utils.template.FlexGetNativeTemplate(source: str | ~jinja2.nodes.Template, block_start_string: str = '{%', block_end_string: str = '%}', variable_start_string: str = '{{', variable_end_string: str = '}}', comment_start_string: str = '{#', comment_end_string: str = '#}', line_statement_prefix: str | None = None, line_comment_prefix: str | None = None, trim_blocks: bool = False, lstrip_blocks: bool = False, newline_sequence: te.Literal['\n', '\r\n', '\r'] = '\n', keep_trailing_newline: bool = False, extensions: ~typing.Sequence[str | ~typing.Type[Extension]] = (), optimized: bool = True, undefined: ~typing.Type[~jinja2.runtime.Undefined] = <class 'jinja2.runtime.Undefined'>, finalize: ~typing.Callable[[...], ~typing.Any] | None = None, autoescape: bool | ~typing.Callable[[str | None], bool] = False, enable_async: bool = False)¶
Bases:
FlexGetTemplate
,NativeTemplate
Lazy lookup support and native python return types.
- class flexget.utils.template.FlexGetTemplate(source: str | ~jinja2.nodes.Template, block_start_string: str = '{%', block_end_string: str = '%}', variable_start_string: str = '{{', variable_end_string: str = '}}', comment_start_string: str = '{#', comment_end_string: str = '#}', line_statement_prefix: str | None = None, line_comment_prefix: str | None = None, trim_blocks: bool = False, lstrip_blocks: bool = False, newline_sequence: te.Literal['\n', '\r\n', '\r'] = '\n', keep_trailing_newline: bool = False, extensions: ~typing.Sequence[str | ~typing.Type[Extension]] = (), optimized: bool = True, undefined: ~typing.Type[~jinja2.runtime.Undefined] = <class 'jinja2.runtime.Undefined'>, finalize: ~typing.Callable[[...], ~typing.Any] | None = None, autoescape: bool | ~typing.Callable[[str | None], bool] = False, enable_async: bool = False)¶
Bases:
Template
Adds lazy lookup support when rendering templates.
- new_context(vars=None, shared=False, locals=None)¶
Create a new
Context
for this template. The vars provided will be passed to the template. Per default the globals are added to the context. If shared is set to True the data is passed as is to the context without adding the globals.locals can be a dict of local variables for internal usage.
- class flexget.utils.template.Interval(start: DateTime | datetime, end: DateTime | datetime, absolute: bool = False)¶
- class flexget.utils.template.Interval(start: Date | date, end: Date | date, absolute: bool = False)
Bases:
Interval
Same as normal Interval, but gives a better string representation for our templates.
- exception flexget.utils.template.RenderError¶
Bases:
Exception
Error raised when there is a problem with jinja rendering.
- flexget.utils.template.evaluate_expression(expression: str, context: Mapping) Any ¶
Evaluate a jinja expression using a given context with support for `LazyDict`s (`Entry`s.)
- Parameters:
expression (str) – A jinja expression to evaluate
context – dictlike, supporting LazyDicts
- flexget.utils.template.filter_d(value, default_value: str = '', boolean: bool = True) str ¶
Override the built-in Jinja default filter to set the boolean param to True by default
- flexget.utils.template.filter_default(value, default_value: str = '', boolean: bool = True) str ¶
Override the built-in Jinja default filter to set the boolean param to True by default
- flexget.utils.template.filter_format_number(val, places: int | None = None, grouping: bool = True) str ¶
Formats a number according to the user’s locale.
- flexget.utils.template.filter_formatdate(val, format_str)¶
Returns a string representation of a datetime object according to format string.
- flexget.utils.template.filter_pad(val: int | str, width: int, fillchar: str = '0') str ¶
Pads a number or string with fillchar to the specified width.
- flexget.utils.template.filter_parse_size(val: str, si: bool = False, match_re: str | None = None) int ¶
Parse human-readable file size to bytes
- flexget.utils.template.filter_parsedate(val)¶
Attempts to parse a date according to the rules in ISO 8601 and RFC 2822
- flexget.utils.template.filter_pathext(val: str | None) str ¶
Extension of a path (including the ‘.’).
- flexget.utils.template.filter_pathname(val: str | None) str ¶
Base name of a path, without its extension.
- flexget.utils.template.filter_pathscrub(val: str, os_mode: str | None = None) str ¶
Replace problematic characters in a path.
- flexget.utils.template.filter_re_replace(val: AnyStr, pattern: str, repl: str) str ¶
Perform a regexp replacement on the given string.
- flexget.utils.template.filter_re_search(val, pattern: str)¶
Perform a search for given regexp pattern, return the matching portion of the text.
- flexget.utils.template.filter_to_date(date_time_val)¶
Returns the date from any date-time object
- flexget.utils.template.get_filters() dict ¶
Returns all built-in and custom Jinja filters in a dict
The key is the name, and the value is the filter func
- flexget.utils.template.get_template(template_name: str, scope: str | None = 'task') FlexGetTemplate ¶
Loads a template from disk. Looks in both included plugins and users custom scope dir.
- flexget.utils.template.is_fs_dir(pathname: str | PathLike) bool ¶
Test whether item is existing directory in filesystem
- flexget.utils.template.is_fs_file(pathname: str | PathLike) bool ¶
Test whether item is existing file in filesystem
- flexget.utils.template.is_fs_link(pathname: str | PathLike) bool ¶
Test whether item is existing link in filesystem
- flexget.utils.template.list_templates(extensions: List[str] | None = None) List[str] ¶
Returns all templates names that are configured under environment loader dirs
- flexget.utils.template.make_environment(manager: Manager) None ¶
Create our environment and add our custom filters
- flexget.utils.template.render(template: FlexGetTemplate | str, context: Mapping, native: bool = False) str ¶
Renders a Template with context as its context.
- Parameters:
template – Template or template string to render.
context – Context to render the template from.
native – If True, and the rendering result can be all native python types, not just strings.
- Returns:
The rendered template text.
- flexget.utils.template.render_from_entry(template: FlexGetTemplate | str, entry: Entry, native: bool = False) str ¶
Renders a Template or template string with an Entry as its context.
- flexget.utils.template.render_from_task(template: FlexGetTemplate | str, task: Task) str ¶
Renders a Template with a task as its context.
- Parameters:
template – Template or template string to render.
task – Task to render the template from.
- Returns:
The rendered template text.
tools
Module¶
Contains miscellaneous helpers
- class flexget.utils.tools.BufferQueue(maxsize=0)¶
Bases:
Queue
Used in place of a file-like object to capture text and access it safely from another thread.
- exception Empty¶
Bases:
Exception
Exception raised by Queue.get(block=0)/get_nowait().
- write(line)¶
- class flexget.utils.tools.ReList(*args, **kwargs)¶
Bases:
list
A list that stores regexps.
You can add compiled or uncompiled regexps to the list. It will always return the compiled version. It will compile the text regexps on demand when first accessed.
- flags = 2¶
- class flexget.utils.tools.TimedDict(cache_time: timedelta | str = '5 minutes')¶
Bases:
MutableMapping
Acts like a normal dict, but keys will only remain in the dictionary for a specified time span.
- classmethod clear_all()¶
Clears all instantiated TimedDicts. Used by tests to make sure artifacts don’t leak between tests.
- class flexget.utils.tools.TitleYear(title, year)¶
Bases:
NamedTuple
- flexget.utils.tools.chunked(seq: Sequence, limit: int = 900) Iterator[Sequence] ¶
Helper to divide our expired lists into sizes sqlite can handle in a query. (<1000)
- flexget.utils.tools.decode_html(value: str) str ¶
- Parameters:
value (string) – String to be html-decoded
- Returns:
Html decoded string
- flexget.utils.tools.encode_html(unicode_data: str, encoding: str = 'ascii') bytes ¶
Encode unicode_data for use as XML or HTML, with characters outside of the encoding converted to XML numeric character references.
- flexget.utils.tools.format_filesize(num_bytes: int | float, si: bool = False, unit: str | None = None) str ¶
Returns given bytes as prettified string.
- flexget.utils.tools.get_config_as_array(config: dict, key: str) list ¶
Return configuration key as array, even if given as a single string :param dict config: Configuration :param string key: Configuration :return: Array
- flexget.utils.tools.get_config_hash(config: Any) str ¶
- Parameters:
config (dict) – Configuration
- Returns:
MD5 hash for config
- flexget.utils.tools.get_latest_flexget_version_number() str | None ¶
Return latest Flexget version from https://pypi.python.org/pypi/FlexGet/json
- flexget.utils.tools.group_entries(entries: Iterable[Entry], identifier: str) Dict[str, List[Entry]] ¶
- flexget.utils.tools.merge_dict_from_to(d1: dict, d2: dict) None ¶
Merges dictionary d1 into dictionary d2. d1 will remain in original form.
- flexget.utils.tools.multiply_timedelta(interval: timedelta, number: int | float) timedelta ¶
`timedelta`s can not normally be multiplied by floating points. This does that.
- flexget.utils.tools.parse_episode_identifier(ep_id: str | int, identify_season: bool = False) Tuple[str, str] ¶
Parses series episode identifier, raises ValueError if it fails
- Parameters:
ep_id – Value to parse
- Returns:
Return identifier type: sequence, ep or date
- Raises:
ValueError – If ep_id does not match any valid types
- flexget.utils.tools.parse_filesize(text_size: str, si: bool = True, match_re: str | Pattern[str] | None = None) int ¶
Parses a data size and returns its value in bytes
- Parameters:
text_size (string) – string containing the data size to parse i.e. “5 GB”
si (bool) – If True, possibly ambiguous units like KB, MB, GB will be assumed to be base 10 units, rather than base 2. i.e. if si then 1 KB = 1000 B else 1 KB = 1024 B
match_re – A custom regex can be defined to match the size. The first capture group should match the number, and the second should match the unit.
- Returns:
an int with the data size in bytes
- flexget.utils.tools.parse_timedelta(value: timedelta | str | None) timedelta ¶
Parse a string like ‘5 days’ into a timedelta object. Also allows timedeltas to pass through.