Skip to content
1 viewer

rust 1,860,549 · typescript 235,540 · javascript 132,956 · python 20,426 · 6,408 files · digest f04d4637f52b · built in 1.5 s · crystals off

crates/cre8-util-legal-text/src/robots.rs

cre8-util-legal-text · module robots · src · 407 lines (324 code, 53 comment) · sha 0c738334edfb248b

Items

const ROBOTS_PRODUCT_TOKEN const AI_CRAWLER_TOKENS struct RobotsRule struct RobotsRules struct RobotsGroup impl RobotsRules const ROBOTS_KEYS const LEGINFO_ROBOTS const USCODE_ROBOTS const OHIO_ROBOTS const NCLEG_ROBOTS const MN_ROBOTS

Functions

By call-graph weight (PageRank; 1.0 is the average function). 18 functions in all.

functionkindlinescyclomaticcallerscalleesweightcrystal
robots_pattern_matchessrc/robots.rs:283fn237102.4
RobotsRules::is_allowedsrc/robots.rs:199method1981012.3
RobotsRules::parsesrc/robots.rs:87method8721941.8
RobotsRules::may_visit_atsrc/robots.rs:176method74201.2
split_fused_directivessrc/robots.rs:262fn205201
visit_windowsrc/robots.rs:225fn124100.9
request_intervalsrc/robots.rs:241fn134100.9
RobotsRules::allow_allsrc/robots.rs:73method31400.8
RobotsRules::visit_windows_labelsrc/robots.rs:190method71300.8
RobotsRules::may_visit_at_unix_mssrc/robots.rs:185method31210.7
RobotsRules::disallow_allsrc/robots.rs:78method81100.6
a_directive_fused_onto_the_user_agent_line_still_appliessrc/robots.rs:321fn121030.5
a_leading_byte_order_mark_does_not_hide_the_first_groupsrc/robots.rs:335fn61020.5
leginfo_robots_disallows_every_page_for_every_crawlersrc/robots.rs:343fn71020.5
uscode_robots_only_names_another_crawlersrc/robots.rs:352fn51020.5
robots_longest_match_wins_and_allow_breaks_ties_and_wildcards_anchorsrc/robots.rs:359fn121040.5
visit_time_opens_a_utc_window_and_a_bare_request_rate_is_not_guessedsrc/robots.rs:376fn185060.5
ai_crawlers_shut_out_by_name_are_recorded_without_binding_cre8src/robots.rs:396fn111020.5

Source not readable from this process.