Man page - libwget-robots(3)

Packages contas this manual

Package: wget2-dev
apt-get install wget2-dev

Manuals in package:

Documentations in package:

wget2-dev

Manual

libwget-robots(3)

Library Functions Manual

libwget-robots(3)

NAME

libwget-robots - Robots Exclusion file parser

SYNOPSIS

Macros

#define parse_record_field(d, f) parse_record_field(d, f, sizeof(f) - 1)

int wget_robots_parse (wget_robots **_robots, const char *data, const char *client)
void wget_robots_free (wget_robots **robots)
int wget_robots_get_path_count (wget_robots *robots)
wget_string * wget_robots_get_path (wget_robots *robots, int index)
int wget_robots_get_sitemap_count (wget_robots *robots)
const char * wget_robots_get_sitemap (wget_robots *robots, int index)

Detailed Description

The purpose of this set of functions is to parse a Robots Exclusion Standard file into a data structure for easy access.

Function Documentation

int wget_robots_parse (wget_robots ** _robots, const char * data, const char * client)

Parameters

data Memory with robots.txt content (with trailing 0-byte)
client Name of the client / user-agent

Returns

Return an allocated wget_robots structure or NULL on error

The function parses the robots.txt data in accordance to https://www.robotstxt.org/orig.html#format and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.

The ROBOTS structure has to be freed by calling wget_robots_free().