Robots Exclusion file parser

Functions
ROBOTS *	wget_robots_parse (const char *data, const char *client)
void	wget_robots_free (ROBOTS **robots)

Detailed Description

The purpose of this set of functions is to parse a Robots Exlusion Standard file into a data structure for easy access.

Function Documentation

wget_robots_parse()

ROBOTS* wget_robots_parse	(	const char *	data,
		const char *	client
	)

Parameters

[in]	data	Memory with robots.txt content (with trailing 0-byte)
[in]	client	Name of the client / user-agent

Returns

Return an allocated ROBOTS structure or NULL on error

The function parses the robots.txt data and returns a ROBOTS structure including a list of the disallowed paths and including a list of the sitemap files.

The ROBOTS structure has to be freed by calling wget_robots_free().

wget_robots_free()

void wget_robots_free

(

ROBOTS **

robots

)

Parameters

[in,out]

robots

Pointer to Pointer to ROBOTS structure

wget_robots_free() free's the formerly allocated ROBOTS structure.