Only URLs of the forms
http://hostname/[localpart]
http://hostname:port/[localpart]
are supported.
Options:
Save output automatically based on URI. Will not warn if the file already exists. This overrides the -o (--out) option. The prefered output name is everything after the last / in the URL, or 'dir-default' if the URL ends with a /.
Don't print the body of the response.
What browser to emulate. Use --emulations to list available browser headers.
Set the cookie header with VALUE.
As of version 1.2, when the response headers indicate a Transfer-Encoding of 'chunked', bget will rename the header (prefixing it with 'Xbget-') and unchunk the response. This is desirable so that chunked responses from HTTP/1.1 servers look right. In some cases it may be desirable to see raw output from the server however, so this behavior can be turned off.
Make a HEAD request instead of a GET. Note that this does not imply -h (--heads) to print the headers, nor -B (--no-body) to supress printing any body content. (Some servers, eg www.yahoo.com, treat HEAD like a GET.)
Read URLs from FILE (one per line) instead of from command line. Use filename "-" for standard input.
If there are two URLs on a line, the first one is used as the referer URL. The referer will remain un use until the next line with two URLs.
If there is an additional field after the URL, that will be used as an -o (--out) output file until the next line with an output file. An output file should not begin with ``http:/'' or ``https:/''.
Fields on each line of the URL file are whitespace separated.
Follow redirects. If printing headers, the redirecting headers and the destination headers will be printed. (No loop detection is attempted.) If printing bodies and not saving via autoname, the redirecting body and the destination body will be printed. If saving via autoname, a new file will be opened for each request made. Some redirects (eg loops) may cause the autonaming to pick the same filename as a previous request, which will cause the earlier file to be clobbered.
Connect to HOST for request (useful for testing virtual hosts before a DNS change or use with -l for proxies).
Print the response headers.
Use LANG for Accept-Language: header. See --languages for a small list.
Use long address on GET line (using the full http://... format, a MUST for HTTP/1.1 server compliance but handy with -H for proxies).
Write output to FILE. Unlike -a (--autoname) this will not use a different file for each request. The autoname option has precedence over this option. Filenames in a -F (--file) URL file will also override this.
Use STRING as a post form contents (forms of type application/x-www-form-urlencoded only).
Use contents of FILE as a post form contents. If the first line is of the form ``Content-Type: foo/bar'' it will be used to set the Content-Type: header. More than just the MIME type is allowed, but it must be all on one line. Typical POST content types are
application/x-www-form-urlencoded
Encoded like a typical CGI URL.
multipart/form-data
Each form element is in a separate MIME part; needed for
file uploads. This type requires a boundary parameter on
the Content-Type: header.
There is a similar allowance for setting the Transfer-Encoding: header. This must be on the second line if Content-Type: is set, and on the first line if not. When Transfer-Encoding: contains the string 'chunked', Content-Length: will not be set for a post. Note that apache 1.3.x (at least) does not allow chunked POST requests.
You may find the tool "mkpost" (available in the scripts section of CPAN) to be helpful in creating CGI interface files for this option. For other content types, like ``text/xml'' for XML-RPC interface requests, other tools will be needed.
Set the (initial) referer header with VALUE.
Print the request headers.
After fetching a page --- including following redirects and printing bits of the response as controlled by other options --- if the HTTP status code is not exactly the one given, bget will exit (returning code 3 to the shell). Useful for looping until one hits a 404 or the like.
Use Benchmark module to time making the command line request(s) N times.
Just like -t/--time, but optimizations apply: if neither heads nor bodies are requested, nothing will be fetched. If body is not requested only heads will be fetched.
Basic authentification in the form {username}:{password}.
Wait N seconds between fetching each URL.
Waits a random number of seconds, average A standard deviation D, between fetching each URL. Requires the Math::Random module. Useful for being subtle when fetching a lot of pages, along with emulating a browser and using per-page referer headers via the -F (--file) method.
Show a help message and exit.
Print version and exit.
Print list of available browser emulations.
Print a sample of language codes.
With -L (--langauge) the Accept-Language: header will not be added if the browser has not been observed to use it.
Amaya is the W3C's combination browser page editor.
Text mode browser for Unix. <http://artax.karlin.mff.cuni.cz/~mikulas/links> Version 0.84 does not do cookies or referer headers, so we might misemulate it that way.
Forked from links, this is another text mode browser. Quirks include giving a bunch away about the system, including window size, in the User-Agent: and including a 'Referer' header in URLs entered by hand. The User-Agent for this is from a Redhat 7.1 x86 system in an 80x24 window. <http://elinks.or.cz/>
Command line web tool that uses libwww. <http://www.w3.org/ComLine/>
Text mode browser for Unix. <http://ei5nazha.yz.yamagata-u.ac.jp/~aito/w3m/>
Dillo is Linux browser, under current development, that focuses on speed, small size, and protocol correctness. It can do cookies, but it defaults to not accepting them. It does not do Referer: headers, but bget may misemulate it on that point. <http://www.dillo.org/>
The browser that started the rush, compiled for Linux. This is an archaic browser. It doesn't do Host: headers or Cookies:. bget can misemulate the Cookies: part, but won't do the Host: header. Many modern sites require this for proper operation, so expect problems. The headers this thing spits out are longer even than the Lynx ones.
Qweb was an early X11 style-sheet capable browser. Too bad it didn't do javascript (needed for some stylesheets) or even Host: headers. bget will misemulate this if you use Cookies, but won't supply a Host: header.
The name 'Chimera' has been used by two different browsers. This is the X11 Chimera developed at the University of Las Vegas, not the Mac Mozilla derivative Chimera. In authentic use this browser does not have cookies or use Referer: headers.
ab, the benchmark tool that comes with the Apache httpd package.
An old version of a popular alternative browser for Windows.
More modern (2003) version of Opera.
As of Opera 6.x there is a linux version.
Lib WWW Perl module (these are the default headers).
Command-line bulk page downloading tool for Unix.
Command-line page upload/download tool for Unix. Prefers HTTP/1.1 but can do HTTP/1.0 upon request. Can do PUTs and DELETEs and other obscure things, too.
Curl in HTTP/1.0 mode.
Popular alternative browser for Macs.
Once popular ad- and cookie-filtering proxy. Junkbuster does a bunch of header editing from the actual browser headers, and thus the headers out of it can vary considerably from this. It looks like Accept-* headers are not edited, allowing identification of the underlying browser sometimes. The Accept-* headers here come from a Netscape 4.7. By default, Junkbuster masquerades as Netscape 3.01 (GOLD) for Mac PPC.
Popular text mode browser, predominately unix.
Mozilla is the open source version of Netscape 7. It exists for many platforms.
Phoenix (formerly Firebird) is Mozilla with a different user-interface library. There are unix, windows and mac variants.
Konqueror is a mostly-Linux browser based on KDE.
OpenOffice is a StarOffice relation, intended to be a free Unix ``Office'' compatible software bundle. It includes an HTML editor that can download pages to edit, but as such it does things like issue PROPFIND requests that are not emulated here.
Explorer 5.0 can be installed with a compatibility mode that emulates (or claims to emaulate) Explorer 4.0.
This is on a system with IE5.5 installed, but this identifies itself as IE4.01. This one is hard to do right, since in my tests I saw two requests for the test file. The first came with this UA, the second had this instead:
User-Agent: Mozilla/4.0 (compatible; MSIE 4.01; MSIECrawler; Windows NT)
The crawler version had an 'Accept-Language: us-en' as well as a different order to the headers (Accept: User-Agent:, Accept-Language: Accept-Encoding, Host:).
Normal mode Windows NT IE 5.0.
IE can optionally crawl pages to cache them for offline browsing. This is Windows NT IE 5.01 in crawl mode.
Mac PPC is System 7, 8 or 9 on PowerPC computers.
Safari is a Mozilla derivative that ships with OS X.
Internet Explorer for OS X. (Comes with OS X?)
Some other lanuages:
af Afrikaans sq Albanian eu Basque bg Bulgarian be Byelorussian ca Catalan zh Chinese zh-cn Chinese/China zh-tw Chinese/Taiwan hr Croatian cs Czech da Danish nl Dutch nl-be Dutch/Belgium fo Faeroese fi Finnish fr French fr-be French/Belgium fr-ca French/Canada fr-fr French/France fr-ch French/Switzerland gl Galician de German de-at German/Austria de-de German/Germany de-ch German/Switzerland el Greek hu Hungarian is Icelandic id Indonesian ga Irish it Italian ja Japanese ko Korean mk Macedonian no Norwegian pl Polish pt Portuguese pt-br Portuguese/Brazil ro Romanian ru Russian gd Scots Gaelic sr Serbian sk Slovak sl Slovenian es Spanish es-ar Spanish/Argentina es-co Spanish/Colombia ex-mx Spanish/Mexico es-es Spanish/Spain sv Swedish tr Turkish uk Ukrainian
This list is from the default set of lanuages in Netscape 4.5. IE has a different set, including more country variations. Note that the country variations are frequently misused. A request with a language header like:
Accept-Language: en-us, es-mx; q=0.7, fr-ca; q=0.3
Would specify a first choice language of US English, second choice Mexican Spanish, third choice Canadian French. If a content-negotiating server only has generic English, generic Spanish, and generic French, then by specification it should return a ``406 Not Acceptable'' error, since it has no languages that match. This could be seen as a deficiency of the spec, but that's the way it is.
By supporting chunked transfer encodings, the author considers bget to be HTTP/1.1 compliant now. A word of warning, some emulations specify various allowed other encodings, like gziped content. You should be prepared to deal with these outside of bget.